Netflix To Re-Encode Entire 1 Petabyte Video Catalogue In 2016 To Save Bandwidth (variety.com) 285
An anonymous reader writes: Netflix has spent four years developing a new and more efficient video-encoding process that can shave off 20% in terms of space and bandwidth without reducing the quality of streamed video. With streaming video accounting for 70% of broadband use, the saving is much-needed, although the advent of 4K streaming, higher frame rates and HDR are likely to account for it all soon after. Netflix video algorithms manager Anne Aaron explained to Variety that certain types of video benefit little from the one-size-fits-all compression approach that Netflix has been using until now: "You shouldn't allocate the same amount of bits for My Little Pony as for The Avengers."
My little pony (Score:5, Funny)
"You shouldn't allocate the same amount of bits for My Little Pony as for The Avengers."
So they're dropping the resolution for The Avengers?
Re: (Score:3)
damn, mod points just ran out. Best first post in a long, long time :D
Re: (Score:3, Funny)
Absolutely. I need to see every little pony in all its unpixelated, 4K glory.
Wait... that didn't come out right.
Re:My little pony (Score:5, Interesting)
On a serious note, animated content is much harder for the 8-bit encoding. It's the hard edges with high contrast cell shading. You get a lot more compression artifacts than a typical movie. You can resolve this by using 10-bit encoding, but there's a lot of Netflix devices with embedding video codecs. They really can't change, and almost none of the chipsets out there support 10-bit decoding. So that leaves option two, which is to increase the bitrate.
Re: (Score:2)
Re:My little pony (Score:5, Informative)
That animated content benefits from 10-bit encoding is true. That has less to do with hard edges and more to do with banding artifacts on flat shaded areas - TFA actually goes into that, mentioning soft focus and fog as producing hard-to-encode gradients, the same kind of gradients present in many kinds of animation and which would benefit from using 10-bit mode. Hard edges do tend to be hard to encode with typical video codecs too (but 10-bit probably won't help you there).
However, My Little Pony isn't a particularly good example, because it's full of completely flat areas that are trivial to encode. It might take a higher quality setting than you might expect to look crisp, but at the end of the day, you're going to be spending fewer bits per frame on it than on The Avengers. Animation has its own set of encoding tradeoffs/challenges (which is why good encoders have presets tuned for animation).
Re: (Score:2)
animated content is much harder for the 8-bit encoding. It's the hard edges with high contrast cell shading.
Nonsense. Animation is optimized for compression.. It's already dithered to a limited color set! Whole swathes of pixels get the same value. Sure, you might be able to SEE the compression.. But it is a breeze for the hardware, and bitrates can be dramatically reduced compared to the baseband 4-2-2 video.
Re: (Score:2, Funny)
I suggest no one do that, because it's fucking horse shit. I know the fucking document you're referring to, and it's wrong, wrong, wrong.
There is no reason to use 10-bit encoding over 8-bit encoding unless you're using a shitty fucking encoder that introduces rounding errors at every step. Decent encoders work in a higher precision before shitting out the encoded result.
Vectors format? (Score:2)
How is modern animation done? If it's done on computers in some sort of vector drawing format then, unless the drawings are really complicated, it seems a little silly to convert the (lossless) vector drawings to a (lossy) video in the first place. Surely modern computers can render 2D vector drawings in real time? I assume that the vector information can be stored relatively compactly.
FWIW, it sounds like some of the animation for My little pony is done with Flash [wikipedia.org], and much as I hate flash, a flash animati
Re: (Score:2)
Sure, but it's awfully hard to protect. If they just spew the original vector drawings around the Internets, you could use them to make those ponies do whatever you want!
Re: (Score:3, Interesting)
I recode a lot of video (don't ask), and in my experience, animation is a lot easier on the resulting files for the same quality. The absolute worst you can do if you want small high quality files is "film grain", whether it is from a bad source or artificially added for artistic reasons. Second worst is a badly compressed source with lots of artifacts. Then there's video with lots of small objects moving across a detailed background. The hard contrasts at the edges of cell shaded videos are only problemati
Re: (Score:2)
Or maybe not, and she's bash
Re: (Score:2)
I can't tell if she's saying the streams have different priorities or if they should be compressed with different codecs.
It seems both. I'd say the best way to save bandwidth is to reduce the popular titles first. All the latest compression technologies can't reduce the existing bandwidth used for watching Vanilla Sky.
Re: (Score:2, Funny)
4-year-olds? I love that show, dude. Don't pixelate the pony, bro!
Compression ruins the glistening of the glitter and the graceful flowing of manes during brisk gallops.
Re: (Score:2)
If you don't already know who "bronies" are, don't look for them.
Another year, another video codec... (Score:5, Interesting)
Shaving 20% off seems pretty optimistic to me. Unless they've suddenly discovered some whole new realm of compression mathematics I'd be surprised if thats anything more than a peak compression in some rare edge cases.
Re:Another year, another video codec... (Score:5, Interesting)
Shaving 20% off seems pretty optimistic to me. Unless they've suddenly discovered some whole new realm of compression mathematics I'd be surprised if thats anything more than a peak compression in some rare edge cases.
Sounds more like as a part of re-compression, they are going to drop the bitrate (and video quality?) for videos that don't "need" it:
certain types of video benefit little from the one-size-fits-all compression approach that Netflix has been using until now: "You shouldn't allocate the same amount of bits for My Little Pony as for The Avengers."
Re: (Score:3)
Sounds more like as a part of re-compression, they are going to drop the bitrate (and video quality?) for videos that don't "need" it:
Is it the bitrate that's being changed? I took the part about Pony vs Avengers to mean animation vs live action are better compressed by completely different algorythms.
Re:Another year, another video codec... (Score:5, Informative)
The article is actually in English, you know.
"The new system will encode from the raw source material more intelligently, considering whether or not the material itself can really benefit from higher bit-rates, or whether identical quality can be maintained with less space and bandwidth."
Re: (Score:2)
The Avengers is not animated?
Pinkie Pie is flatter than Emma Peel (Score:2)
The shaders in MLP:FIM are much simpler than those in The Avengers, no matter whether Emma "Black Widow" Peel is played by Scarlett Johansson or Uma Thurman [geektyrant.com]. Flat-shaded animation has harder edges and less texture than photoreal-shaded animation and thus may need different compression techniques to improve the rate for a given level of distortion.
Re: (Score:2)
They suggest that they are going to have an automated process to transcode content a few ways and use some automated quality check to decide what meets the threshold. Or they are trying to make changing from CBR to VBR sound more impressive than it is.
Re: (Score:2)
Re:Another year, another video codec... (Score:5, Insightful)
I suspect that by the time of this announcement they have already done the testing, so have a good idea of how much they can optimize. From the article, it's more about optimizing compression parameters to fit the source material rather than taking a one-size-fits-all approach.
Re: (Score:2)
They might also be able to employ different and better options on the transcoder. When I first started transcoding my own stuff I optimized for quality and total processing time. The files were meant for HTPC use and mobile video devices weren't terribly common yet. What I ended up with was something that early iDevices couldn't even handle (although Archos could).
They may simply be adjusting for more modern and more capable devices.
Re:Another year, another video codec... (Score:4, Interesting)
I am in the process of moving my fiancee's dvd/bluray collection to my server and putting her physical copies in storage. Using Handbrake, switching from x264 to x265 saves me at lease 10 % on dvd sources and closer to 30+% on the bluray sources.
Re:Another year, another video codec... (Score:4, Interesting)
I remember ripping my CD collection to ogg, only have to do it again years later to flac when space got cheaper. The ogg was fine, but not a good source for re-encoding to another format such as mp3.
If I was going to rip movies, I'd keep the original streams. You'll never spare the time again to re-rip, even if you you think now that you will.
Re: (Score:2)
Re: (Score:2)
And now you have x265 that's not hardware accelerated on anything but the most modern GPUs (and even then, only partially [anandtech.com] -- certainly not suitable for any set-top-box, tablet or mobile phone. Heck, even a laptop that has partial GPU supporting (or none) will burn through tons of battery watching it on a flight with no power plugs.
Netflix has to support all those platforms (and probably worse ones) -- and then you enter the idea of having multiple copies for every asset :-(
Re: (Score:2)
I'm not using very aggressive encoding settings and my google player box has been able to play all of the movies I ripped so far (I know it's not hardware accelerated but as long as it plays without stuttering, it makes no difference to me.)
Guess I'll double check my amazon fire (not the 4k model) and make sure it's able to play those too.
Re: (Score:2)
My smart TV can play HEVC content just fine (using the web browser). I doubt it could do it without any hardware acceleration.
Most smart TVs use underpowered smartphone chips.
Re: (Score:2)
10 % for dvds and 30 % for blurays. And that was compared to the x264 rip not the source.
Now that you mention it though, I might have to debate the sd rips going to x265. Not sure if it's worth the extra encoding time on those. It is easy to see the plus, though, for the HD stuff. The storage savings add up fast.
As far as just putting the original on my server, i thought about it but 20-30 GB a movie would add up pretty fast (and that is with removing most of the sound tracks and all the extras).
Re: (Score:3, Informative)
Re-compressing BD kind of defeats the whole point of bothering with BD. Plus you magnify the aforementioned decode support issues.
I disagree there. They have 50GB to fill and they're going to use as much as they can and likely CBR. If you use Handbrake with CRF at 18 or so, you're not going to see a difference, and you're going to save a bit on hardware if you have a large collection to rip.
But when ripping TV content where several episodes are crammed onto one disc, compare the output to the original. You may have made a larger file.
As for DTS-MA, you could probably extract DTS core and still have way better than DVD audio. For n
Re: (Score:2)
I'll look at that. In real world, do you notice much difference between a NAS harddrive and a desktop harddrive in a home server?
Re: (Score:3)
You shouldn't. "NAS" drives, at least as they come from WD, are just Greens with TLER so they don't drop out of RAID. If you're not RAIDing, or if you're doing software RAID with MDADM or ZFS, shouldn't matter what you use. Even with hardware RAID, it really only matters if you're doing parity RAID. 1 or 10 won't care.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
On the contrary, I'm surprised they're only saving 20%. Look at the difference between VP9 or X265 and the VC-1 encoder they used exclusively for their first few years. And given the costs they incur using all this bandwidth they could definitely throw a good number of top quality engineers at figuring out a rate control algorithm that's more suited to their rather unusual use case.
Re: (Score:2)
I think its more a matter of (potentially) saving 20% versus what they are using now.
Didn't read the whole blurb, but they are probably going full-in on HEVC ( https://en.wikipedia.org/wiki/... [wikipedia.org] )
The Wikipedia article mentions 50-60% savings using HEVC over H.264 depending on resolution, but chances are these are optimum subjective results. My Little Pony (and other animated shows) would probably encode much tighter compared to media that is action/visually dynamic.
Netflix may be looking at that 20% as a re
Re: (Score:2)
20% savings is believable. Pirates easily achieve that much without sacrificing video quality. Of course pirates do it by hand, eyeballing each scene and adjusting the bitrate and compression, so it's a lot of labor. Also it's almost an art form. It's incredible how much video quality a good pirate encoder can fit inside 700mb (to fit on a CD-R... this used to be the standard back in the day)
Of course Netflix can't hire teams of pirates to encode their billion hours of video, so they'd have to automate it.
Neat... but why? (Score:2)
Re: (Score:2)
Re: (Score:3, Interesting)
Actually I doubt they'd reduce the color precision from 24 bit (it's not 32 bit).
In fact, content with lots of synthetic content can sometimes be smaller by being 30 bit instead of 24 bit (think how often synthetic content puts in gradients, with 24 bit those gradients are more dithered than 30 bit, and the compression algorithms struggle a bit more with what appears to be 'noisy' content from dithering compared to less noisy undithered content).
Of course this is using general purpose algorithms that are us
Re: (Score:2)
Horse shit.
There is more "noise" (signal) in a 10-bpc source than an 8-bpc source OR an 8-bpc conversion of a 10-bpc source.
You just can't see it as easily with your eyes.
Re:Neat... but why? (Score:5, Insightful)
What problem is this trying to address?
Saving on bandwidth costs?
Providing a better streaming experience for customers on poor or throttled connections?
Storage space savings?
Getting the satisfaction of doing something better because why not?
Re: (Score:3)
So nothing that's REALLY important then?
So what would something REALLY important be?
Re: (Score:2)
So nothing that's REALLY important then?
So what would something REALLY important be?
Increased fidelity for rendering skin tones?
Re: (Score:3)
Improved sarcasm enhancement
Re:Neat... but why? (Score:5, Insightful)
Three words: Comcast data cap...
Peter.
Re: (Score:2)
I'm struggling with a data cap too, so I'm saving Netflix bandwidth by getting all my content on DVDs.
Re: (Score:3)
Haven't been to Dollar General lately, have you? They've been selling off the majority of their DVDs and not replacing damaged ones. I have movies that have been on my queue for years that are now (permanently) unavailable. I had a big backlog and now I don't really have a way to watch them - they're certainly not available for streaming. And they already put rental stores out of business. I would literally have to buy some of these movies to watch them.
Re: (Score:3)
Re: (Score:3)
Maybe better implementations of existing algorithms, but they have to tread carefully about new algorithms.
I assume today they use H264 and the re-encode would just be newer implementation of H264 encode/different settings than they used before.
They could get more from jumping to HEVC, but netflix is on crap tons of smart TVs and such.
Of course that's not to say they transcode to HEVC and client advertises whether it's H264 or HEVC and netflix just keeps both H264 and HEVC live on their CDN, if capacity is
Re: (Score:2)
Re: (Score:2)
90's sat tech (Score:3)
Back in the MPEG2 Sat days they regularly used different bit rates depending on content talking heads very little compared to full out for sports and action movies. An actual knowledgeable encoding tech can do wonders, higher quality source material can also do wonders.
Re: (Score:2)
Re: (Score:2)
It could make sense for highly viewed content. While the entire library is that big the highly viewed content not so much. Can guestimate from the their appliance has 288 raw TB or so that should be enough to fit all the high demand content.
Re: (Score:2)
288 TB should be enough for anybody
Re:90's sat tech (Score:4, Interesting)
Netflix probably aren't too keen on the idea of paying people to puzzle over what compression would best suit each and every item in their 1-Petabyte video library.
The summary says they spent four years developing the new approach. I suspect that paying people to puzzle over (in layman's terms: do research) how to improve the encoding across their Petabyte video library was exactly what they did.
Re: (Score:2)
Netflix probably aren't too keen on the idea of paying people to puzzle over what compression would best suit each and every item in their 1-Petabyte video library.
I think they would just encode each item with every compression algorithm they support, then spot check with humans to find the lowest-bandwidth option deemed "acceptable" for a given viewing device and style of video. It need not be the same algorithm for each device; an animated TV show going to a Wii or a cell phone might allow for a lower-bandwidth, lower-quality option than streaming to a Roku or Apple TV, for example, and both could be lower bandwidth than a Hollywood movie. Their cost isn't in stor
Re: (Score:2)
They still do this today. Cable companies even mess with the bitrates for different channels. An HD antenna can usually beat the re-compressed version you get over cable.
Multipass (Score:2)
Almost sounds to me like they have switched to multiple pass encoding, rather than a fixed quality/bandwidth setting.
"The new system will encode from the raw source material more intelligently, considering whether or not the material itself can really benefit from higher bit-rates, or whether identical quality can be maintained with less space and bandwidth."
Re: (Score:2)
That's a good WRONG. Love it.
Repetition (Score:4, Insightful)
They can save about 500% of my bandwidth by just letting me perma-download Family Guy, American Dad, and Buffy, which I keep watching over and over and over again.
Re: (Score:2)
They can save about 500% of my bandwidth by just letting me perma-download Family Guy, American Dad, and Buffy, which I keep watching over and over and over again.
I would love the ability to preload a fixed number of shows to a device even if it was for a time limited period.
Re:Repetition (Score:4, Funny)
They can save about 500% of my bandwidth by just letting me perma-download Family Guy, American Dad, and Buffy, which I keep watching over and over and over again.
Look into something called "Boxed Sets".
Re:Repetition (Score:4, Funny)
Look into something called "Boxed Sets".
So then .. looks like I'll have to buy the white album .. again.
Re: (Score:3)
If you really care about what you're watching and you're planning on watching it again, it really makes much more sense to just get the boxed set. Rip or not but you will still have it after it disappears from Netflix.
Plus you won't have to worry about the quality of the stream you're getting from Netflix or any other shenanigans they might pull with the original content.
You don't have to have the entire run of 200 series on your media server. So the HTPC option doesn't need to be too complicated. '-)
class action suit (Score:5, Funny)
Re:class action suit (Score:4, Funny)
Are you sure you haven't agreed to a binding arbitration agreement? >:-)
Re: (Score:2)
If you're a fan, why would you want to dumb-down My Little Pony?
Re: (Score:2)
Pony haters will hate.
Wasn't this the entire plot.... (Score:2)
Wasn't this the entire plot to that HBO series "Silicon Valley", where a group of geeks form a company to re-encode petabytes of porn?
I never really got to see the whole series, but it came on after Game of Thrones, so I saw the occasional episode.....
The Open Edge Content Delivery Network, perhaps? (Score:2)
What we need is the death of "bandwidth caps" (Score:3, Interesting)
I watch YouTube a lot, on average about 2-3 hours a day. As of late, I live in a country where there is a bandwidth cap of 40 GB/month. And I have no option but to YouTube at 144p to avoid extra bandwidth charges.
I applaud all efforts by tech companies to reduce bandwidth usage (and not to forget, making inter-webs more exciting). Then again, none of those efforts matter, if bandwidth caps are forcing consumers to use internet like back in 90s.
Re: (Score:2)
How else do you expect internet providers to limit people from using up all the available bandwidth? Even with a relatively slow connection of 5 Mbit/s, you could pull down 1.6 Terabytes every month if you left your connection running 24/7. And a 5 Mbit connection is probably slower than most people want in their homes. Because they want to be able to download a small number of things relatively quickly.
I'd much rather have a 100 Mbit connection that was limited to a certain number of gigabytes a month th
Re: (Score:2, Interesting)
GB limited bandwidth caps only serve to show the ISP is third tier at best. They obviously don't buy internet bandwidth in any normal fashion.
Why? A 40 GB cap just serves to tell people "Only use it for an hour or two at peak times when it is the most congested!". Now you need really fat pipes that sit unused most of the day.
A proper method to limit usage that will actually save the ISP money would be to figure out your 95th percentile peak hours (You know, the same way you're billed by the first tier pr
Re: (Score:2)
Or find another hobby.
Re: (Score:3)
I hear everything's up to date in Kansas City - they've gone about as far as they can go.
Netflix's catalog is already shrinking (Score:4, Funny)
With all their efforts concentrated on their original series, it seems like their movie and TV offerings already shrink every month already, without any compression.
Doesn't VBR already do this? (Score:2)
"The new system will encode from the raw source material more intelligently, considering whether or not the material itself can really benefit from higher bit-rates, or whether identical quality can be maintained with less space and bandwidth."
I thought existing VBR algorithms already account for the absence of interframe changes by reducing the effective bit rate for those frames.
Need a new TV (Score:2)
Re: (Score:2)
I know my current LG has already limited upgrade support for older SmartTV functions, so it's possible (likely) that my current TV will not support this upgrade. Crap.
That's why I have a "dumb" TV and a smart bd player. It's cheaper to upgrade a smart bd player, then pass the decoded content to my tv.
auto-play (Score:2)
They could save 20% of their bandwidth by having a way to disable the auto-play of the next episode.
Re: (Score:2)
I love the autoplay next episode option. I keep meaning to look for a plugin that does that with KODI.
Re: (Score:2)
Re: (Score:2)
There already a is a way to do this [netflix.com]. I'm pretty sure it's been available for quite a long time. Although I don't know why autoplay is on by default. Same goes for Youtube. Their's is even worse, as I can't find a way to disable it permanently. You can disable it for a session, but if you leave and come back the next day, then the setting is back on again.
Four Years! (Score:2)
They've taken 4 years to figure out that parameters need to be optimized per film to produce optimum results for that film, and then to re-encode some films. The decisions for new parameters apparently aren't being made automatically, it's human choice. They're just beginning to consider that they may want to change parameters dynamically as a film progresses. This could and should all have been done in 2 months.
They haven't invented a new codec technology; they haven't advanced the state of the art at all.
How's 4K streaming ever going to happen? (Score:2)
4K is not going to happen unless Comcast, TWC and the rest of the evil monopolies build pipes that can handle it, or they get some competition, which is not going to happen as long as crooks run the state and local governments.
Re: (Score:3)
4K is not going to happen unless [...]
It seems a bit odd to suggest it can't happen when 4K streaming is already happening. The main selling point of Netflix's top streaming tier is that it grants you access to their "Ultra HD" (i.e. 4K) catalog. YouTube has had 4K support since late 2013 and has quietly been adding support for even higher resolutions in the last two years. You can already find content available at resolutions as high as 8K (e.g. this video [youtube.com]). One of the production houses I follow on YouTube makes most of their animated content
How about offering a 720p option? (Score:2)
Re: (Score:2)
Why not? 0.7 vs 3 is a small step. I'd rather have them push higher quality 1080p (as in 10GB per hour) instead.
What about sound compression? (Score:2)
When are they going to fix their sound encoding?! I often have to switch to plain stereo from the default 5.1, because the higher-frequency is distorted which makes speech/dialog sound especially "tinny". Don't know if it's due to higher compression for 5.1 sound, or something else, but it annoys the hell out of me. I'm no audiophile either, so it is pretty apparent. Using PS3 hooked to a receiver, so it could be the PS3 client. Tried reporting this issue to Netflix, but there is no way to do it except by c
My two hours will look fabulous (Score:2)
Re: (Score:2)
RTFA. They claim no loss of quality so who cares about the bit rate.
Re: (Score:2)
Re: (Score:2)
Generally rec rooms/brorooms are wider/longer than the 3 to 4 feet that is the maximum viewing distance for 'decent' sized TV's, anything smaller than 60" and your sitting even closer to get any benefit...just another scam like 3D if you ask me. Don't get me wrong I'm sure it would benefit a small group but certainly not the majority so it's just another way to hopefully get the masses to replace their existing good TV's...what a waste.
I felt this way for the VHS to DVD conversion then even more so during the DVD to bluray conversion. I was completely wrong both times so I think I'll be shutting up during the jump to 4k and just look forward to enjoying it in 10 years when the price allows me to.
(I encode everything to 720p just so I won't start getting use to 1080p)
Re: (Score:2)
> WTF are you blabbering about!? I got a 49" 4K and I am sitting about 3+ meters (10 feet for muricans) away with no need to move closer.
> WTF watches TV from less than 2 meters?
People stuck in small apartments in Europe where there may not be 10 whole feet from one wall to the other. [snicker]
Re: (Score:2)