Follow Slashdot stories on Twitter

 



Forgot your password?
typodupeerror
Check out the new SourceForge HTML5 internet speed test! No Flash necessary and runs on all devices. ×
Data Storage Businesses Movies Entertainment IT

Netflix To Re-Encode Entire 1 Petabyte Video Catalogue In 2016 To Save Bandwidth (variety.com) 285

An anonymous reader writes: Netflix has spent four years developing a new and more efficient video-encoding process that can shave off 20% in terms of space and bandwidth without reducing the quality of streamed video. With streaming video accounting for 70% of broadband use, the saving is much-needed, although the advent of 4K streaming, higher frame rates and HDR are likely to account for it all soon after. Netflix video algorithms manager Anne Aaron explained to Variety that certain types of video benefit little from the one-size-fits-all compression approach that Netflix has been using until now: "You shouldn't allocate the same amount of bits for My Little Pony as for The Avengers."
This discussion has been archived. No new comments can be posted.

Netflix To Re-Encode Entire 1 Petabyte Video Catalogue In 2016 To Save Bandwidth

Comments Filter:
  • by Eunuchswear ( 210685 ) on Tuesday December 15, 2015 @11:01AM (#51122085) Journal

    "You shouldn't allocate the same amount of bits for My Little Pony as for The Avengers."

    So they're dropping the resolution for The Avengers?

    • damn, mod points just ran out. Best first post in a long, long time :D

    • Re: (Score:3, Funny)

      by Anonymous Coward

      Absolutely. I need to see every little pony in all its unpixelated, 4K glory.

      Wait... that didn't come out right.

    • Re:My little pony (Score:5, Interesting)

      by Kagato ( 116051 ) on Tuesday December 15, 2015 @11:51AM (#51122525)

      On a serious note, animated content is much harder for the 8-bit encoding. It's the hard edges with high contrast cell shading. You get a lot more compression artifacts than a typical movie. You can resolve this by using 10-bit encoding, but there's a lot of Netflix devices with embedding video codecs. They really can't change, and almost none of the chipsets out there support 10-bit decoding. So that leaves option two, which is to increase the bitrate.

      • by PRMan ( 959735 )
        This isn't true and the article goes into great depth about how much easier it is than Avengers.
        • Re:My little pony (Score:5, Informative)

          by marcansoft ( 727665 ) <hector&marcansoft,com> on Tuesday December 15, 2015 @12:33PM (#51122951) Homepage

          That animated content benefits from 10-bit encoding is true. That has less to do with hard edges and more to do with banding artifacts on flat shaded areas - TFA actually goes into that, mentioning soft focus and fog as producing hard-to-encode gradients, the same kind of gradients present in many kinds of animation and which would benefit from using 10-bit mode. Hard edges do tend to be hard to encode with typical video codecs too (but 10-bit probably won't help you there).

          However, My Little Pony isn't a particularly good example, because it's full of completely flat areas that are trivial to encode. It might take a higher quality setting than you might expect to look crisp, but at the end of the day, you're going to be spending fewer bits per frame on it than on The Avengers. Animation has its own set of encoding tradeoffs/challenges (which is why good encoders have presets tuned for animation).

      • by subk ( 551165 )

        animated content is much harder for the 8-bit encoding. It's the hard edges with high contrast cell shading.

        Nonsense. Animation is optimized for compression.. It's already dithered to a limited color set! Whole swathes of pixels get the same value. Sure, you might be able to SEE the compression.. But it is a breeze for the hardware, and bitrates can be dramatically reduced compared to the baseband 4-2-2 video.

      • How is modern animation done? If it's done on computers in some sort of vector drawing format then, unless the drawings are really complicated, it seems a little silly to convert the (lossless) vector drawings to a (lossy) video in the first place. Surely modern computers can render 2D vector drawings in real time? I assume that the vector information can be stored relatively compactly.

        FWIW, it sounds like some of the animation for My little pony is done with Flash [wikipedia.org], and much as I hate flash, a flash animati

        • by ceoyoyo ( 59147 )

          Sure, but it's awfully hard to protect. If they just spew the original vector drawings around the Internets, you could use them to make those ponies do whatever you want!

      • Re: (Score:3, Interesting)

        by Anonymous Coward

        I recode a lot of video (don't ask), and in my experience, animation is a lot easier on the resulting files for the same quality. The absolute worst you can do if you want small high quality files is "film grain", whether it is from a bad source or artificially added for artistic reasons. Second worst is a badly compressed source with lots of artifacts. Then there's video with lots of small objects moving across a detailed background. The hard contrasts at the edges of cell shaded videos are only problemati

    • by Falos ( 2905315 )
      I can't tell if she's saying the streams have different priorities or if they should be compressed with different codecs. Live action and animation act rather differently on screens. My understanding was jpgs are better suited to photographs and pngs for flatter images (ie website screencap, logos), so I imagine there's also best-choice temporal/spatial tech for video. And it shouldn't be too hard to write an algorithm to classify the strea- wait, they would already be tagged.

      Or maybe not, and she's bash
      • I can't tell if she's saying the streams have different priorities or if they should be compressed with different codecs.

        It seems both. I'd say the best way to save bandwidth is to reduce the popular titles first. All the latest compression technologies can't reduce the existing bandwidth used for watching Vanilla Sky.

  • by Viol8 ( 599362 ) on Tuesday December 15, 2015 @11:02AM (#51122107) Homepage

    Shaving 20% off seems pretty optimistic to me. Unless they've suddenly discovered some whole new realm of compression mathematics I'd be surprised if thats anything more than a peak compression in some rare edge cases.

    • by hawguy ( 1600213 ) on Tuesday December 15, 2015 @11:07AM (#51122153)

      Shaving 20% off seems pretty optimistic to me. Unless they've suddenly discovered some whole new realm of compression mathematics I'd be surprised if thats anything more than a peak compression in some rare edge cases.

      Sounds more like as a part of re-compression, they are going to drop the bitrate (and video quality?) for videos that don't "need" it:

      certain types of video benefit little from the one-size-fits-all compression approach that Netflix has been using until now: "You shouldn't allocate the same amount of bits for My Little Pony as for The Avengers."

      • Sounds more like as a part of re-compression, they are going to drop the bitrate (and video quality?) for videos that don't "need" it:

        Is it the bitrate that's being changed? I took the part about Pony vs Avengers to mean animation vs live action are better compressed by completely different algorythms.

        • by Anonymous Coward on Tuesday December 15, 2015 @11:20AM (#51122261)

          The article is actually in English, you know.

          "The new system will encode from the raw source material more intelligently, considering whether or not the material itself can really benefit from higher bit-rates, or whether identical quality can be maintained with less space and bandwidth."

        • by adolf ( 21054 )

          The Avengers is not animated?

        • by Junta ( 36770 )

          They suggest that they are going to have an automated process to transcode content a few ways and use some automated quality check to decide what meets the threshold. Or they are trying to make changing from CBR to VBR sound more impressive than it is.

    • by c0d3g33k ( 102699 ) on Tuesday December 15, 2015 @11:11AM (#51122181)

      I suspect that by the time of this announcement they have already done the testing, so have a good idea of how much they can optimize. From the article, it's more about optimizing compression parameters to fit the source material rather than taking a one-size-fits-all approach.

      • by jedidiah ( 1196 )

        They might also be able to employ different and better options on the transcoder. When I first started transcoding my own stuff I optimized for quality and total processing time. The files were meant for HTPC use and mobile video devices weren't terribly common yet. What I ended up with was something that early iDevices couldn't even handle (although Archos could).

        They may simply be adjusting for more modern and more capable devices.

    • by JackieBrown ( 987087 ) <dbroome@gmail.com> on Tuesday December 15, 2015 @11:26AM (#51122305)

      I am in the process of moving my fiancee's dvd/bluray collection to my server and putting her physical copies in storage. Using Handbrake, switching from x264 to x265 saves me at lease 10 % on dvd sources and closer to 30+% on the bluray sources.

      • by bagofbeans ( 567926 ) on Tuesday December 15, 2015 @12:05PM (#51122687)

        I remember ripping my CD collection to ogg, only have to do it again years later to flac when space got cheaper. The ogg was fine, but not a good source for re-encoding to another format such as mp3.

        If I was going to rip movies, I'd keep the original streams. You'll never spare the time again to re-rip, even if you you think now that you will.

      • by PRMan ( 959735 )
        But Netflix can't do this since most of the hardware consuming their movies has an x264 chip built in.
      • And now you have x265 that's not hardware accelerated on anything but the most modern GPUs (and even then, only partially [anandtech.com] -- certainly not suitable for any set-top-box, tablet or mobile phone. Heck, even a laptop that has partial GPU supporting (or none) will burn through tons of battery watching it on a flight with no power plugs.

        Netflix has to support all those platforms (and probably worse ones) -- and then you enter the idea of having multiple copies for every asset :-(

        • I'm not using very aggressive encoding settings and my google player box has been able to play all of the movies I ripped so far (I know it's not hardware accelerated but as long as it plays without stuttering, it makes no difference to me.)

          Guess I'll double check my amazon fire (not the 4k model) and make sure it's able to play those too.

        • My smart TV can play HEVC content just fine (using the web browser). I doubt it could do it without any hardware acceleration.
          Most smart TVs use underpowered smartphone chips.

    • by PRMan ( 959735 )
      They have. Mathematically, My Little Pony doesn't need 5.8 Mbps for 1080p. It can be done at about 1.5 Mbps because of the simple color palette.
    • my guess is that cartoons, especially older cartoons have large parts of the frame with the same color which can be compressed more easily. or they will simply stream cartoons at lower bitrates
    • by jensend ( 71114 )

      On the contrary, I'm surprised they're only saving 20%. Look at the difference between VP9 or X265 and the VC-1 encoder they used exclusively for their first few years. And given the costs they incur using all this bandwidth they could definitely throw a good number of top quality engineers at figuring out a rate control algorithm that's more suited to their rather unusual use case.

    • I think its more a matter of (potentially) saving 20% versus what they are using now.

      Didn't read the whole blurb, but they are probably going full-in on HEVC ( https://en.wikipedia.org/wiki/... [wikipedia.org] )

      The Wikipedia article mentions 50-60% savings using HEVC over H.264 depending on resolution, but chances are these are optimum subjective results. My Little Pony (and other animated shows) would probably encode much tighter compared to media that is action/visually dynamic.

      Netflix may be looking at that 20% as a re

    • 20% savings is believable. Pirates easily achieve that much without sacrificing video quality. Of course pirates do it by hand, eyeballing each scene and adjusting the bitrate and compression, so it's a lot of labor. Also it's almost an art form. It's incredible how much video quality a good pirate encoder can fit inside 700mb (to fit on a CD-R... this used to be the standard back in the day)

      Of course Netflix can't hire teams of pirates to encode their billion hours of video, so they'd have to automate it.

  • So they should be able to do this without effecting quality due to better compression algorithms being available now, but what problem is this trying to address? Other then keeping that one intern they don't like locked in a small room running HandBreak 24/7.
    • Look at the comparison. I'm willing to bet a large amount of their audience happen to be watching cartoon, or anime. When your color pallet is reduced you don't need to save the entire thing in 32bit color to get the same exact quality. Some will be shaved, and others wont be shaved that much.
      • Re: (Score:3, Interesting)

        by Junta ( 36770 )

        Actually I doubt they'd reduce the color precision from 24 bit (it's not 32 bit).

        In fact, content with lots of synthetic content can sometimes be smaller by being 30 bit instead of 24 bit (think how often synthetic content puts in gradients, with 24 bit those gradients are more dithered than 30 bit, and the compression algorithms struggle a bit more with what appears to be 'noisy' content from dithering compared to less noisy undithered content).

        Of course this is using general purpose algorithms that are us

        • Horse shit.
          There is more "noise" (signal) in a 10-bpc source than an 8-bpc source OR an 8-bpc conversion of a 10-bpc source.
          You just can't see it as easily with your eyes.

    • by c0d3g33k ( 102699 ) on Tuesday December 15, 2015 @11:16AM (#51122223)

      What problem is this trying to address?

      Saving on bandwidth costs?
      Providing a better streaming experience for customers on poor or throttled connections?
      Storage space savings?
      Getting the satisfaction of doing something better because why not?

    • by loony ( 37622 ) on Tuesday December 15, 2015 @11:30AM (#51122343)

      Three words: Comcast data cap...

      Peter.

    • by Junta ( 36770 )

      Maybe better implementations of existing algorithms, but they have to tread carefully about new algorithms.

      I assume today they use H264 and the re-encode would just be newer implementation of H264 encode/different settings than they used before.

      They could get more from jumping to HEVC, but netflix is on crap tons of smart TVs and such.

      Of course that's not to say they transcode to HEVC and client advertises whether it's H264 or HEVC and netflix just keeps both H264 and HEVC live on their CDN, if capacity is

    • saving bandwidth and saving them money from buying new CDN servers. they are probably at the point where a lot of their servers are running at capacity and it's cheaper to lower the bitrate than buy new servers, routers and pay for the installation and everything that goes along with it
    • by pnutjam ( 523990 )
      Pied Piper strikes again? Hulu just brought piss to a shit fight.
  • by silas_moeckel ( 234313 ) <silas AT dsminc-corp DOT com> on Tuesday December 15, 2015 @11:06AM (#51122143) Homepage

    Back in the MPEG2 Sat days they regularly used different bit rates depending on content talking heads very little compared to full out for sports and action movies. An actual knowledgeable encoding tech can do wonders, higher quality source material can also do wonders.

    • Netflix probably aren't too keen on the idea of paying people to puzzle over what compression would best suit each and every item in their 1-Petabyte video library.
      • It could make sense for highly viewed content. While the entire library is that big the highly viewed content not so much. Can guestimate from the their appliance has 288 raw TB or so that should be enough to fit all the high demand content.

      • Re:90's sat tech (Score:4, Interesting)

        by c0d3g33k ( 102699 ) on Tuesday December 15, 2015 @11:46AM (#51122469)

        Netflix probably aren't too keen on the idea of paying people to puzzle over what compression would best suit each and every item in their 1-Petabyte video library.

        The summary says they spent four years developing the new approach. I suspect that paying people to puzzle over (in layman's terms: do research) how to improve the encoding across their Petabyte video library was exactly what they did.

      • Netflix probably aren't too keen on the idea of paying people to puzzle over what compression would best suit each and every item in their 1-Petabyte video library.

        I think they would just encode each item with every compression algorithm they support, then spot check with humans to find the lowest-bandwidth option deemed "acceptable" for a given viewing device and style of video. It need not be the same algorithm for each device; an animated TV show going to a Wii or a cell phone might allow for a lower-bandwidth, lower-quality option than streaming to a Roku or Apple TV, for example, and both could be lower bandwidth than a Hollywood movie. Their cost isn't in stor

    • They still do this today. Cable companies even mess with the bitrates for different channels. An HD antenna can usually beat the re-compressed version you get over cable.

  • Almost sounds to me like they have switched to multiple pass encoding, rather than a fixed quality/bandwidth setting.

    "The new system will encode from the raw source material more intelligently, considering whether or not the material itself can really benefit from higher bit-rates, or whether identical quality can be maintained with less space and bandwidth."

  • Repetition (Score:4, Insightful)

    by Impy the Impiuos Imp ( 442658 ) on Tuesday December 15, 2015 @11:15AM (#51122209) Journal

    They can save about 500% of my bandwidth by just letting me perma-download Family Guy, American Dad, and Buffy, which I keep watching over and over and over again.

    • by OzPeter ( 195038 )

      They can save about 500% of my bandwidth by just letting me perma-download Family Guy, American Dad, and Buffy, which I keep watching over and over and over again.

      I would love the ability to preload a fixed number of shows to a device even if it was for a time limited period.

    • by c0d3g33k ( 102699 ) on Tuesday December 15, 2015 @11:59AM (#51122611)

      They can save about 500% of my bandwidth by just letting me perma-download Family Guy, American Dad, and Buffy, which I keep watching over and over and over again.

      Look into something called "Boxed Sets".

      • by OzPeter ( 195038 ) on Tuesday December 15, 2015 @12:04PM (#51122671)

        Look into something called "Boxed Sets".

        So then .. looks like I'll have to buy the white album .. again.

        • by jedidiah ( 1196 )

          If you really care about what you're watching and you're planning on watching it again, it really makes much more sense to just get the boxed set. Rip or not but you will still have it after it disappears from Netflix.

          Plus you won't have to worry about the quality of the stream you're getting from Netflix or any other shenanigans they might pull with the original content.

          You don't have to have the entire run of 200 series on your media server. So the HTPC option doesn't need to be too complicated. '-)

  • by sunking2 ( 521698 ) on Tuesday December 15, 2015 @11:18AM (#51122239)
    As a My Little Pony enthusiast who pays the same per month as everyone else I demand the same quality as the Avengers.
  • Wasn't this the entire plot to that HBO series "Silicon Valley", where a group of geeks form a company to re-encode petabytes of porn?

    I never really got to see the whole series, but it came on after Game of Thrones, so I saw the occasional episode.....

  • TOECDN would have fixed this shit already. We don't want less quality, we want more!
  • by nerdyalien ( 1182659 ) on Tuesday December 15, 2015 @11:23AM (#51122277)

    I watch YouTube a lot, on average about 2-3 hours a day. As of late, I live in a country where there is a bandwidth cap of 40 GB/month. And I have no option but to YouTube at 144p to avoid extra bandwidth charges.

    I applaud all efforts by tech companies to reduce bandwidth usage (and not to forget, making inter-webs more exciting). Then again, none of those efforts matter, if bandwidth caps are forcing consumers to use internet like back in 90s.

    • How else do you expect internet providers to limit people from using up all the available bandwidth? Even with a relatively slow connection of 5 Mbit/s, you could pull down 1.6 Terabytes every month if you left your connection running 24/7. And a 5 Mbit connection is probably slower than most people want in their homes. Because they want to be able to download a small number of things relatively quickly.

      I'd much rather have a 100 Mbit connection that was limited to a certain number of gigabytes a month th

      • Re: (Score:2, Interesting)

        by Anonymous Coward

        GB limited bandwidth caps only serve to show the ISP is third tier at best. They obviously don't buy internet bandwidth in any normal fashion.

        Why? A 40 GB cap just serves to tell people "Only use it for an hour or two at peak times when it is the most congested!". Now you need really fat pipes that sit unused most of the day.

        A proper method to limit usage that will actually save the ISP money would be to figure out your 95th percentile peak hours (You know, the same way you're billed by the first tier pr

  • by NotDrWho ( 3543773 ) on Tuesday December 15, 2015 @11:25AM (#51122299)

    With all their efforts concentrated on their original series, it seems like their movie and TV offerings already shrink every month already, without any compression.

  • From the article:
    "The new system will encode from the raw source material more intelligently, considering whether or not the material itself can really benefit from higher bit-rates, or whether identical quality can be maintained with less space and bandwidth."

    I thought existing VBR algorithms already account for the absence of interframe changes by reducing the effective bit rate for those frames.
  • I know my current LG has already limited upgrade support for older SmartTV functions, so it's possible (likely) that my current TV will not support this upgrade. Crap.
    • I know my current LG has already limited upgrade support for older SmartTV functions, so it's possible (likely) that my current TV will not support this upgrade. Crap.

      That's why I have a "dumb" TV and a smart bd player. It's cheaper to upgrade a smart bd player, then pass the decoded content to my tv.

  • They could save 20% of their bandwidth by having a way to disable the auto-play of the next episode.

    • I love the autoplay next episode option. I keep meaning to look for a plugin that does that with KODI.

    • by Bengie ( 1121981 )
      I wish they allowed you to continuously auto-play. They have a limit on how many episodes right now. Many times we have Netflix on just for background noise as I read books or the wife does her thing. Instead, we need to either use Hulu or actually turn on the TV.
    • There already a is a way to do this [netflix.com]. I'm pretty sure it's been available for quite a long time. Although I don't know why autoplay is on by default. Same goes for Youtube. Their's is even worse, as I can't find a way to disable it permanently. You can disable it for a session, but if you leave and come back the next day, then the setting is back on again.

  • They've taken 4 years to figure out that parameters need to be optimized per film to produce optimum results for that film, and then to re-encode some films. The decisions for new parameters apparently aren't being made automatically, it's human choice. They're just beginning to consider that they may want to change parameters dynamically as a film progresses. This could and should all have been done in 2 months.

    They haven't invented a new codec technology; they haven't advanced the state of the art at all.

  • 4K is not going to happen unless Comcast, TWC and the rest of the evil monopolies build pipes that can handle it, or they get some competition, which is not going to happen as long as crooks run the state and local governments.

    • 4K is not going to happen unless [...]

      It seems a bit odd to suggest it can't happen when 4K streaming is already happening. The main selling point of Netflix's top streaming tier is that it grants you access to their "Ultra HD" (i.e. 4K) catalog. YouTube has had 4K support since late 2013 and has quietly been adding support for even higher resolutions in the last two years. You can already find content available at resolutions as high as 8K (e.g. this video [youtube.com]). One of the production houses I follow on YouTube makes most of their animated content

  • Instead of just giving the user the choice to watch at crap 480p quality or use ALL THEIR BANDWIDTH, how about a 720p option? The bandwidth jumps from .7 MB per hour with SD to 3 GB or more per hour for the next option. Why?
    • Why not? 0.7 vs 3 is a small step. I'd rather have them push higher quality 1080p (as in 10GB per hour) instead.

  • When are they going to fix their sound encoding?! I often have to switch to plain stereo from the default 5.1, because the higher-frequency is distorted which makes speech/dialog sound especially "tinny". Don't know if it's due to higher compression for 5.1 sound, or something else, but it annoys the hell out of me. I'm no audiophile either, so it is pretty apparent. Using PS3 hooked to a receiver, so it could be the PS3 client. Tried reporting this issue to Netflix, but there is no way to do it except by c

  • I look forward to watching my six hours of Netflix every month. Good thing that Comcast lets me watch as many of their movies as I want on Hexfinicky.

Trying to be happy is like trying to build a machine for which the only specification is that it should run noiselessly.

Working...