'Hobbit' Creates Big Data Challenge 245
CowboyRobot writes "In the past five years there has been an 8x increase in the amount of content being generated per every two-hour cinematic piece. Although 3D is not new, modern 3D technologies add from 100% to 200% more data per frame. In 2009, Avatar was one of the first movies to generate about a petabyte of information. The Hobbit: An Unexpected Journey was shot in a new digital format called High Frame Rate 3-D, which displays the movie at 48 frames per second, twice the standard 24-fps rate that's been in place for more than 80 years."
But with digital storage transcending some other limitations of conventional projection techniques, it's not just framerate that directors are now able to play with more easily; it's the length of movies themselves, which stats suggest just keep getting longer.
How big was the hobbit? (Score:5, Interesting)
I read TFA, and nowhere does it say how big The Hobbit was.. only that Avatar was about a Petabyte. Why isn't this stated anywhere? It's very frustrating, and also makes the article less useful, since its entire premise is that "The Hobbit creates big data challenge" with no specificity.
Re:How big was the hobbit? (Score:5, Funny)
According to the torrent sites; 2.32 GB though they do use the lossy video camera conversion...
Re: (Score:3, Funny)
For some reason I find this observation hilariously funny...
Re: (Score:2)
thats a cam version
I don't bother downloading pirated movies so I gotta ask - Do people *really* download and watch movies that have been filmed in the theater with a camcorder? How the smeg is that even watchable? Sounds absolutely horrid.
Re: (Score:3)
Re: (Score:3)
Re: (Score:3)
I did it once, and actually watched all the way through. Before the second Evangelion movie came out in the US, it was only available as a cam version with fansubs. Once it came out in the US, I bought the DVD and deleted the crappy cam.
Re: (Score:3)
Do people *really* download and watch movies that have been filmed in the theater with a camcorder?
Yep, I know people that do.
I know it because they tell me how proud they were that they did it: "I downloaded The Hobbit the other day! Do you want a copy?"
Also people with kids who can't afford the time/money to take their kids to see every goddam movie but their kids can say they've seen it. etc.
Not everybody is a video/audiophile. They just want the latest stuff.
Re: (Score:3, Informative)
Well, I can't find an official number, but we can estimate using data from here [wikipedia.org] and here [imdb.com]:
From the first link, which says the max data rate is 250Mbps, and doubling that to account for HFR, we have a 500Mbps data rate. Multiply that by the 169 minute running time and you get
500e6 bit/sec x 1/8,589,934,592 GB/bit [google.com] x 169 min x 3600 sec/min = 35,400 GB
(assuming the limit on precision is the running time at three significant digits).
Divide that by 1024 GB/TB and you have about 34.6 TB. Not impossible to set up
Wikipedia to the rescue (Score:5, Funny)
Re: (Score:2)
That petabyte must be without any compression. The Hobbit (HFR, 3D) as used by a digital cinema projector still fits on a 500GiB hard drive.
Re:How big was the hobbit? (Score:4, Informative)
Actually, it's all the data required as input to make the *final* frames. We're talking many layers of video at 32 bit per channel (128bit images), VFX cache data which can be GBs per second of footage, thousands of textures that are also GBs in size, point clouds... All of that is meant to retain a maximum amount of flexibility before finalizing the footage. Read up on the REYES pipeline for detailed info.
Disclaimer: Film and animation professional and professor.
Re:How big was the hobbit? (Score:4)
Having the additional resolution helps preserve quality during the making process. For example, most music is recorded at least at 24bit 48-96khz per track (dozens to hundreds depending on the music), even tho the destination will usually be 16bit 44-48khz. The extra fidelity makes all the eventual filtering, dithering, resampling, correction, compression, etc ultimately that much higher in quality.
Disclaimer: worked in TV and music production.
Re: (Score:3)
Comment on Movie length (Score:5, Interesting)
Am I the only one who longs for the return of an intermission? If only for a little relief rather than ducking out for 3-4 minutes and missing that one important little line of dialogue on which the whole thing pins?
Re: (Score:2)
Apparently not the only one:
http://hardware.slashdot.org/comments.pl?sid=3355921&cid=42463785 [slashdot.org]
Re:Comment on Movie length (Score:5, Funny)
Re: (Score:3)
Oh, it's an app for the phone?
At first I thought about something like a hose and a plastic bag [shieldhealthcare.com].
Re:Comment on Movie length (Score:4, Funny)
Yeah, but you wouldn't believe the compression algorithms necessary to let a phone replace the plastic bag..
Re: (Score:2)
It's not girls who spit?
Re:Comment on Movie length (Score:4, Insightful)
People who use phones in the theater should be beaten with rubber hoses and then forced to watch Barney.
You aren't using it in the theater (Score:2)
You activate the app before the movie starts, it vibrates in your pocket st the start of "pee friendly" zones.
Re: (Score:2)
So it can be silent, and you don't need to turn on the screen and distract anyone. So it's possible it could be used responsibly.
Re:Comment on Movie length (Score:4, Insightful)
Re: (Score:3)
I kept waiting for Peter Jackson to put down his X-Box controller and get on with the movie
Thats brilliant! A more concise assessment of the film could not be had. We can only hope the next two films have less of a premature ejaculation vibe to them.
Re: (Score:3)
An intermission would give me just enough time to think seriously about the horrible decision I've made and how hours of my life would be better spent by going home for a beer and a book.
May I suggest The Hobbit? At 1.4Mb (including the illustrations) the .epub would fit on a floppy and, I suspect, still end up saying more than Jackson's multi-petabyte trilogygasm.
"Why, I feel all thin, sort of stretched, if you know what I mean: like butter that has been scraped over too much bread." - Bilbo Baggins.
Re: (Score:2)
When I lived in Germany the movies had about a 10 - 15 intermission. They'd come around with snack trays like venders at a ball game here in the states. Even for 90 minute comedies there were intermissions. It was wonderful during longer movies. And probably made the theaters more money as people would use the restroom and usually by another beer, soda, snacks (the concept of free refills doesn't really exist in Europe).
Re: (Score:2)
We have a few dine in theaters here in Florida that do that too. They're rare (and more expensive per person) but it's a nice change from the standard theater style
Re:Comment on Movie length (Score:4, Funny)
Re: (Score:3)
Coffee in Scandinavia is often with free refills and sometimes free altogether, especially with meals. And much, much stronger[*] than what Americans are used to, so I'm not surprised you didn't want a refill.
Soda, on the other hand, is typically served in much smaller glasses, often without ice unless you ask for ice, and no refills. Scandinavians just drink a lot more coffee than soda, unlike over here.
Kastrup airport outside Copenhagen was, incidentally, the first place I went where I got charged for a
Re: (Score:2)
Re: (Score:2)
Yes please.
One of the reasons I don't buy concessions is because if I do, I'll want a drink. If I drink something, I'll have to use the restroom during the movie.
I there were an intermission, I would buy those overpriced snacks.
Re: (Score:2)
Re: (Score:2)
My copy of "Doctor Zhivago" on DVD is on two discs, because it didn't fit on one. When it gets to the end of the first disc it plays the original theatrical intermission card on a loop. I just wish the DVD allowed for it, like buffering the scene so it could keep playing it while you changed disc.
Bring back the intermission. (Score:5, Insightful)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
The Cooper had not one, but TWO concession stands IN the theater at the sides, totally unobtrusive, it was glorious. 800+ seats, Cinerama (146-degree arc) screen.
I saw one of the last movies shown there - re-mastered Lawrence of Arabia.
I sat at the absolute focal point of the screens, having skipped my morning classes to catch that. I was one of 5 people in the theater.
http://cinematreasures.org/theaters/930 [cinematreasures.org]
Re: (Score:2)
Yes it was. I love the intermission during 2001, there's just something about a pause in the story accompanied by Ligeti that's just surreal.
it's the length of movies themselves (Score:4, Interesting)
which stats suggest just keep getting longer"
And in the Hobbit's case, longer, and longer, and...... just waay too long. LOTR movies had 1000 pages of book to fill them with interesting content. Hobbit, not so much. In many of the scenes you can almost feel the director guy just out of camera view making that "stretch" motion with his hands.
Re:it's the length of movies themselves (Score:4, Informative)
How many times do we need to see Goblins getting knocked off wooden plank bridges by dwarves with a pole?
Not enough it seems.
Re: (Score:3)
No one edits a dwarf!
Re: (Score:2)
If it was shorter you would be bitching about all the things he skipped.
I loved the movie and I wish LOTR would have been done in a similar style. This is how you convert a book into a movie.
Re: (Score:2)
Re: (Score:2)
"The Hobbit" needs a fan edit to bring it below the two hour mark. This should be easy for part 1, though the real editing challenge would be to do it for the entire trilogy. Tricky, but possible, because it's not a long book.
I much preferred the LotR approach of releasing shorter versions to theaters and then releasing long versions on DVD for dedicated fans.
Re: (Score:2)
A fan edit to shorten it?
Sounds like the opposite of fan to me, if you want to shorten it.
If your attention span is so short how did you ever make it through the book?
Re: (Score:3)
Well, I thought I was a Peter Jackson fan, but I guess I'm not, since real fans don't criticise.
I don't recall getting bored during the book at all. But I was bored during the film. It really dragged on. It's not so much the plot development and the story - those are fine. It's the action sequences. They are repetitive and interminable. Some of them could be cut out completely, while others could be significantly shortened, and the film would be better for it. There is a tradition of "fan edits" that make
Re: (Score:2)
If you want it to cover every detail you might have too.
Re: (Score:2)
Works converted from one media to another are called adaptations because they're supposed to adapt, not carbon-copy, the work, because different medias have different strengths. This means that some parts get expanded, others cut, and the viewpoint is often different. A director who wants to cover every detail is not doing his job.
Re: (Score:3)
Battlefield Earth (Score:2)
I mean they did the 1000 page Battlefield Earth in under 2 hours in film and that turned out great...
Look at even LOTR, special editions probably make it a 12 hour film for the same page count.
Re: (Score:3, Insightful)
I mean they did the 1000 page Battlefield Earth in under 2 hours in film and that turned out great...
"Battlefield Earth" + "turned out great" in the same sentence?... Travolta, is that you?
Re: (Score:2)
Except the Battlefield Earth movie covered less than half of the book.
Re: (Score:2)
Re:it's the length of movies themselves (Score:4, Funny)
Granted, the special effects were well done, but "a decent story"? All I remember was this:
WARNING: SPOILER ALERT!!!
Some dwarves, a hobbit and a wizard go on an "adventure". They get into trouble, and then the wizard saves them. Then they get into trouble again, and the wizard saves them again. Then they get into trouble again, and the wizard summons some really big birds to save them again.
I still don't understand why they didn't just take the birds from the start, and all the way to the end. It would've saved a lot of trouble, not to mention hard disk space.
Technology is fine... (Score:3)
Re: (Score:2)
... and has replaced a light-hearted adventure story with the dark themes from LotR.
You mean like the changes Tolkien himself made and wanted to make? http://en.wikipedia.org/wiki/The_Hobbit#Revisions [wikipedia.org]
Re: (Score:2)
The more I see this debte, the more I realize much of LOTR was Tolkien's personal "fan fiction". It was stuff he wrote down for fun.. Mostly to practice applying "archeology" skills for languages and myths. Languages and history is what he taught at university, tracing back origins of languages and myths to the events and people that it happened to. LOTR was some fun working "backwards" with the same principals to build his personal LARP.
I begging to agree with George R R Martin in that he's going to wrap
Re: (Score:3)
The trolls talking about turning to stone in daylight I just wrote off as Jackson assuming not everyone knew that bit of lore - it's not a terrible assumption. Without there would be some people going "what just happened?"
Re: (Score:2)
completely gutted
What, changing a few lines of dialogue to make the transition from book to screen easier? That's gutting?
Adding some scenes that tell the backstory of the Oakenshields? That's gutting?
Bringing in some canon characters to make the story a better prequel to LOTR, entirely consistent with the canon of the milieu? That's gutting?
The only element that was out of place was having Azog gallavant all over the place chasing Thorin, but still, the basic conflict is entirely consistent [wikia.com] with canon.
Re: (Score:2)
Well *obviously* authors are allowed to gut their own stuff, because it's *theirs*. Unless it's George Lucas.
I don't really find any of Jackson's changes all that galling. Some of the additions are a bit "why?" but most are "fair enough".
Re: (Score:2)
I didn't mind him trying to make the Hobbit part of the LOTR cycle. it wasn't originally written that way, but as the other reply stated, Tolkien wanted to do that too. in fact, some of the dialogue changes he made when the book was re-released DID change certain things to make the Hobbit fit better with the later books. And I guess he pulled in some of the pieces of The Quest of Erebor and the Similarrion (sp?) to try to tie things together. But really, the main difference is that The Hobbit was written a
I love long films if... (Score:3, Interesting)
I'm a HUGE Tolkien fan, and went to the LOTR Extended Version Trilogy Marathon recently before seeing The Hobbit.
I was surprised at how well the longer versions of the films held up, after not watching them for around five years.
However, The Hobbit film was a let down on several levels, most of which I won't go into here. My main complaint? You do not need three films to tell the story. PJ has thrown in everything but the kitchen sink into The Hobbit, and it drags. Even the uber-videogame-esque "escape from the Goblins" scene drags... Too much of a good thing can ruin a film.
I would also say the same thing about the last Batman film. Too long and drawn out. Scenes that should be edited or removed alltogether. Thats why they call it the Directors Cut!
It makes me wonder if there aren't people involved in the film such as producers or editors who tell guys like PJ or Nolan, "hey bro, you might want to trim things down, just a smidge... You know, just to kind of keep the flow of the film going"
Re: (Score:2)
Even the uber-videogame-esque "escape from the Goblins" scene drags
Ironic, isn't it? By adding more action, the movie became less riveting. You don't need a threat of violence to keep a movie entertaining (not that there was a threat, since one guy got smashed full-force in the face with a mace and remained conscious).
For me, one of the most attention-holding scenes was at the beginning, when the dwarves ate, sang, and cleaned dishes. That was a party I'd love to hang out with. The farther they got from Tolkien in telling the story, the weaker the story became. The reven
Re: (Score:2)
Ironic, isn't it? By adding more action, the movie became less riveting.
Good point (The Matrix: Reloaded immediately comes to mind as the posterchild movie in this regard).
Also, it is a little odd to see some people complain about too long of a runtime when most of the complaints about the original LOTR movies was that too much was cut out of the story (Tom Bombadil, Scouring of the Shire, etc.).
Re: (Score:2)
Also, it is a little odd to see some people complain about too long of a runtime when most of the complaints about the original LOTR movies was that too much was cut out of the story (Tom Bombadil, Scouring of the Shire, etc.).
Yeah, I don't think the problem with the Hobbit is the long runtime, the problem is the runtime was extended by adding things that were made up out of nowhere.
If they'd stayed true to the story (or even true to the Tolkien universe) no one would mind a long runtime.
Re: (Score:2)
I think most people would be fine with the length of the Hobbit if it only included stuff that actually needed to be in the movie.
It all depends on how much material they actually have to work with. I don't want to watch a 2 hour movie stretched in to 3, and I don't want to watch a 3 hour movie cut down to 2. The Hobbit was the former.
Re: (Score:2)
I appreciated Batman Rises much more after the second watching from home.
As for scene cuts, a movie this large where scenes cost millions of dollars to shoot / produce, you can damn well bet that the movie was story boarded to death before a single roll of film was shot. Its not like the old days were you can just randomly have great actors run on a scene longer, or just make up an extra dialogue inter-cut and then drop it depending on how well it fit in with the rest of the movie. Hell it happens, but ulti
Re: (Score:2)
Yea, it's too bad, because Nolan is a brilliant director. I mean, "Inception" is the best sci-fi I've seen in years. It's hard for me to square it really, because I love most of his films.
But "The Dark Knight Rises", with Bane and his annoying and non-understandable heavily accented blabber via too much audio effects to the police being sent notes via the sewer locked underground, the whole thing was a mess, a much too long mess.
Re: (Score:2)
Raw Data (Score:2)
The Hobbit was also shot (and maybe shown?) at 4K resolution.
That's another bump in the data size.
Render time still a limitation (Score:2)
At the London preview screening Peter Jackson said that because 48fps + 3d is 4x the frames it's taken longer to render and the last scene with the coins was only finished a couple days before the premiere. He did mention the complexity in moving coins though
This is news? (Score:2)
I wonder why this is news... I guess it's a good time to invest in ENTERPRISE level storage!!
The Hobbit was shot in 5k resolution, and in "true 3D" with two cameras on every mount. And shot at 48fps (5k x 48 fps x 2 cameras) is MASSIVE film stock. Add multiple shoots for setup, testing lights, and then the actual acting. Not to mention all the digital elements that have to be stocked at high resolution as well.
Yup that's a LOT of data. But then so was Star Wars. I suppose with those really long copyrights
Not a Big Data Problem (Score:5, Insightful)
How does this have anything to do with Big Data? Storing large amount of data isn't the important part, it is being able to analyze that data. You do not analyze a movie's data file. You just load and display the movie, which can easy be stored in one large continous file. A Big Data problem would be Netflix trying to determine what kinds of movies to recommend, not storing and then displaying a long movie to users.
Re: (Score:2)
Tolkien... (Score:2)
Tolkien's work is love-it-or-hate-it and unfortunately I fall squarely on the "hate-it" side. I guess it's good to know that we can enjoy hours of tedium at a higher-than-normal frame rate, though.
Are they getting longer? (Score:2)
Call me when the average time of a Disnet movie is 150 minut
Re: (Score:2)
Suck it up big data (Score:2)
Everybody wants to go to digital distribution, but I refuse to accept compromise for the experience. I want uncompressed sound and stunning visual clarity in my movies, not some overly compressed barely HD content with stereo sound split to 5.1 false channels.
Everybody wants to move to the cloud but I live in a G8 country where my bandwidth is throttled and still stuck at 20th century download speeds and upload speeds that are barely better then dial-up.
So yes, the next big challenge for big data is to del
Re: (Score:2)
Re:Smart play by the studios (Score:4, Funny)
Re:Smart play by the studios (Score:5, Funny)
I know people who are still downloading it, you insensitive clod!
Re: (Score:3)
I just decided to settle for the compressed version [imdb.com]. Much smaller file size and I don't think it degraded the quality appreciably.
Re:Smart play by the studios (Score:4, Informative)
Huh? Most of the stuff you find on the various pirate channels is compressed down - at the most, you'll get a raw Blu-Ray rip. You can still get an xvid (avi usually under 2GB) version of just about anything that is available in any other format.
Re:Smart play by the studios (Score:5, Informative)
The standard pirate sizes ate 700MB for a recompressed DVD-rip and 4.4GB for a recompressed blu-ray rip. These sizes are used because they are just small enough to fit onto a CD-R and DVD-R respectively. It's not universal though. Especially long or difficult films might go up to 8GB in HD, and there has been a recent trend towards smaller files where quality wouldn't be compromised rather than just assuming media-size for everything.
Re: (Score:3)
Private trackers that I have access to offer full-frame blu-rays which are between 15 and 45 GB in size (or close to that). Using metropolitan P2P connections, I can download those monsters in 1-2 hours, depending on size. Movies which I download and manage to watch entirely deserve me buying a cinema ticket which I don't use. I buy the cinema ticket online, I don't go, everyone's happy. Rare are the movies I watch more than once, and those I usually buy as a hard-copy anyway.
Re:Smart play by the studios (Score:4, Interesting)
That's only relevant for the first copy.
Beyond that, content will be heavily re-compressed. It doesn't matter if it's Apple pushing the bits or The Pirate Bay.
Re:Smart play by the studios (Score:5, Informative)
Re: (Score:3)
Even for 48fps 3D, it would require less than 2TB/hour of uncompressed 4:2:2 video (at 1920x1080), so although nobody is shipping a petabyte around, it's possible that the uncompressed data is being shipped around.
Except that they're probably storing it with some kind of RGBA (32-bit) uncompressed standard, which brings you to ~2.6TB/hour. And then if you decide to shoot it in 4K (4096x2160) that brings you to 11.2TB/hour or a bit over 30TB for the raw version of The Hobbit (48fps, 3-D). Now add in all of the rough cuts, editing revisions, unused footage, CGI, and everything else and you could see it *very* easily getting up over a petabyte. That's just for the studio though. What goes out the door, even in its r
Re: (Score:2)
Why wouldnt someone simply re-encode the film?
Not everything is a conspiracy; increasing filesize in a format where excess data can easily be stripped out is a pretty terrible way to fight piracy.
Re: (Score:3)
BS the size of the versions released to consumers will remain limited by the capacity of the media it is on and in any case the pirates can always recompress.
Afaict what this sort of thing is really about is flexibility. Want if they want to zoom in on something? or run something in slow motion? even remove something from a scene? it's much much cheaper if they can reprocess the existing data than if they have to re-shoot the scene. Compression artifacts that are invisible to the human eye during normal pla
Re: (Score:3)
the biggest problem actually should be how to present 48fps content in the home screen...
Why should this be a problem? It's close to 20 years since I bought my first double-rate TV[*], and today, they tend to be quad-rate (200 or 240 Hz depending on where you live).
48 fps in stereo won't need more than 96 Hz progressive.
If you think of Blu-Ray discs, they're not locked to 30/60 (or 25/50) either, like older generations of video.
[*]: A Grundig, which could do 24-frame movies in 48 Hz non-interlaced or 96 Hz interlaced or video in 50p/100i PAL or 30p/60i NTSC. Of course, there wasn't a lot of
Re: (Score:3)
The Hobbit is projected at 192 images/second. Each frame is shown four times, left, right, left right.
Re: (Score:3)
I guess they mean it is easy to get carried away during the recording phase because "film" is very cheap now. This presents a storage challenge. I imagine keeping 1 PB in redundant, geographically diverse storage would get spendy fast.
I suspect it would be more cost-effective to convert the raw bits to some kind of common standard and then just save the raw footage and the finished copy. Saving all the editing stuff won't help in the long run anyway - chances are the editing programs 20 years on won't know
Re: (Score:2)
Re: (Score:2)
Goodfellas, The Departed, The Aviator, etc are all over two hours, but they don't drag or slog like most newer films that are > 2 hours.
Re: (Score:2)
Re: (Score:2)
It's a story that's not meant to be split into parts like that. It's not a trilogy. LOTR was a trilogy. I won't bother with this thing until the whole thing is available. Then I will just rent it and watch it on the projector upstairs.
It's like tacking 3D onto movies at the end.
Re: (Score:2)
Re: (Score:2)
They could check for recording devices too. Shh... Don't tell the TSA.