Light Field Photography Is the New Path To 3-D 79
waderoush writes "In November, Lytro, the maker of the first light field camera for consumers, upgraded its viewer software to enable a feature called 'Perspective Shift.' In addition to refocusing pictures after they've been taken, Lytro audiences can now pivot between different virtual points of view, within a narrow baseline. This 3-D capability was baked into Lytro's technology from the start: 'The light field itself is inherently multidimensional [and] the 2-D refocusable picture that we launched with was just one way to represent that,' says Eric Cheng, Lytro's director of photography. But while Perspective Shift is currently little more than a novelty, the possibilities for future 3-D imaging are startling, especially as Lytro develops future devices with larger sensors — and therefore larger baselines, allowing more dramatic 3-D effects. Cheng says the company is already exploring future versions of its viewer software that would work on 3-D televisions. 'We are moving the power of photography from optics to computation,' he says. 'So when the public really demands 3-D content, we will be ready for it.'"
Bought a Lytro (Score:3, Interesting)
Returned it.
It was awful, and the resolution wasn't hot
Re: (Score:2)
While I hope Lytro manages to stay in business - It's nice to have somebody doing something different, they're a long way from any sort of realistic 3D. You're asking for a lot of computational and sensor power to create anything beyond postage stamp images (the major issue with the current Lytro products).
Imagine You Tube videos in 3D - stupid cat pictures without any editing in three dimensions! Yeah!
Re: (Score:2)
"Imagine You Tube videos in 3D - stupid cat pictures without any editing in three dimensions! Yeah!"
Already there. Fuji had a point and shoot camera that did 3d video. there is a lot of stupid cat 3d youtube videos.
3D imagine that may leads to 3D printing !! (Score:3, Interesting)
I have a Lytro as well. I know that currently its limitations are so severe that have rendered the Lytro cameras to nothing but a novelty.
Its limitations right now are in the computational power --- it does take a whole lot more computational power to make it useful --- and the HORRENDOUS AMOUNT OF DATA to make it any useful.
But, I still have hope in this 3D imaging thing --- I do not see it as mere toy, I see a future link, in between 3D imagine and 3D printing, and beyond.
Currently, to gather data on 3D i
Re:3D imagine that may leads to 3D printing !! (Score:5, Insightful)
"Currently, to gather data on 3D imagery we use technologies such as MRI, which in itself not really portable."
We use things like MRI to gather tomographic data. It's for seeing inside. Lytro doesn't do that, and never will.
Currently if you want to do the kind of 3D we're talking about you can buy a Lytro and get low resolution with a lot of data and processing, or you can buy one of the commonly available compact cameras that include two lenses (like this one [dpreview.com]) and get instant, high res results.
Re: (Score:3)
Currently if you want to do the kind of 3D we're talking about you can buy a Lytro and get low resolution with a lot of data and processing, or you can buy one of the commonly available compact cameras that include two lenses (like this one [dpreview.com]) and get instant, high res results.
To be fair, that gives you stereography, not 3D. From stereography, you can compute, tada, something resembling 3D along a short baseline, i.e. not only showing the image through one of the lenses, but a theoretical image from anywhere close to where they were located. The Lytro is a much more solid way of achieveing that, though, if they can create sensors that are both wide enough and carry enough resolution. Currently, they are a long way off. I would even think that you could put to Lytros in kind of a
Re: (Score:2)
The Lytro is also effectively doing stereography. Which is fine, because that's how we perceive 3D anyway.
Re: (Score:2)
This is just the beginning of a disruptive technology. I don't think the Lytro will necessarily be the answer for consumer products. If you look up "Super Resolution" and the "Pelican slimphone camera array" you'll see that there are emerging technologies that will be at least as effective and collecting both spatial and 3 dimensional data. In fact, its easy to imaging 6 -10 camera arrays being placed on the back of an iPad producing images with higher resolution that a top professional camera, as well as t
Re: (Score:1)
Consider a stadium of light field cameras + location + time stamps Crowd source to the cloud and view all the pictures you wish you had taken.
Re: (Score:1)
They're not really the only player in this space-
LinX Imaging [ http://image-sensors-world.blogspot.com/2012/07/linx-imagings-multi-aperture-camera.html [blogspot.com] ]
Pelican [ http://image-sensors-world.blogspot.com/2012/11/pelican-imaging-capabilities-presented.html [blogspot.com] ]
Is starting to be applied to high-precision 3D measurement:
Raytrix [ http://image-sensors-world.blogspot.com/2012/12/raytrix-presentation-from-vision-2012.html [blogspot.com] ]
Ascentia Imaging [ http://www.ascentiaimaging.com/ai_web_2812_003.htm [ascentiaimaging.com] ]
Re: (Score:2)
Actually, that sounds kind of awesome.
Re: (Score:2)
Re:Bought a Lytro (Score:4, Insightful)
An AC calling someone a piece of shit. What a surprise.
I don't recall claiming I was interested in technology, but the fact that I am means I have actually studied what Lytro does. That's why I understand that it will be a failure. I suggest you funnel your anger toward a big, fat investment in Lytro if you feel otherwise.
Re: (Score:2)
Lytro trades off enormous, and irrecoverable, losses in resolution for features of dubious worth that are of no value to a photographer.
They may be of no value to a conventional photographer, but it's not unheard of for new technologies to lead to new art forms.
Re: (Score:3)
Perhaps you should study more closely the problems Lytro is trying to solve, then speculate on the "new art forms" it may lead to...or perhaps you'd be happier continuing to believe that Lytro could be useful for something.
Here's a hint...depth of field is directly related to the resolving power of a system. Lytro trades resolving power in order to gain variable depth of field after the fact. What it actually does is destroy resolution thereby inflating apparent depth of field, then allow selective degrad
Re: (Score:3)
Now you're the one throwing around non sequiturs. I'm disputing your description of post-processing refocussing as useless: I'm not claiming anything about the current resolution of any particular product, and I'm certainly not planning to buy a light field camera in the next few years.
My initial idea for a work of art was to combine real-time refocussing with eye tracking to unfocus the area which the viewer is currently looking, focussing instead on something in the peripheral vision, conveying an impress
Re: (Score:3)
depth of field is directly related to the resolving power of a system
um... no,
Depth of field has to do with the physics of optics. Lytro is using a large array of lenses then combining the collected data in interesting ways. Yes they are sacrificing resolution, but their depth of field "magic" could just as easily be achieved by taking a large number of images taken with a DSLR, but the images would not all be taken at the same time, and the processing power needed would be immense. You might be more correct in saying that Lytro is trading resolution for time...
Re: (Score:1)
Returned it.
It was awful, and the resolution wasn't hot
Sure you should buy a Nikon D600 or digital back for a large format camera.
This light field thing is new and novel. New and novel to the degree that it is easy to see that sensor and post processing technology have room to improve.
It will take a while but this may prove to be the trick that solves the most common cell phone camera image errors. The technology is flat out amazing.
Welcome to the future of imaging.
My first thought (Score:5, Interesting)
Enhance 224 to 176. Enhance, stop. Move in, stop. Pull out, track right, stop. Center in, pull back. Stop. Track 45 right. Stop. Center and stop. Enhance 34 to 36. Pan right and pull back. Stop. Enhance 34 to 46. Pull back. Wait a minute, go right, stop. Enhance 57 to 19. Track 45 left. Stop. Enhance 15 to 23. Give me a hard copy right there.
Re: (Score:2)
I could never figure that scene out. Deckard seemed to be looking around a foreground object.
Re: (Score:1)
There was a paper some months ago about revealing an object obstructed by another thru the scatered rays that were bounced by it. Maybe if we could merge the two tecnologies, that amazing camera could be constructed.
Re: (Score:2)
the tech depended on the photograph itself being of insane, insane resolution(possibly higher than physically possible on hard medium) and the machine at the police hq(it was a remote terminal he was using) having immense computational power and fancy algorithms to calculate from reflections in the picture what was "outside" the picture, as viewed from a different angle.
Lytro's 3-D is inherently limited (Score:5, Informative)
Wow, TFA is really glossing over an inherent limitation:
If the goal of this is to produce useful stereo content that replicates the parallax seen by humans, then the image sensor needs to be at least as big as the average distance between two human pupils. That's roughly six centimeters. The Lytro's sensor is around six millimeters. Somehow I doubt they're going to increase their form factor by ten times in each dimension, and since the point of a Lytro is to avoid fancy lenses they can't bend the light path to compensate.
Re:Lytro's 3-D is inherently limited (Score:5, Interesting)
No, but they might be able to avoid the lens entirely and do microlensing on a flat surface. For example, I could totally see the entire back of a cell phone be a light field camera, automatically throwing out data from where your fingers overlap the edges from holding it by using capacitive sensors in some way. I mean, we're probably talking twenty or thirty years out here, but that's the direction I see things heading eventually. And that would give you a believable stereo spread, not to mention much more usable resolution.
Re:Lytro's 3-D is inherently limited (Score:5, Interesting)
As long as you've got enough parallax to work out the depth information from your scene, you can push the effect to recreate viewpoints that are wider then you have real data for.
You will end up with tiny slivers of image that you don't have pixel data for when there's a foreground element that diverges more then it did before, but that's easy to recreate. All post converted 3d films have this problem to an even greater extent, there's algorithms out there to clone the surrounding pixels or even use pixels from other frames if the object is moving through the scene
There are lightfield cameras out there that instead of using a single chip, they use an array of small cameras (think cell phone cameras) The adobe one is 500 Megapixels
See the research by Todor Georgiev http://tgeorgiev.net/ [tgeorgiev.net] The Lytro camera is a nice cheap toy, but there's some stunning results form researchers.
Re:Lytro's 3-D is inherently limited (Score:4, Informative)
Typical IPD is 54-68mm. IMAX film stock is 70mm wide. This is not an insurmountable problem, at least for professional use.
Imagine a 70mm lightfield motion picture camera. Other than the fact that the data throughput would be positively insane, the requirements for physical size would be substantially less than a current IMAX camera.
I suspect that you can actually get away with less than the typical IPD and still produce a convincing effect. In which case, you can buy the required sensor today; you can get 48mm wide medium format digital sensors, and there's nothing special about the sensor in the Lytro. It's the array of microlenses and software that make it special. So it would be possible today to build a Lytro motion picture camera with a 48mm digital sensor, and I suspect that 48mm is close enough to the typical IPD to produce a convincing effect. Such sensors also have the resolution to make lightfield work for a motion picture (50 MP models turned in up the first page of results on B&H), and the cameras themselves are smaller than most motion picture cameras (or even ENG cameras)...
I suspect that the primary problem would be, again, the data throughput. Uncompressed 24fps 50 megapixel 36-bit images, those would pump out 41 gigabits per second... Compression would be pretty much required. If we use redcode as a benchmark (because apparently motion picture productions are happy with the quality of the compression enough to use it), where the minimum camera-supported compression ratio (on the RED ONE) is 8:1 and the highest is 12:1... This gives us about 5.1 Gbps and 3.4 Gbps... Heck, that's easy to handle. Existing communications tech can handle that, you could have a single 10 Gbps ethernet cable running out the back of your camera to an on-site storage box, and storing that sort of data rates isn't hard. Even a 4TB on-camera SSD module could store 156 minutes of footage... and handle those kinds of write speeds.
Re: (Score:2)
Re: (Score:2)
voxels (Score:2, Interesting)
Combine the data from 12 or so of these in a matrix and you have a really powerful, accurate, self optimizing point cloud capture device for voxel 3d content.
Re: (Score:2)
My first thought was that this can do for "true" 3D recording what the Kinect couldn't because of its interference with other Kinects. We could put a bunch of these around someone and reconstruct a very complete 3D scene, including normal information (the camera knows what direction light is coming from), which is useful for motion capture, videobloggers who want a neat gimmick, and -porn-... and the latter has driven all sorts of innovations.
And the geek in me is giddy at the thought of the data you could
Re: (Score:2)
I forgot to mention why recording a 3D scene is useful for anything but 3D scanning. I hope most will already know, but for those that don't: you could do all sorts of things with it. The first things to come to mind is that stereoscopic 3D viewing is easy to achieve with realistic results. You could also look at the subject from different angles, and post-processing could use the 3D data to make extremely accurate green-screen type cutouts even without a green screen or anything like it. If you're clever,
Re: (Score:2)
Sure are a lot of buzzwords applied in ignorance here.
when the public really demands 3-D content (Score:5, Insightful)
'when the public really demands 3-D content'
When it doesn't require glasses and doesn't give you headaches.
Re: (Score:2, Interesting)
I wonder if it would be possible to make a 'light field' display, which rather than each pixel emitting light in all directions like in current 2d and faux 3d displays, it would be able to emit light in both the frequency and vector that was detected by the camera. This would be true autostereoscopic 3d, as the emitted light would have the same properties as the original light allowing the eye to naturally focus on it. I wonder if this would be possible by perfecting lenticular display technology, or if it
Re:when the public really demands 3-D content (Score:4, Interesting)
A lenticular lens array in front of LCD screens are a nice do-it-yourself solution that almost does the trick. It makes an autostereoscopic display that can display more than 2 images in different directions, making it possible to move around in front of the screen and see a stereo image without glasses. However, there are a couple of limitations. The LCD resolution suffers tremendously and the number of zones that you can create still isn't very high. Maybe it gets better with retina displays, but I'm not sure. Even paper printouts of 20 to 30 images at 600dpi are barely good enough.
Another interesting idea is this proposal: http://gl.ict.usc.edu/Research/LFD/ [usc.edu] - replace each pixel on a *huge* screen with a microprojector acting as a directional light source. It is insane in its own special way, but this research group has successfully thrown massive amounts of hardware at problems in the past.
Re: (Score:1)
If we could make displays at the same resolutions that we make image sensors, then it would be quite easy to make the display, since it would operate just like a Lytro except in reverse (the Lytro uses a microlens array in front of a regular high resolution image sensor).
Now, performing the computations to know what to display, that's another story. Of course, you could just display the data from the Lytro camera directly.
Re:when the public really demands 3-D content (Score:4, Informative)
There was an article earlier about Tensor Displays (slashdot link) [slashdot.org], (MIT link) [mit.edu], which used a sandwich of three high-refresh-rate LCD screens to simulate a light field by using the screens to selectively block light in multiple directions.
Re: (Score:2)
That's right, Lytro produces a shitty substitute for missing phase information and it does so in a computationally intensive manner that ruins resolving power. The founder knew this from the start but assumed Moore's Law would solve the problem. Thing is...it can't. Photography needs resolution AND dynamic range. Lytro makes the trade-off between those two competing goals much worse. That means Lytro can only be interesting for low resolution work. It may find its way into your next cell phone but not
Re: (Score:3)
Re:The 3D thing is kind of fake (Score:5, Informative)
Look again. Pay attention to the bubble that masks the rear stair column. As you shift the image, the placement of the column within the bubble shifts (the amount of plant displayed varies).
This is beyond stereoscopy, and is a direct result of how they are capturing images. Stereoscopy only works on a single axis of view. This works on multiple axis.
The reason why it looks "fake" to you, is because none of this is done by the camera itself. Everything is done by running computations on the captured image. And as a result, the only way to display it is via an interactive container like Flash. The computations aren't done in realtime, so you only get whatever focus planes the algorithm (or artist) picked out when running the processing. Presumably, you could reprocess for different focal depth points.
Re: (Score:3)
Huh? The image with glasses works perfectly. I'm almost certain that you're completely incorrect here. It looks like it loads a full set of pre computed images for all available POV and focal points. Sure there is probably a rough threshold but that's a limitation of bandwidth, not the data.
Re: (Score:2)
It looks like it loads a full set of pre computed images for all available POV and focal points. Sure there is probably a rough threshold but that's a limitation of bandwidth, not the data.
No. This is an image taken from one camera position, with one sensor, with tiny lenses of different focal length over adjacent pixels. There is only one point of view, but depth can be inferred from depth of focus.
This is not a hologram, a time of flight image, or a 3D composite of images taken from multiple locations. It's much simpler than that.
Re: (Score:2)
No, you're completely wrong. The image through the bubbles and glass move. Also, you're thinking 3D, this isn't 3D. Your monitor isn't 3D. This is a change in perspective. It's like having remote control over a far away webcam. But it's a single image, from a single moment in time. It's like they took a picture of a scene and now you can travel around and look at it form different angels.
Re: (Score:1)
How does one form different angels? Why, what's wrong with the one's we've got?
Re: (Score:1)
There isn't a single point of view. The lens isn't a pinhole lens: there's a circular area of view. That's what Lytro is doing here: simulating pinhole cameras at various positions on the lens. You can't get a point of view that isn't covered by the camera's lens.
Re: (Score:2)
If you're talking about the last image, where they have a row of drinking glasses, it only works at first glance. The background through that first glass moves as it should. But the background through the further glasses does not.
Re: (Score:2)
All three images you point to seem okay.
In particular the row of glasses clearly shows a different perspective in the background; you can see from the kid's arm and the second glass from the background of the first glass.
As you describe from the plant image, I gather you were expecting all kinds of visual layers, but this is pretty much what you should expect; objects at a distance have far less parallax effect, this holds true for normal photography a well.
I think you have to keep in mind that the change i
Medical Applications of the Lytro? (Score:3)
In November, Lytro, the maker of the first light field camera for consumers, upgraded its viewer software to enable a feature called 'Perspective Shift.' In addition to refocusing pictures after they've been taken, Lytro audiences can now pivot between different virtual points of view, within a narrow baseline.
It sounds like the techniques Lytro uses could make for a really good Borescope/Endoscope. Imagine being able to virtually shift your view to get another perspective (even if only a few millimeters), without moving your scope. If you could process the shifting fast enough, you might use it as a way to compensate for the motion of a beating heart or moving probe. Or upon reviewing a recording, re-focusing on a newly-found item of interest, even after you've pulled your scope out of the patient.
It might also be used to build a compact yet superior type of Fundus Camera -- current cameras are often rather bulky things. The Lytro has a single aperture, yet might be capable of imagine the retina in 3-D (it is a multi-layered structure). The light field info might even allow you to compensate for some kinds of cornea or lens aberration.
Tilted focal plane... a real world example (Score:2)
Here [flickr.com]'s a virtual focus photo I did a few years ago, placing the focal plane on a skew.
If you take photos from a large enough set of positions with a normal camera and some time, you can get the same thing lytro does, but only with still subjects.
Blade Runner anyone...? (Score:2)
Ha! Now all those nitpickers who complained that Deckard's inspection of Leon's photo in his Esper machine shows an impossible "perspective shift" will have to eat their words!
I guess with a good-sized light field, you really can photograph around corners!
Re: (Score:2)
as far as I could tell it worked by calculating stuff from reflections in the picture.
it was a 2d picture after all that he had. just one with crazy, crazy dpi.
Re: (Score:2)
How can you and others comment on how it "worked," when it's a visual effects sequence on a fantasy film? It wasn't intended to prove any theoretical process, and I doubt that the sequence was conceived by physicists; it was intended to look futuristic and cool.
dZ.
Re: (Score:2)
Even a reflection will only give you a single POV from a traditional camera, no matter how high the resolution. But there's definitely a point in the sequence where Deckard tells it to track left, and part of the door edge moves across the background, revealing more of the woman and nightstand behind it. That's the part nitpickers complain about - looking "around" an object in the picture.
Re: (Score:2)
The nitpickers do not understand the scene, or follow Deckard's mindset as he examines the photo.
The view was taken in one room looking into a second room(hallway?) through a doorway. On the opposing wall (in the other room looking through the doorway) is a convex silver mirror. In the right portion of the curved mirror is reflected the image of a partially open door that impinges into the room where the picture was taken. A full length mirror set in the center of the door creates the correct incident a
Re: (Score:2)
sorry for replying to my own post... correcting a slight error in my description.
Erm cant keep left and right straight... the cot/bed is on the right... and also it is not unusual in remodeled victorians converted to rooming houses to have a room that has an entrance at ether end of the central hallway.
Re: (Score:2)
Watch this section, from 1:57 to about 2:08:
http://www.youtube.com/watch?v=qHepKd38pr0#t=1m57s [youtube.com]
You will see the movement of the door edge that reveals more detail as Deckard tracks and zooms. There's no possible way for a flat, static photo taken from a single POV to do this. It has to be 3D, or layered, or the Esper has to be doing some kind of interpolation from the distorted reflection.
Re: (Score:2)
That the Esper is applying interpolative corrections makes perfect sense for the scene. "Enhance" would seem to mean, "figure out what is going on in this picture and correct for possible distortion(s)."
The tools we have today can do this. They can't do it automagically. It generally requires telling the software, "In this region of the picture... Treat *this* curved edge as if it was straight." Doing so would result in an acceptable image.
3D? (Score:2)
So when the public really demands 3-D content, we will be ready for it.
I thought the public had already weighed in on 3D and their opinion is basically, "Meh".
Examples of Artifacts (Score:1)
Mostly they show up when in creative mode because you can have areas of the scene that can't be pulled into focus, and when shooting dirty glass. I've been really happy with the picture quality in general though, and am sure that improvements in the software/algorithms will help a lot.
https://pictures.lytro.com/tophertuttle/pictures/544030 [lytro.com]
https://pictures.lytro.com/tophertuttle/pictures/544050 [lytro.com]
https://pictures.lytro.com/tophertuttle/pictures/531986 [lytro.com]
Also this article makes for an interesting read.
http://eclect [eclecti.cc]