Marc Levoy on the Balance of Camera Hardware, Software, and Artistic Expression (theverge.com) 35
A major focus of any smartphone release is the camera. For a while, all eyes were on the camera's hardware -- megapixels, sensors, lenses, and so on. But since Google's Pixel was introduced, there's been a lot more interest in the camera's software and how it takes advantage of the computer it's attached to. Marc Levoy, former distinguished engineer at Google, led the team that developed computational photography technologies for the Pixel phones, including HDR+, Portrait Mode, and Night Sight, and he's responsible for a lot of that newfound focus on camera processing. An excerpt from the wide-ranging interview: Nilay Patel: When you look across the sweep of smartphone hardware, is there a particular device or style of device that you're most interested in expanding these techniques to? Is it the 96-megapixel sensors we see in some Chinese phones? Is it whatever Apple has in the next iPhone? Is there a place where you think there's yet more to be gotten?
Marc Levoy: Because of the diminishing returns due to the laws of physics, I don't know that the basic sensors are that much of a draw. I don't know that going to 96 megapixels is a good idea. The signal-to-noise ratio will depend on the size of the sensor. It is more or less a question of how big a sensor can you stuff into the form factor of a mobile camera. Before, the iPhone smartphones were thicker. If we could go back to that, if that would be acceptable, then we could put larger sensors in there. Nokia experimented with that, wasn't commercially successful.
Other than that, I think it's going to be hard to innovate a lot in that space. I think it will depend more on the accelerators, how much computation you can do during video or right after photographic capture. I think that's going to be a battleground.
Nilay Patel:When you say 96 is a bad idea -- much like we had megahertz wars for a while, we did have a megapixel war for a minute. Then there was, I think, much more excitingly, an ISO war, where low-light photography and DSLRs got way better, and then soon, that came to smartphones. But we appear to be in some sort of megapixel count war again, especially on the Android side. When you say it's not a good idea, what makes it specifically not a good idea?
Marc Levoy: As I said, the signal to noise ratio is basically a matter of the total sensor size. If you want to put 96 megapixels and you can't squeeze a larger sensor physically into the form factor of the phone, then you have to make the pixels smaller, and you end up close to the diffraction limit and those pixels end up worse. They are noisier. It's just not clear how much advantage you get. There might be a little bit more headroom there. Maybe you can do a better job of de-mosaicing -- meaning computing the red, green, blue in each pixel -- if you have more pixels, but there isn't going to be that much headroom there. Maybe the spec on the box attracts some consumers. But I think, eventually, like the megapixel war on SLRs, it will tone down, and people will realize that's not really an advantage.
Marc Levoy: Because of the diminishing returns due to the laws of physics, I don't know that the basic sensors are that much of a draw. I don't know that going to 96 megapixels is a good idea. The signal-to-noise ratio will depend on the size of the sensor. It is more or less a question of how big a sensor can you stuff into the form factor of a mobile camera. Before, the iPhone smartphones were thicker. If we could go back to that, if that would be acceptable, then we could put larger sensors in there. Nokia experimented with that, wasn't commercially successful.
Other than that, I think it's going to be hard to innovate a lot in that space. I think it will depend more on the accelerators, how much computation you can do during video or right after photographic capture. I think that's going to be a battleground.
Nilay Patel:When you say 96 is a bad idea -- much like we had megahertz wars for a while, we did have a megapixel war for a minute. Then there was, I think, much more excitingly, an ISO war, where low-light photography and DSLRs got way better, and then soon, that came to smartphones. But we appear to be in some sort of megapixel count war again, especially on the Android side. When you say it's not a good idea, what makes it specifically not a good idea?
Marc Levoy: As I said, the signal to noise ratio is basically a matter of the total sensor size. If you want to put 96 megapixels and you can't squeeze a larger sensor physically into the form factor of the phone, then you have to make the pixels smaller, and you end up close to the diffraction limit and those pixels end up worse. They are noisier. It's just not clear how much advantage you get. There might be a little bit more headroom there. Maybe you can do a better job of de-mosaicing -- meaning computing the red, green, blue in each pixel -- if you have more pixels, but there isn't going to be that much headroom there. Maybe the spec on the box attracts some consumers. But I think, eventually, like the megapixel war on SLRs, it will tone down, and people will realize that's not really an advantage.
Native Advertising (Score:1)
Is all of tech "journalism" these days just native advertising? Here's an article about how awesome Apple is. Here is an article about how evil Apple is. Here is an article about how awesome Android cameras are. Here is an article about how awful the Google App store is. Why bother writing an article when you can stick your byline on a press release?
Re: (Score:2)
Especially since nobody cares about the cameras since they became OK a few years ago, beside half a dozen bozos who can't be bothered to buy professional equipment and always complain.
Re: (Score:3)
Smartphones have been good enough for a few years now. People only want two things: better battery life and displays that are harder to scratch and break.
Re: (Score:2)
Excuse me, cameras are serious, life or death business. At least if you go by the comments on camera reviews: https://www.dpreview.com/revie... [dpreview.com]
Re: (Score:2)
> Excuse me, cameras are serious, life or death business.
If you think so, go buy "a serious camera". If it's a wedding or other big solemn event, I'd hire a pro photographer to bring in his $10,000+ worth of gear to do the photos. The other 99% of the time you can get by with a $100, compact, 16 megapixel, pocket camera from Walmart or your local "drug store" chain. Even more convenient is a "good enough" camera in the smartphone that you're already lugging around.
I don't get the hobbyists who spend $5,0
Re:Native Advertising (Score:5, Interesting)
Well, there are a LOT of people out there that have a lot of disposable income.
Believe it or not....
And well, some appreciate good sounding sound systems and enjoy a really nice AV unit.
Some like to take good images, suitable for blowing up to hang on the wall.
With camera gear, I started out mostly wanting to shoot some decent resolution videos, and yes, even though YouTube trashes them, quality in still looks good there.
But you get into hobbies....and over time you buy gear.
I'm by no means wealthy, BUT...I have some more disposable $$ than others. And from the time I was a kid, I rarely would spend little bits of money here and there on frivolous things. I always preferred to save up money and buy 1-2 BIG things a year.
I started working as young as I could...doing yards, babysitting, and when I was 16yrs, I started in a restaurant washing dishes and moved my way up the chain.
I started way back then saving and buying my stereo....took a couple years, but I bought a Marantz (good at the time) receiver...and later saved for speakers, etc....and over the years I'd find deals and upgrade parts of it, till today I have tube amps running Klipshorn speakers.
I love it, worth every penny over the years.
I've done the same with cameras.
I recently bought a used Fuji GFX100 digital "medium format" camera. Yes, I spent some $$$....but it makes me happy and over the past couple years, I've made a few $$'s selling my "art".
I've gotten in to be selected to shoot JazzFest and VooDoo Fest in New Orleans and worked my way into other concert shoots.
Those are fun with the access you get and fun to get on stage with great acts and get shots no one else gets.(It really pays off to get there early and make friends with the stage manager).
But anyway, I've dropped thousands of dollars on Audio/Visual equipment. I've dropped thousands on stills and video equipment.
Why? It makes me happy.
I rarely eat out, I prefer to cook my own meals from scratch, so I save money there. I dress decently, but not flashy. I live a fairly modest lifestyle and am not in debt.
I save and buy what I really want and go for as much quality as I can.
It's simple really.
It depends on what's worth what to you. I prefer to save and have VERY nice toys. I don't care what other people may think about them, I buy them for me and the joy they bring me.
Re: (Score:2)
"Serious" cameras and smartphone cameras use essentially the same sensors apart from the size. The low light performance improvements mentioned in the summary were one of the biggest reasons for me to upgrade the DSLR I inherited from my Grandad. (Not that I really have a serious DSLR: it cost me about 300 EUR. But since 90% of my shots use focal lengths of 300mm or more, a compact doesn't cut it).
Re: (Score:2)
I don't get the hobbyists who spend $5,000+ on "serious cameras" or "serious sound systems" or "serious gaming rigs". If you're a pro photographer or DJ or rock band or professional gamer who makes their living off their gear, that's another story. But that doesn't apply to most of us.
That's ok, you don't have to 'get' us. We'll continue to enjoy our hobbies, and if that includes spending money to better enjoy them, that's a decision we can make.
Sure, a phone camera is 'good enough' for many things but it lacks many features available in the $5000 price range. Why should only the professionals use those features?
Re: (Score:1)
...buy professional equipment...
And why would anyone think buying "professional equipment" makes, or should make, any difference? Years ago I told myself I would go digital when sensors reached 24MP, the approximate of 135 Kodachrome 25. But I never did. In fact these days I use an emulsion with considerably lower 'pixel density' than Kodchrome had. I take pictures for different reasons and different results now, and it works for me. Some are very good if I say so myself.
The issue with "cameras" is that you can do almost anything to a
Re: (Score:2)
I recently decided not to upgrade my phone because my chosen replacement showed artifacts in photos in reviews--likely due to overprocessing in software. So I'm very interested in the possibility of utilizing low level camera logic that doesn't rely on the manufacturer being as good at image processing as they are at hardware.
Re: Native Advertising (Score:2)
These days? It's always been like that. Companies write their own press releases, then they send them to magazines or journals or trades, who in turn eat up all the free content they can use as filler to go between the ads which provide revenue. This has always been the business model in tech journalism (and many others).
We use to make fun of Doctor Who Sonic Screwdriver (Score:2)
Back in the days we use to make fun of Doctor Who, who used the Sonic Screwdriver for practically every type of job with the exception of driving in screws (which might have happened once or twice)
While today we carry around a device that we call a phone, which one of the least used features is actually making a telephone call.
In terms of its design and development, the Phone feature is just kinda of an afterthought. Oh yea, lets put in the phone software to interact with the Cell Systems Phone service.
People care about megapixels? (Score:4, Insightful)
Camera resolutions already were 100% fine years ago. More megapixels seems like such a stupid thing to determine what phone you buy.
Re: (Score:3)
Well, depends on how much you value photography.
I can't carry my GFX100 everywhere I go...so, when I'm without that, I do want to have as powerful as I can the ubiquitous camera I almost always have on me, in my phone.
Now, as the author mentioned, megapixels alone do not make the quality camera, depends on sensor, size of pixels, etc.
But it is one factor.
I'd be looking at
Re: (Score:2)
Re: (Score:2)
Yep...about right on the RAW images size I get out of the GFX.
And....trying things like focus stacking, the largest so far was over 160 images.
That not only takes up drive space, but gives programs like Helicon Focus and others a real run for their money, but oh WOW...the images.
I'm currently having to sort of pause shooting and spend time going through my external work drive (3TB)
Re: (Score:2)
I know a professional that uses a GFX for weddings in-between his other work.
I hate him.
He's a really nice bloke but nobody should be that good a photographer and also have a camera like that. It just makes the rest of us look bad.
Re: (Score:2)
Re: (Score:2)
Camera resolutions already were 100% fine years ago.
Fine for what? Certainly not fine for enhanced imaging techniques that provide a world of wonders for noise reduction, colour enhancement and compensation for external factors like vibration.
Give me raw (Score:2)
I have an old pixel. There's a 3rd party FOSS app that takes raw photos I can import into standard tools (Lightroom, Darktable, etc). It's not as good as a real DSLR, but it's a whole better than jpg. Can't do this with video yet. 10bit has become standard in the DSLR scene and has been broadcast standard for over a decade. 12 bit raw on the BPMCC 4k/6k is really something else on Resolve. You can push color and exposure around almost like real film negatives. Ultimately, having that on a phone would be rea
Re: Give me raw (Score:1)
It's not as good as a real DSLR, but it's a whole better than jpg.
It's less about image quality and far more about the geometry of the optics; parallax and all that.
How many megapixels do you need ... (Score:2)
Re: (Score:3)
And that's why Aperture Science was created.
Re:How many megapixels do you need ... (Score:5, Interesting)
It depends on the design. Wide-angle lens-sensor combinations and big apertures yield higher theoretical diffraction-limited megapixels. Figure 20 to 40 megapixels as an impractical limit. Using the principle of oil-immersion microscopes, a higher number may be possible if the lens has a high index of refraction element in contact with the sensor.
I'm curious: how many of you would rather have: (Score:3)
Re: I'm curious: how many of you would rather have (Score:1)
Re: (Score:2)
..a phone that's just a phone, ..a camera that's just a camera, ..a media player that's just a media player, ..and a portable computer that's just a portable computer, ..all of the above which excel at being what they are, instead of just one device that's mediocre at being everything simultaneously?
You forgot the a level 5 bag of holding that gives +10 strength to the wearer that is just a level 5 bag of holding that gives +10 strength to the wearer because holy crap who the hell wants to carry all of THAT arouund.
I have a good $30k worth of camera equipment. No it doesn't excel at what it does because it spends 99.9% of it's time at home instead of in my pocket when I need it. I think it and the iPod are having a contest, though the iPod is winning given it hasn't been used in years because who the
Re: (Score:2)
Re: (Score:2)
I remember when I had a phone, a camera, a media player, a personal organiser.
I absolutely wanted convergence. I'm fucking delighted that it happened so quickly.
I do still have a camera that's just a camera but I don't carry it around with me all of the time, so I still want and use the camera that's built into my multi-function device.
That device is not mediocre at phone calls, at playing media, at being a portable computer and it's actually pretty good at being a fixed lens camera. Maybe you bought the wr
Since the google pixel? What about Lytro (Score:2)
Really, the first computational photography for the masses was the Lytro camera. Then came the guy (who Apple hired) who was able to pull some ridiculous frame rate out of the iOS camera by rewriting JPEG natively, or something like that.
The Pixel guy is late to the party.