Is there an application that will take a selected region of a photo, and return its estimate of the lighting parameters required to emulate the lighting conditions in the selected region of the photo?

My guess at parameters it would return include: lighting sources, source-diffuseness, source-color, source-duration, reflective/incident-light-source, source-motion, etc. (I welcome feedback on these guesses, and on the likelihood that it's possible to statically model them.)

  • 1
    That seems like a tall order. When we look at a photo, we apply some context-aware knowledge about what we're looking at (in other words, we have some version of "normal" in our minds that we can compare to the photo). I'll be impressed if there's something out there that can do this. – D. Lambert Jun 10 '11 at 16:20
  • @D. Lambert: Agree, though believe some of the algorithms within PhotoShop do calculations for things like this, but there's no output that's readable for external use as far as I know; meaning those calculations are used as input for other calculations. – blunders Jun 10 '11 at 16:23
  • 2
    The work required to create software like this is million-fold compared to the 2-3 hours of training someone would need to fairly accurately guess most parameters. Also, some of these things are impossible to figure out... source-duration? – Jędrek Kostecki Jun 10 '11 at 18:58
  • +1 @Jędrek Kostecki: Good point regarding that humans could do it faster(currently), though what's the fun in that... :-) ..."source-duration" for example a flash was used. – blunders Jun 10 '11 at 19:33
  • Ah, the best software in the world is probably the human brain. Algorithms could be developed for this that could make reasonable approximations based on highlights and shadows, but why do it? The human brain is likely to jump to the answer faster under todays technology than a computer will. – John Cavan Jun 11 '11 at 3:18

There has actually been quite a bit of research into this area:

The results are limited, however as the problem is massively underconstrained, in that there are far more unknowns than there is data. This means exact solutions are impossible, and any answer you get is subject to ambiguity.

Another problem for what you're suggesting is that the research in this area is directed towards machine understanding of visual images. Being able to estimate illumination would be important for robots navigate a maze visually as they'd be able to judge the angle of walls etc. These applications will have different demands on the software than the artistic goal to recreate the lighting in a good portrait for example.

On the subject of the difference between research and commercial software, the research of today forms the basis of the software of tomorrow (one of the reasons I trawl through the proceedings of SIGGRAPH every year). Automatic panorama stitching was a research project once and is now taken for granted. I remember reading about content aware resizing when it was published in a computer vision conference (back then it was called "seam carving") and it was only a couple of years before it became a standard feature in Photoshop.

There is a difference, however between something content aware fill and what you're proposing, and this is that content aware fill can save hours of retouching and thus there is a large demand from it. Estimating the illuminations conditions of a photograph is a very quick process for someone adept at lighting.

One final glimmer of hope lies in the area of video post production. Estimating/modelling the original lighting conditions is important for realistically compositing computer generated animation into real footage (lighting inconistencies are far more likely to be noticed in moving imagery than in a still photo). That plus the extra amount of data available in a video stream, and I'd imaging you'd see the feature you're after appearing first in video editing software.

  • +1 @Matt Grum: Thanks, the research helps "illuminate" the issues with solving this problem. My guess is that robot sensors and AI would most likely utilize the correlation of longitudinal observations over time and space, possible including filters for different wave lengths of light; for example, infrared to detect heat. As for artistic goals, do you mean they most likely would require inferences that would go beyond just the location of the camera, lights, and objects. – blunders Jun 10 '11 at 18:17
  • I guess one should point out that when it comes to automatic pano. stitching, you have all the necessary information available in the images you are stitching. That is in strong contrast with guessing the lighting illuminating a scene, since (outside of a light IN the scene) you can never actually know what the light sourced were. In pano stitching, you are information rich...in light source guessing, you are information anemic. When it comes to video and CG, you have the option if explicitly inputting EXACT lighting information, and guessing is unnecessary. – jrista Jun 10 '11 at 18:17
  • 1
    @jrista that's true, but in content aware fill, you have no information about the area you're trying to fill, it's all inference - like the illumination estimation problem, it would have seemed pretty improbable at one time but now it's a standard feature in Photoshop, GIMP – Matt Grum Jun 10 '11 at 18:30
  • +1 @Matt Grum: In fact, content aware fill was what I'd been thinking of as an rough example of this, here's an video showing an example of its use. – blunders Jun 10 '11 at 18:38
  • 2
    @MattGrum: If you research how content aware fill works, it is not nearly as information-anemic as guestimating lighting. A significant part of content-aware fill is cloning of nearby content, along with some basic intelligent algorithms and pattern matching to make the filled in content congruent and pattern-consistent. You DO have a LOT of information available to make content aware fill work...like pano stitching. The only information you have to guess lighting is what is reflected by the scene. The amount of useful information available is orders of magnitude different. – jrista Jun 10 '11 at 19:11

Frankly, no - not with currently available software. There may be research going on here and who knows what the future holds, but right now, no.

There's waay too much information for the software to process to make a go at it - especially at a consumer level. At the very least, you'd have to have some way to tell the software what the subject looked like UNLIT. Thats the advantage a human has - we have a pre-visualization in our minds of what the subject would look like normally. This allows a person to say "well, he doesn't NORMALLY have a fading dark area there or a bright spot over there".

Additionally, there's MANY different ways to produce the same effect in lighting (although many are more common than others).

  • @rfusca: Yes, the "pre-awareness" of the reflective qualities objects, and the camera relation to the lights and objects would increase the likelihood that this would be possible, still think this is possible; estimates are after all not perfect. – blunders Jun 10 '11 at 16:51
  • @blunders: At best, the estimates could give you rough information about how much REFLECTANCE can be observed in a scene...but I stress rough. Trying to get anything more than a rouge assumption about number of light sources, source attributes (color, diffuseness, duration of emission), etc. is really just asking for a lot of guesswork based on a judgment based on observation. If there is one thing computers suck at...thats it. Making an accurate, educated guess about what lighting may have lit a scene requires an experienced mind. – jrista Jun 10 '11 at 16:57
  • You could make the argument that a sufficiently advanced artificial intelligence could learn enough to become an experienced mind in relation to observing the lighting of photographs...but at the moment I doubt there is even anything like this on a rudimentary scale, let alone something that could produce consistently useful results. – jrista Jun 10 '11 at 16:59
  • @jrista - Right, my answer reflects whether there's likely something out there with current technology that produces realistic, usable, accurate results. – rfusca Jun 10 '11 at 17:02
  • I think this is likely the case. However, I wouldn't be surprised at all if there were research into this area. It seems like an interesting problem. So, I'd hesitate to say "no" without some study. – mattdm Jun 10 '11 at 17:07

Your Answer

