Example-based image manipulation
CC Attribution - NoDerivatives 2.0 UK: England & Wales:
You are free to use, copy, distribute and transmit the work or content in unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Reinhard, Erik
Many color-related image adjustments can be conveniently executed by exposing at most a small number of parameters to the user. Examples are tone reproduction, contrast enhancements, gamma correction, and white balancing. Others require manual touch-ups, applied by means of brush strokes. More recently, a new class of algorithms has emerged, which transfers specific image attributes from one or more example images to a target. These attributes do not have to be well-defined and concepts that are difficult to quantify with a small set of parameters, such as the “mood” of an image, can be instilled upon a target image simply through the mechanism of selecting appropriate examples. This makes example-based image manipulation a particularly suitable paradigm in creative applications, but also finds uses in more technical tasks such as stereo pair correction, video compression, image colorization, panorama stitching and creating night-time images out of day-light shots.
 So yeah, that's indeed what I want to talk about. But first I'd like to thank the organizers for inviting me. It's a beautiful place, Amsterdam as a whole, and in particular this venue. And I think it's the nicest place that I've given a presentation in so far. So it's an honor to be here. And also thank you to the audience because you had to drag yourself away fairly early from the warmth of your beds. So thank you for showing up. If you have any compelling questions, there is, I believe, some time at the end. But feel free to interrupt. I'm fully interruptable. So shout or inflate your orange balloon if you have a question. All is good. So before I go too fast, I have sort of made my career out of high dynamic range imaging. And in particular, tone reproduction. And so in that field, we want to be as precise as possible. And we want to reduce the dynamic range of the real world to something that our limited displays can display. And so that research gave me the need to understand color a little bit better. And so this is how I wandered into the field of color science, almost accidental. But on the side, I'm interested in the precise color science. But I'm also interested in just having a bit of fun and just messing with images a bit. And that is what this talk is about. So there's not the same level of precision here. But there is, I hope, a higher degree of amusement and entertainment. And also, I hope you enjoy this type of research as well. So if I can try forward to the next slide, I'll aim for that there. For most algorithms, the input, if you're in imaging at least, is an image. And then some parameters, sliders if you like. And you drag the sliders until you're happy. If you want to do white balancing, you begin by an image. And then you say, I want that to become a D65 or an E or an A or whatever it is you like. And that's then your parameter adjustment. And that's all good and fine. But sometimes you might want to do things to images that are not so easy to specify. Sometimes you might want to change an image according to, well, some other image that you may have found. And if there's another image somewhere, the internet is full of them, of images that I like and that are better than my own images, I might be able to use these images as an example. And so I started messing with example images, largely for the purposes of improving my holiday snapshots. Turns out that there is slightly more serious applications to it as well. And I'll get to that later on. And it's sort of given rise, I think, to a little sub area, certainly in graphics imaging, whereby you can take an example image, find some attributes in it that you like, and transfer them to another image. So my own work in this area started around 2001. My reasoning for this was this. In computer graphics, we at the time at least started to get quite good at modeling. And there were some fantastic images from Oliver Deuysen out who did plant ecosystems. Fantastically complex images came out of it, but he still had to assign his colors manually. And people are in general not that good at assigning colors and specifications into renderers. The images came out too saturated. I think I'll have an example of that later on. So my thinking was, can we somehow mess with an example image and apply the more sane color distribution that you find in an example image and apply it to our rendered images as a post-process? Much easier than specifying materials. Of course, the next thing I discovered is you can do this then on everything else as well. And so the example here is, well, we take a lighthouse, for instance, as the image to be manipulated. I tend to refer to that as the source image. And then the target image is, well, the image that we want to steal the color palette from. And that's a plant. I'm not that good with plants, so I don't know what species this is. And we certainly haven't given this a name either. But then it leads to the image on the right. And of course, you can then cycle through all your images that you have as examples and create other images. And it turns out that you can fairly successfully make nighttime images as well. Now, I don't know if, I think the projector has almost allows you to see that in the image on the right, the result image, you can see that there's still a smidge of red on the lighthouse left. I can see it here anyway. But it's subtle, so it may not have come out. So this is the color transfer problem. We begin with an image that we want to manipulate. We have an example image. And then somehow we get the color palette out of this to produce our desired output image. So in general, in go two images and out comes one image. That is as almost as deep as my maths will go today. Of course, two images, that's quite a lot of pixels. We need to somehow simplify this. So in general, the type of solutions that I would propose involve, well, not do this on images, but on derived values from the images. Histograms are good. At least the first incarnation of this work is really a stupidly simple algorithm. I'll get to it in a minute. There are some constraints. Basically, we want access to the color palette and transfer it to another image. And then that second innocent looking line here says no artifacts. We want no artifacts. Question though is, how do you know that there are no artifacts in an algorithm like this? What is the ground truth? How do you know that you've succeeded? And the answer is, we don't. And this is one of the problems, if you're a researcher, because if you do the first paper on this topic, you're fine. Nobody's seen it. You publish it. And I was lucky there. If you do the second paper, you have to compare your results to that previous work and then somehow argue that your results are better. But there's no metric. We have absolutely no idea how, when we succeed and when we don't. And that's now, for 10 years, been a problem. And I have not been able to solve this. So I will just argue that if the image looks good, then it's probably successful. And that's about the best we can do. And so I apologize for that. So let's have a quick look at the innards of such an algorithm. And you'll find how stupidly simple it can be. Of course, we deal with color. Every pixel is a triplet, typically. We're not doing spectral rendering or anything. So it's just triplets. That means that every pixel can be plotted in 3D space somewhere. And if you do that, you get a point cloud of pixels, as you see on the right. And we could conceivably try to manipulate all of these pixels in some way to get different results. That's complicated. I mean, I started in computer graphics with three-dimensional everything, and I have realized that that is actually too complicated for me. That's why I started doing image processing. Only two dimensions, much easier. So here, I would be back to 3D, and I thought, no. No, I can't do that. We need to make this simpler. Of course, we could treat each color channel separately. But then we have to be a little bit careful about our color space. But what we would have is three 1D problems. Now, a 1D problem, I can get my head around. That's much easier. And doing that three times, that's OK, too. But the choice of color space matters. And now we're getting to a little insight that I've gained in color, which made this whole color transfer business possible in the first place. And to me, this is just such a beautiful coincidence. Let's look at this. Any image on the left? It's mostly green, I suppose, if you look at it. But if you analyze this image, I think it was specified in sRGB color space. And if I plot the red against the green channel, so the image is normalized, so it goes from 0 to 1, you'll see that if I have a red pixel, or a pixel with a high value in the red component, it tends to have a high value in the green component as well. There is good correlation between the color channels. You can tell because this whole point cloud sits more or less on the diagonal. Next slide. Red against blue. It's spread out a little bit more, but predominantly, it's a diagonal structure. Full correlation in RGB space for completeness. Green against blue. And this is on a single image. If you do this statistically on a lot of images, it becomes a stronger effect. Lots of correlation. That means if I want to manipulate one channel, I will have to co-vari the other channels to get any sort of results. I'm back to a 3D problem. I can't, in RGB space, go for my approach of three times a simple 1D problem. I just can't. So we have to somehow reduce the correlation and the dependence between the channels. And I came across this paper, which was an eye opener for me, so I'll briefly describe it. This was Rudman's paper from 1998. And he was interested in natural image statistics. The reason that people in general are interested in natural image statistics is that they want to understand how human vision works. And the reasoning is if you understand the input to the human visual system, then it might become easier to understand the workings of the human visual system. So he was in that field, and he was also interesting color, as it happens. So he ended up taking a bunch of spectral images, converted them then to LMS cone space, roughly what the photo receptors transduce. And then he ran on a set of images, principal component analysis. And that's just a mathematical technique that rotates the data in such a way that it maximally decorrelates the three color channels. At least that's one way you can use PCA. So that point cloud that was all on the diagonal would then be rotated to be axis aligned as much as possible. And if you do that, then you have maximally decorrelated the data. Now, the beauty and the drive thing of that work is that if you do that, you find axis that have meaning. You find a different color space, a linear transform from LMS that happens to be a color opponent space. Beautiful. Humans who compute color opponents even in the retina are obviously decorrelating natural images. If you look at any random image, that's not necessarily happening. But overall, in the natural world, we decorrelate the data. Now, I would speculate, but I have no proof for it, that because you decorrelate the data, every single fiber that goes through the optic nerve to the brain carries significant information. It's efficient use of the optic nerve. And because it's a data bottleneck, it's a good way to go. So humans decorrelate their input. And he also found that in practice, of course, PCA decorrelates the input. It doesn't do more. It doesn't make independent. But it turns out that it becomes actually very close to independent, officially not, but in practice, close to independent. And I was thinking, oh, if my axes are independent, if I go to this color space, I can mess with the individual axes without having to do a 3D problem. I can do this three times in one day because the chances that I have cross talk between the channels has been reduced. It's not absent, but it's reduced. So here's the opponent space that he came up with that came out of the data. It's generally known as L alpha beta. And I thought, I go use that space. And if you go and convert to this space, my image was normalized, and then I went to log space, and then to L alpha beta from there. So this is a color opponent space, L luminance channel against alpha, which I can never remember. It's either red, green, or the yellow, blue channel, one of the two. But sorry, I can't remember. And you see, because it's a normalized image I started with, the horizontal axis goes between 0 and some negative number. And it's an opponent space, so the other channel can go positive and negative. It's a nice compact blob. There's no specific structure in it. If I were to plot this in linear space, it would be a nice elongated blob along the horizontal axis. For completeness, the other two channels, there and there. And then I was at the time mostly good at high school statistics and high school mathematics. So I thought, let's do something that is just simple and see what happens. So I thought I'd go and subtract the mean of my image. And then I scale the image by the ratio of the standard deviations in each channel. So I do this three times. Go to L alpha beta space, compute the mean and the standard deviation in each of the three channels. So I have six numbers, three in one image, three in the other. And then per channel, subtract the mean, I scale, and then I add the mean of the other image. And if you do that, then your image that you want to manipulate will have the same mean and standard deviation as your example image. Surely that ought not to work. That's too simple. But then it turns out that in enough cases, this started to work fairly nicely. Here's an example. On the left, a holiday snapshot in the middle, a van Gogh. And on the right, the result. And you have to admit that this is possible that this could be a reasonable transfer. Of course, it was a JPEG image because it was a holiday snapshot and it has amplified all the blocking artifacts. Not a problem I was trying to solve at the time. I can also make my images look old. The bottom left image is one of these plant ecosystem images. So that's a rendering. It's not a real photograph. And you see the color choices was a bit saturated. Of course, I've stuck the color out completely in this example. I'll show you a better example in a minute. The bottom image is rendered. Top image is photographed. But the rendered image had a bit of a pale sunset. So we thought, can we do this in a more interesting one? Can we make that more interesting? The answer is, yeah. Of course, I'm showing you my best results. There are plenty of cases where this just doesn't work. And one of the reasons is the following. Maybe slightly hard to see. But the shifting and scaling that we do here, the effect on the histogram is that it may be squashed between left and right, and it may be shifted to the left and to the right. That's all that happens in the three channels. That's a bit limited. You may want to get the histograms to look more similar than that. And so the next obvious step is to make sure that the histogram is equal. So the next obvious step then is to go and do that. Of course, you could just do a histogram equalization and make the histograms exactly the same. But then you tend to get a very harsh result that looks unpleasant. So I wouldn't propose that. So then we came up with effectively a scale space approach. And this was work by my wife, Tanya, and myself. I think this is the next obvious step. So the next obvious step is if you can do the first and the second moments, maybe you can do the third and the fourth moment, transfer skewing cartosis. But then it becomes not so easy because it no longer a simple scaling. You need to do it as an optimization problem. Turns out you get almost nothing by doing that, from experience. So here's the next obvious step then. Compute the histograms. And rather than shift and scale the whole histogram, we effectively do some sort of edge detection almost on the histogram. We figure out where the peaks are and the troughs and select regions on their basis and then do some processing on each of these regions. So here's one region. Here's another region. And here's a third region. And then together, all we do then is in each of these regions, we scale the bin counts up and down and stretch the bin values out according to the standard deviation within that region. And that gives you some control because you can choose whether you want large regions or smaller regions. And you can effectively on histograms do a little scale space approach. So scale space but then on histograms. I have a feeling there's other applications for this, but we haven't worked on that yet. So here's an example. And as it happens, you can do tone reproduction with this. On the left, you see two images at the top and the bottom. They are the same image. The top one is linearly scaled. And so it looks too dark. And the bottom one is tone mapped with another tone mapping operator. And then the other three images at the top are example images. And those are low dynamic range images. And they serve as input as exemplars. And so with this algorithm, you can actually get the dynamic range reduction plus the appearance transfer that is desirable in this class of algorithms. I thought it was kind of neat. And you see that there's three results that each look like their exemplar. And yet they're completely different. This algorithm is more robust than the first algorithm. Now the choice of color space to come back to this for a minute, it's actually fairly important. Here's the original intent of color transfer again. A rendered image at the top on the left and on the right, the output image transferred according to the exemplar in the bottom left. So the exemplar is more desaturated. So you'd hope that the output image is more desaturated as well. In particular, wanted the greens to look more natural. And it didn't if you do this in RGB space. And that's, I think, due to the full correlation between the channels, you get very little effect here. But in this opponent space, it worked a treat. So we've seen two color spaces used here. Alpha beta, which is color opponent space. The histogram reshaping thing was done in C-Lab. And that's a very good reason for that, which we discovered effectively slightly later. C-Lab, of course, also a color opponent space. Slightly different space, though. And there's many other opponent spaces used, as you can see here, all sorts of different places. And of course, if you can use PCA once to derive a color space, why not do it per image or per image category? And there are people who have tried that as well. And I would argue, fairly briefly, because I'm running towards 9 now, that that is actually not even the best way to do it. And let me tell you why real quickly. So computation time for color space. What color space should we use? We know it's some sort of an opponent space. We did some very preliminary research, different scene types, comparison of color spaces. We had also some scene-specific and image-specific spaces. So here's how we did it. Four different scene types. Indoors, which we encode in the following slides, is IN. Indoors. Outdoor scenes, night scenes, and the natural scenes that don't have man-made objects in them. And then we thought, well, covariance is going to be a good measure to see how the correlated the channels are. For each of the color spaces we looked at, for each of the different scene categories that we have played with. And if you look at the results, then you see that smaller bars are better than larger bars in this case. You see that the bars labeled E are lowest. That's C-lap, but then with illuminant E. And one next to it, D65. That's C-lap with illuminant D65. Seems that E correlates slightly better. But our set of images was reasonably small. And so I would call these preliminary results. The ranking, though, across the different types, fairly consistent, C-lap E, across the board, would be the predicted color space based on covariance. So that's kind of nice. So then we thought, OK, so what about these PCA-based color spaces? So now we have four image ensembles. We can compute four color spaces, one per ensemble, and see what happens then. You would expect that the color space that is created for a specific ensemble will give you the best results for that ensemble. So here is the way it's all computed. I'll skip through it for time's sake. And here are the results. So a whole mess of bars again. From left to right, the four color spaces with the results. Yeah, five minutes. Be ready in five. Four color spaces. And let loose on the different image ensembles. So for the first color space, I would expect that the MD bar, the pink one, would be lowest. Because I expect that color space to decorrelate that ensemble most. But it almost does. But that pattern doesn't go along through the other spaces. So again, we get highly consistent results, which I didn't expect. I thought if you do PCA for an ensemble, then surely that PCA space for that ensemble is going to be most correlated. It wasn't. I didn't expect that, that is a little bit counterintuitive. The covariance is lower than for the standard color spaces, though. And so we have gained something in terms of covariance. The question is, does that translate to quality of color transfer? So we did a second pilot study. And really, this is a pilot study, because the number of participants was too small. Two participants. Here's one. And the other one is in the audience over there. We looked at a whole bunch of images. And did effectively a very simple preference study. Do we like the color transfer or not? And so we did color transfer. We computed them for a whole bunch of images in a whole bunch of different spaces. And then looked at them, limited the size of the data set to preserve our sanity. There's enough color spaces as it is. And then here are the results, of course. Here it's a ranking study. So we want higher bars rather than lower bars, because that gives a higher rank. And so to summarize these results, no, I can't summarize them just yet, you see again that C-Lab E is doing rather well. And then there's also the results for the four ensemble-based PCAs. And then the two on the right are the image specific ensembles. Of course, if you have two images as inputs, you can run PCA on the first image or on the second image, the source or the target. And so we have two different versions of that as well. And so to summarize all of these results real quick, we found that even in the presence of these dedicated PCA spaces, C-Lab E still gives you the best overall ranking based on my and my partner's opinion. PCA2, the conversion on a per image base of the target image, gives the second best result. And then there's C-Lab and then that alpha beta space, with which it all started, ranks number four. And then after that is everything else, with lower, lower rankings. So that leads me to about the end. We need to do more of this work to understand better what is the best way to go about things. I suppose there are some caveats to talk about. The sample size, only five images per sample size in the pilot study. That's why I'm still standing up straight. Because we had more, I would have lost my sanity. It's quite a lot of work. I mentioned that I would come back to applications. It's not just toy applications anymore. In the movie industry, it seems that color transfer is used to make night images from day images as post process. Much easier to capture day image than a night image, and then you can process it. Stereo pair matching. If you record stereo pairs, the system, the two images are recorded through possibly different lens systems. They're not quite the same. These can be transferred to be made the same. Similar with panorama stitching. I've seen applications in video compression, because you can simply remove the color altogether. And then we introduce it later. We've done dynamic range reduction. And strangely enough, remote sensing seems to be using color transfer type techniques a lot. So we're not that many example images in the second half of the talk. I hope you enjoyed it anyway. I like this, because it's simply messing with images in a nice way. And I found that it has applications. So that's two good things all in one go. I think it's time to sign off. And thank you for your attention. Thank you very much. Thank you very much. Just tumble off stage. We have time for a couple of questions. Well, that was a really good presentation. I like the practical way that you're engaging with the images and trying things. But am I right in thinking in this study you've done, the source images were sRGB color space and JPEG encoded? Yes. And then you're looking quite detail at the decorrelation of the different components. But I think a lot of that's already been done or been lost in the encoding. To do this properly, you need to go back to linear spectral images, don't you? And then do the transformation to LMS and so forth. So if I recall, these images were all high dynamic range images to begin with. So not spectral, but full high dynamic range. Yeah, but I'm talking about this. You need a multi-spectral images, don't you? Just talk with something that each pixel gives you the best representation of the spectral power distribution of the original scene at that point. Yeah. I think you would gain insights from using spectral images to begin with. Could I suggest that you took some images from, for example, the Foster and Nashi Mento hyperspectral image set as the source images and then did all this again and see how that turns out. That's a very good idea. Yeah, I would like to try this. Yeah, thank you very much. Any other questions? Good morning. Do you have any idea how to go from 3 times 1D to 1 times 3D space? So what was the good? Do you have any idea how leave the 3 times 1D processing and to come to processing in 3D space? Yeah, there are various examples of that. And I'm sorry that I have bias. They talk very much into the direction of the stuff that I know best. But there is a whole collection of papers out there that do this in 3D. I think the most obvious way to go about it is to use Gaussian mixture models. So you effectively find clusters in 3D space and fit ellipsoids to them. So if I have a blob here and a blob here, you can make an ellipsoid around it. And that's characterized by a point in space, the center, and then a three axis. And then you have an ellipsoid. And if you match these with corresponding ellipsoids in the second image, then you can warp and shift and scale each blob separately. So if I have a lot of dark greens and a whole lot of bright blues somewhere, then they will be transformed separately. I think it's the most obvious way to do it. Sorry, I forgot the authors, but that's an existing piece of work. Of course, in those cases, the color space doesn't matter because you align your blobs with the data anyway in any space. Does that help? OK, thank you. Well, let's thank our speaker once again. Thank you.
 So yeah, that's indeed what I want to talk about.  But first I'd like to thank the organizers for inviting me.  It's a beautiful place, Amsterdam as a whole,  and in particular this venue.  And I think it's the nicest place  that I've given a presentation in so far.  So it's an honor to be here.  And also thank you to the audience  because you had to drag yourself away fairly early  from the warmth of your beds.  So thank you for showing up.  If you have any compelling questions,  there is, I believe, some time at the end.  But feel free to interrupt.  I'm fully interruptable.  So shout or inflate your orange balloon  if you have a question.  All is good.  So before I go too fast, I have sort of made my career out  of high dynamic range imaging.  And in particular, tone reproduction.  And so in that field, we want to be as precise as possible.  And we want to reduce the dynamic range of the real world  to something that our limited displays can display.  And so that research gave me the need  to understand color a little bit better.  And so this is how I wandered into the field of color science,  almost accidental.  But on the side, I'm interested in the precise color science.  But I'm also interested in just having a bit of fun  and just messing with images a bit.  And that is what this talk is about.  So there's not the same level of precision here.  But there is, I hope, a higher degree of amusement  and entertainment.  And also, I hope you enjoy this type of research as well.  So if I can try forward to the next slide,  I'll aim for that there.  For most algorithms, the input, if you're in imaging at least,  is an image.  And then some parameters, sliders if you like.  And you drag the sliders until you're happy.  If you want to do white balancing,  you begin by an image.  And then you say, I want that to become a D65 or an E or an A  or whatever it is you like.  And that's then your parameter adjustment.  And that's all good and fine.  But sometimes you might want to do things to images that  are not so easy to specify.  Sometimes you might want to change an image according  to, well, some other image that you may have found.  And if there's another image somewhere, the internet is full  of them, of images that I like and that are better than my  own images, I might be able to use these images as an example.  And so I started messing with example images, largely for the  purposes of improving my holiday snapshots.  Turns out that there is slightly more serious  applications to it as well.  And I'll get to that later on.  And it's sort of given rise, I think, to a little sub  area, certainly in graphics imaging, whereby you can take  an example image, find some attributes in it that you like,  and transfer them to another image.  So my own work in this area started around 2001.  My reasoning for this was this.  In computer graphics, we at the time at least started to get  quite good at modeling.  And there were some fantastic images from Oliver Deuysen out  who did plant ecosystems.  Fantastically complex images came out of it, but he still  had to assign his colors manually.  And people are in general not that good at assigning colors  and specifications into renderers.  The images came out too saturated.  I think I'll have an example of that later on.  So my thinking was, can we somehow mess with an example  image and apply the more sane color distribution that you  find in an example image and apply it to our rendered  images as a post-process?  Much easier than specifying materials.  Of course, the next thing I discovered is you can do this  then on everything else as well.  And so the example here is, well, we take a lighthouse,  for instance, as the image to be manipulated.  I tend to refer to that as the source image.  And then the target image is, well, the image that we want  to steal the color palette from.  And that's a plant.  I'm not that good with plants, so I don't know what species  this is.  And we certainly haven't given this a name either.  But then it leads to the image on the right.  And of course, you can then cycle through all your images  that you have as examples and create other images.  And it turns out that you can fairly successfully make  nighttime images as well.  Now, I don't know if, I think the projector has almost  allows you to see that in the image on the right, the result  image, you can see that there's still a smidge of red on  the lighthouse left.  I can see it here anyway.  But it's subtle, so it may not have come out.  So this is the color transfer problem.  We begin with an image that we want to manipulate.  We have an example image.  And then somehow we get the color palette out of this to  produce our desired output image.  So in general, in go two images and  out comes one image.  That is as almost as deep as my maths will go today.  Of course, two images, that's quite a lot of pixels.  We need to somehow simplify this.  So in general, the type of solutions that I would propose  involve, well, not do this on images, but on derived values  from the images.  Histograms are good.  At least the first incarnation of this work is really a  stupidly simple algorithm.  I'll get to it in a minute.  There are some constraints.  Basically, we want access to the color palette and transfer  it to another image.  And then that second innocent looking line here says no  artifacts.  We want no artifacts.  Question though is, how do you know that there are no  artifacts in an algorithm like this?  What is the ground truth?  How do you know that you've succeeded?  And the answer is, we don't.  And this is one of the problems, if you're a  researcher, because if you do the first paper on this  topic, you're fine.  Nobody's seen it.  You publish it.  And I was lucky there.  If you do the second paper, you have to compare your results  to that previous work and then somehow argue that your  results are better.  But there's no metric.  We have absolutely no idea how, when we succeed and when we  don't.  And that's now, for 10 years, been a problem.  And I have not been able to solve this.  So I will just argue that if the image looks good, then it's  probably successful.  And that's about the best we can do.  And so I apologize for that.  So let's have a quick look at the innards of such an  algorithm.  And you'll find how stupidly simple it can be.  Of course, we deal with color.  Every pixel is a triplet, typically.  We're not doing spectral rendering or anything.  So it's just triplets.  That means that every pixel can be plotted in 3D space  somewhere.  And if you do that, you get a point cloud of pixels, as you  see on the right.  And we could conceivably try to manipulate all of these  pixels in some way to get different results.  That's complicated.  I mean, I started in computer graphics with three-dimensional  everything, and I have realized that that is actually too  complicated for me.  That's why I started doing image processing.  Only two dimensions, much easier.  So here, I would be back to 3D, and I thought, no.  No, I can't do that.  We need to make this simpler.  Of course, we could treat each color channel separately.  But then we have to be a little bit careful about our color  space.  But what we would have is three 1D problems.  Now, a 1D problem, I can get my head around.  That's much easier.  And doing that three times, that's OK, too.  But the choice of color space matters.  And now we're getting to a little insight that I've  gained in color, which made this whole color transfer  business possible in the first place.  And to me, this is just such a beautiful coincidence.  Let's look at this.  Any image on the left?  It's mostly green, I suppose, if you look at it.  But if you analyze this image, I think it was specified in  sRGB color space.  And if I plot the red against the green channel, so the  image is normalized, so it goes from 0 to 1, you'll see  that if I have a red pixel, or a pixel with a high value in  the red component, it tends to have a high value in the  green component as well.  There is good correlation between the color channels.  You can tell because this whole point cloud sits more or less  on the diagonal.  Next slide.  Red against blue.  It's spread out a little bit more, but predominantly, it's  a diagonal structure.  Full correlation in RGB space for completeness.  Green against blue.  And this is on a single image.  If you do this statistically on a lot of images, it becomes a  stronger effect.  Lots of correlation.  That means if I want to manipulate one channel, I will  have to co-vari the other channels to get any sort of  results.  I'm back to a 3D problem.  I can't, in RGB space, go for my approach of three times a  simple 1D problem.  I just can't.  So we have to somehow reduce the correlation and the  dependence between the channels.  And I came across this paper, which was an eye opener for  me, so I'll briefly describe it.  This was Rudman's paper from 1998.  And he was interested in natural image statistics.  The reason that people in general are interested in  natural image statistics is that they want to understand how  human vision works.  And the reasoning is if you understand the input to the  human visual system, then it might become easier to  understand the workings of the human visual system.  So he was in that field, and he was also interesting color,  as it happens.  So he ended up taking a bunch of spectral images, converted  them then to LMS cone space, roughly what the photo  receptors transduce.  And then he ran on a set of images, principal  component analysis.  And that's just a mathematical technique that  rotates the data in such a way that it maximally  decorrelates the three color channels.  At least that's one way you can use PCA.  So that point cloud that was all on the diagonal would then  be rotated to be axis aligned as much as possible.  And if you do that, then you have maximally  decorrelated the data.  Now, the beauty and the drive thing of that work is that if  you do that, you find axis that have meaning.  You find a different color space, a linear transform  from LMS that happens to be a color opponent space.  Beautiful.  Humans who compute color opponents even in the retina  are obviously decorrelating natural images.  If you look at any random image, that's not necessarily  happening.  But overall, in the natural world, we decorrelate the data.  Now, I would speculate, but I have no proof for it, that  because you decorrelate the data, every single fiber that  goes through the optic nerve to the brain  carries significant information.  It's efficient use of the optic nerve.  And because it's a data bottleneck, it's a good way to  go.  So humans decorrelate their input.  And he also found that in practice, of course, PCA  decorrelates the input.  It doesn't do more.  It doesn't make independent.  But it turns out that it becomes actually very close to  independent, officially not, but in practice, close to  independent.  And I was thinking, oh, if my axes are independent, if I go  to this color space, I can mess with the individual axes  without having to do a 3D problem.  I can do this three times in one day because the chances  that I have cross talk between the channels has been  reduced.  It's not absent, but it's reduced.  So here's the opponent space that he came up with that  came out of the data.  It's generally known as L alpha beta.  And I thought, I go use that space.  And if you go and convert to this space, my image was  normalized, and then I went to log space, and then to L  alpha beta from there.  So this is a color opponent space, L luminance channel  against alpha, which I can never remember.  It's either red, green, or the yellow, blue channel, one of  the two.  But sorry, I can't remember.  And you see, because it's a normalized image I started  with, the horizontal axis goes between 0 and some negative  number.  And it's an opponent space, so the other channel can go  positive and negative.  It's a nice compact blob.  There's no specific structure in it.  If I were to plot this in linear space, it would be a  nice elongated blob along the horizontal axis.  For completeness, the other two channels, there and there.  And then I was at the time mostly good at high school  statistics and high school mathematics.  So I thought, let's do something that is just simple  and see what happens.  So I thought I'd go and subtract the mean of my image.  And then I scale the image by the ratio of the standard  deviations in each channel.  So I do this three times.  Go to L alpha beta space, compute the mean and the  standard deviation in each of the three channels.  So I have six numbers, three in one image, three in the  other.  And then per channel, subtract the mean, I scale, and  then I add the mean of the other image.  And if you do that, then your image that you want to  manipulate will have the same mean and standard deviation as  your example image.  Surely that ought not to work.  That's too simple.  But then it turns out that in enough cases, this started to  work fairly nicely.  Here's an example.  On the left, a holiday snapshot in the middle, a van Gogh.  And on the right, the result.  And you have to admit that this is possible that this  could be a reasonable transfer.  Of course, it was a JPEG image because it was a holiday  snapshot and it has amplified all the blocking artifacts.  Not a problem I was trying to solve at the time.  I can also make my images look old.  The bottom left image is one of these plant  ecosystem images.  So that's a rendering.  It's not a real photograph.  And you see the color choices was a bit saturated.  Of course, I've stuck the color out completely in this  example.  I'll show you a better example in a minute.  The bottom image is rendered.  Top image is photographed.  But the rendered image had a bit of a pale sunset.  So we thought, can we do this in a more interesting one?  Can we make that more interesting?  The answer is, yeah.  Of course, I'm showing you my best results.  There are plenty of cases where this just doesn't work.  And one of the reasons is the following.  Maybe slightly hard to see.  But the shifting and scaling that we do here, the effect  on the histogram is that it may be squashed between left and  right, and it may be shifted to the left and to the right.  That's all that happens in the three channels.  That's a bit limited.  You may want to get the histograms to look more  similar than that.  And so the next obvious step is to make sure that the  histogram is equal.  So the next obvious step then is to go and do that.  Of course, you could just do a histogram equalization and  make the histograms exactly the same.  But then you tend to get a very harsh result that  looks unpleasant.  So I wouldn't propose that.  So then we came up with effectively a scale space  approach.  And this was work by my wife, Tanya, and myself.  I think this is the next obvious step.  So the next obvious step is if you can do the first and the  second moments, maybe you can do the third and the fourth  moment, transfer skewing cartosis.  But then it becomes not so easy because it no longer a  simple scaling.  You need to do it as an optimization problem.  Turns out you get almost nothing by doing that, from  experience.  So here's the next obvious step then.  Compute the histograms.  And rather than shift and scale the whole histogram, we  effectively do some sort of edge detection almost on the  histogram.  We figure out where the peaks are and the troughs and select  regions on their basis and then do some processing on each  of these regions.  So here's one region.  Here's another region.  And here's a third region.  And then together, all we do then is in each of these  regions, we scale the bin counts up and down and stretch  the bin values out according to the standard deviation  within that region.  And that gives you some control because you can choose  whether you want large regions or smaller regions.  And you can effectively on histograms do a little scale  space approach.  So scale space but then on histograms.  I have a feeling there's other applications for this, but  we haven't worked on that yet.  So here's an example.  And as it happens, you can do tone reproduction with this.  On the left, you see two images at the top and the  bottom.  They are the same image.  The top one is linearly scaled.  And so it looks too dark.  And the bottom one is tone mapped with another tone  mapping operator.  And then the other three images at the top are example  images.  And those are low dynamic range images.  And they serve as input as exemplars.  And so with this algorithm, you can actually get the  dynamic range reduction plus the appearance transfer that  is desirable in this class of algorithms.  I thought it was kind of neat.  And you see that there's three results that each look like  their exemplar.  And yet they're completely different.  This algorithm is more robust than the first algorithm.  Now the choice of color space to come back to this for a  minute, it's actually fairly important.  Here's the original intent of color transfer again.  A rendered image at the top on the left and on the right,  the output image transferred according to the  exemplar in the bottom left.  So the exemplar is more desaturated.  So you'd hope that the output image is more  desaturated as well.  In particular, wanted the greens to look more natural.  And it didn't if you do this in RGB space.  And that's, I think, due to the full correlation between  the channels, you get very little effect here.  But in this opponent space, it worked a treat.  So we've seen two color spaces used here.  Alpha beta, which is color opponent space.  The histogram reshaping thing was done in C-Lab.  And that's a very good reason for that, which we discovered  effectively slightly later.  C-Lab, of course, also a color opponent space.  Slightly different space, though.  And there's many other opponent spaces used, as you can see  here, all sorts of different places.  And of course, if you can use PCA once to derive a color  space, why not do it per image or per image category?  And there are people who have tried that as well.  And I would argue, fairly briefly, because I'm running  towards 9 now, that that is actually not even the best  way to do it.  And let me tell you why real quickly.  So computation time for color space.  What color space should we use?  We know it's some sort of an opponent space.  We did some very preliminary research, different scene  types, comparison of color spaces.  We had also some scene-specific and image-specific  spaces.  So here's how we did it.  Four different scene types.  Indoors, which we encode in the following slides, is IN.  Indoors.  Outdoor scenes, night scenes, and the natural scenes that  don't have man-made objects in them.  And then we thought, well, covariance is going to be a  good measure to see how the correlated the channels are.  For each of the color spaces we looked at, for each of the  different scene categories that we have played with.  And if you look at the results, then you see that smaller  bars are better than larger bars in this case.  You see that the bars labeled E are lowest.  That's C-lap, but then with illuminant E. And one next to it,  D65.  That's C-lap with illuminant D65.  Seems that E correlates slightly better.  But our set of images was reasonably small.  And so I would call these preliminary results.  The ranking, though, across the different types, fairly  consistent, C-lap E, across the board, would be the predicted  color space based on covariance.  So that's kind of nice.  So then we thought, OK, so what about these PCA-based  color spaces?  So now we have four image ensembles.  We can compute four color spaces, one per ensemble, and  see what happens then.  You would expect that the color space that is created for a  specific ensemble will give you the best results for that  ensemble.  So here is the way it's all computed.  I'll skip through it for time's sake.  And here are the results.  So a whole mess of bars again.  From left to right, the four color spaces with the results.  Yeah, five minutes.  Be ready in five.  Four color spaces.  And let loose on the different image ensembles.  So for the first color space, I would expect that the MD bar,  the pink one, would be lowest.  Because I expect that color space to decorrelate that  ensemble most.  But it almost does.  But that pattern doesn't go along through the other spaces.  So again, we get highly consistent results, which I  didn't expect.  I thought if you do PCA for an ensemble, then surely that  PCA space for that ensemble is going to be most  correlated.  It wasn't.  I didn't expect that, that is a little bit counterintuitive.  The covariance is lower than for the standard color spaces,  though.  And so we have gained something in terms of covariance.  The question is, does that translate to quality of color  transfer?  So we did a second pilot study.  And really, this is a pilot study, because the number of  participants was too small.  Two participants.  Here's one.  And the other one is in the audience over there.  We looked at a whole bunch of images.  And did effectively a very simple preference study.  Do we like the color transfer or not?  And so we did color transfer.  We computed them for a whole bunch of images in a whole  bunch of different spaces.  And then looked at them, limited the size of the data set  to preserve our sanity.  There's enough color spaces as it is.  And then here are the results, of course.  Here it's a ranking study.  So we want higher bars rather than lower bars, because that  gives a higher rank.  And so to summarize these results, no, I can't summarize  them just yet, you see again that C-Lab E is doing  rather well.  And then there's also the results for the four  ensemble-based PCAs.  And then the two on the right are the image specific  ensembles.  Of course, if you have two images as inputs, you can run  PCA on the first image or on the second image, the source  or the target.  And so we have two different versions of that as well.  And so to summarize all of these results real quick, we  found that even in the presence of these dedicated PCA  spaces, C-Lab E still gives you the best overall ranking  based on my and my partner's opinion.  PCA2, the conversion on a per image base of the target  image, gives the second best result.  And then there's C-Lab and then that alpha beta space,  with which it all started, ranks number four.  And then after that is everything else, with lower,  lower rankings.  So that leads me to about the end.  We need to do more of this work to understand better what is  the best way to go about things.  I suppose there are some caveats to talk about.  The sample size, only five images per sample size in the  pilot study.  That's why I'm still standing up straight.  Because we had more, I would have lost my sanity.  It's quite a lot of work.  I mentioned that I would come back to applications.  It's not just toy applications anymore.  In the movie industry, it seems that color transfer is used  to make night images from day images as post process.  Much easier to capture day image than a night image, and  then you can process it.  Stereo pair matching.  If you record stereo pairs, the system, the two images are  recorded through possibly different lens systems.  They're not quite the same.  These can be transferred to be made the same.  Similar with panorama stitching.  I've seen applications in video compression, because you  can simply remove the color altogether.  And then we introduce it later.  We've done dynamic range reduction.  And strangely enough, remote sensing seems to be using  color transfer type techniques a lot.  So we're not that many example images in the second half of  the talk.  I hope you enjoyed it anyway.  I like this, because it's simply messing with images in a  nice way.  And I found that it has applications.  So that's two good things all in one go.  I think it's time to sign off.  And thank you for your attention.  Thank you very much.  Thank you very much.  Just tumble off stage.  We have time for a couple of questions.  Well, that was a really good presentation.  I like the practical way that you're engaging with the  images and trying things.  But am I right in thinking in this study you've done, the  source images were sRGB color space and JPEG encoded?  Yes.  And then you're looking quite detail at the decorrelation of  the different components.  But I think a lot of that's already been done or been  lost in the encoding.  To do this properly, you need to go back to linear spectral  images, don't you?  And then do the transformation to LMS and so forth.  So if I recall, these images were all high dynamic range  images to begin with.  So not spectral, but full high dynamic range.  Yeah, but I'm talking about this.  You need a multi-spectral images, don't you?  Just talk with something that each pixel gives you the best  representation of the spectral power distribution of the  original scene at that point.  Yeah.  I think you would gain insights from using spectral images  to begin with.  Could I suggest that you took some images from, for  example, the Foster and Nashi Mento hyperspectral image set  as the source images and then did all this again and see how  that turns out.  That's a very good idea.  Yeah, I would like to try this.  Yeah, thank you very much.  Any other questions?  Good morning.  Do you have any idea how to go from 3 times 1D to 1 times  3D space?  So what was the good?  Do you have any idea how leave the 3 times 1D processing and  to come to processing in 3D space?  Yeah, there are various examples of that.  And I'm sorry that I have bias.  They talk very much into the direction of the stuff that I  know best.  But there is a whole collection of papers out there that do  this in 3D.  I think the most obvious way to go about it is to use Gaussian  mixture models.  So you effectively find clusters in 3D space and fit  ellipsoids to them.  So if I have a blob here and a blob here, you can make an  ellipsoid around it.  And that's characterized by a point in space, the center,  and then a three axis.  And then you have an ellipsoid.  And if you match these with corresponding ellipsoids in the  second image, then you can warp and shift and scale each  blob separately.  So if I have a lot of dark greens and a whole lot of  bright blues somewhere, then they will be  transformed separately.  I think it's the most obvious way to do it.  Sorry, I forgot the authors, but that's an  existing piece of work.  Of course, in those cases, the color space doesn't matter  because you align your blobs with the data anyway in any  space.  Does that help?  OK, thank you.  Well, let's thank our speaker once again.  Thank you.
example-based_image_manipulation_0_263.png example-based_image_manipulation_264_3155.png example-based_image_manipulation_3156_5351.png example-based_image_manipulation_5352_8783.png example-based_image_manipulation_8784_9899.png example-based_image_manipulation_9900_11495.png example-based_image_manipulation_11496_11747.png example-based_image_manipulation_11748_14207.png example-based_image_manipulation_14208_15275.png example-based_image_manipulation_15276_16463.png example-based_image_manipulation_16464_16739.png example-based_image_manipulation_16740_21083.png example-based_image_manipulation_21084_21431.png example-based_image_manipulation_21432_24131.png example-based_image_manipulation_24132_24863.png example-based_image_manipulation_24864_25811.png example-based_image_manipulation_25812_26735.png example-based_image_manipulation_26736_26987.png example-based_image_manipulation_26988_27035.png example-based_image_manipulation_27036_27131.png example-based_image_manipulation_27132_28091.png example-based_image_manipulation_28092_29567.png example-based_image_manipulation_29568_29615.png example-based_image_manipulation_29616_31031.png example-based_image_manipulation_31032_32207.png example-based_image_manipulation_32208_32807.png example-based_image_manipulation_32808_33011.png example-based_image_manipulation_33012_33659.png example-based_image_manipulation_33660_33839.png example-based_image_manipulation_33840_34163.png example-based_image_manipulation_34164_34967.png example-based_image_manipulation_34968_35867.png example-based_image_manipulation_35868_36215.png example-based_image_manipulation_36216_36563.png example-based_image_manipulation_36564_36815.png example-based_image_manipulation_36816_36863.png example-based_image_manipulation_36864_36959.png example-based_image_manipulation_36960_38015.png example-based_image_manipulation_38016_38363.png example-based_image_manipulation_38364_38867.png example-based_image_manipulation_38868_39791.png example-based_image_manipulation_39792_40727.png example-based_image_manipulation_40728_40979.png example-based_image_manipulation_40980_42059.png example-based_image_manipulation_42060_42359.png example-based_image_manipulation_42360_42791.png example-based_image_manipulation_42792_44795.png example-based_image_manipulation_44796_48215.png example-based_image_manipulation_48216_50255.png example-based_image_manipulation_50256_50297.png