Temporally consistent disparity estimation using PCA dual cross-bilateral grid
CC Attribution - NoDerivatives 2.0 UK: England & Wales:
You are free to use, copy, distribute and transmit the work or content in unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Zhu, Jin
Vandewalle, Patrick
Haan, Gerard de
Disparity estimation has been extensively investigated in recent years. Though several algorithms have been reported to achieve excellent performance on the Middlebury website, few of them reach a satisfying balance between accuracy and efficiency, and few of them consider the problem of temporal coherence. In this paper, we introduce a novel disparity estimation approach, which improves the accuracy for static images and the temporal coherence for videos. For static images, the proposed approach is inspired by the adaptive support weight method proposed by Yoon et al. and the dual-cross-bilateral grid introduced by Richardt et al. Principal component analysis (PCA) is used to reduce the color dimensionality in the cost aggregation step. This simple, but efficient technique helps the proposed method to be comparable to the best local algorithms on the Middlebury website, while still allowing real-time implementation. A computationally efficient method for temporally consistent behavior is also proposed. Moreover, in the user evaluation experiment, the proposed temporal approach achieves the best overall user experience among the selected comparison algorithms. 
 Okay, hello everyone and thanks for coming to my talk. I will present you our work on temporarily consistent disparity estimation using a PCA dual cross bilateral grid approach. Now that's a whole mouthful of words but through the presentation I hope I'll be able to clear that up. This is work done by Jin Zhu while she did her graduation project at Philips and was done together with Kierar Tahan and myself. So by disparity we mean the shift between the left and right image of a stereo pair that you get when you take two pictures from a slightly different point of view and there's a strong resemblance with the depth maps given by the previous presenter. Here they actually represent actual pixel shifts between the left and right image but just as in the previous presentation you can see that white surfaces come more to the foreground, black surfaces are more in the background. Now why would we need disparity maps if we already have stereo information available? Well it can be for multiple targets. One of them is if you have multi-view displays like for example most auto stereoscopic displays there are multi-view and then you need to generate like a multitude of views both between the original stereo pair and outside the original stereo pair and then you would typically use the depth information. Similarly if you want to adjust the depth effect on stereoscopic screens for many movies I think the depth effect for me personally is often too strong when I watch them on a TV and I would like to be able to reduce it. That's also an application where you would use disparity maps. Now a large number of disparity estimation algorithms already exist and many of them are gathered and evaluated on the Middlebury website which is well known to many researchers in the field. The best algorithms there now achieve an accuracy of around 4% of bad pixels which is quite good. On the other hand if you get to those numbers it often starts becoming difficult I think to quantify the quality in terms of a percentage of bad pixels. But yeah it is a measure and you have ground truth data available for that for those data so that's the best we can do as evaluation for now I think. Another indication that the quality of those algorithms is achieving an acceptable level is that more and more algorithms get implemented in hardware and are released in actual products. Many stereoscopic TVs now allow you to change the depth effect which means that some kind of disparity estimation is done in most of them. On the other hand my... So why would we still do yet another disparity estimation algorithm? My impression is that the quality is still lacking in some respects. One of them is as I said the location of those bad pixels is starting to get very important. If you do a poor estimation within let's say a blue sky it doesn't matter that much. If you do a poor estimation on the edges of an object you immediately see the effect if you watch the generated views out of that. And then this Middlebury dataset is a set of still images and most people work only on still images. I was very happy to see the previous speaker also talk about video. I think the temporal consistency in video disparity maps is very important because otherwise you start seeing flickering and a lot of artifacts that are not visible if you just show a number of frames one by one. So if you look into those 150 methods that are evaluated now on the Middlebury website you can... They are generally classified in global and local methods. And I will look here mostly into the local methods because they are generally more easily transformed into something that can be implemented in real time. Most of those algorithms are subdivided in four main parts. First there is some calculation of an initial cost which can be seen here below. This is... This leads to an initial cost volume where you have a matching cost for every pixel and every potential disparity value. And then those data are aggregated or filtered to get somewhat more smoothness and try to extend certain information to the neighboring pixels. And then from there the disparity map is calculated. So until here we work with a cost volume. And then in the third step we actually reduce this to a two-dimensional disparity map. And then after that most algorithms still perform some refinement step which could do for example some correction in areas that are occluded around edges or similar things. So if we would just from the initial cost go directly to a disparity computation we would get an image like this one which I think you'll all agree is by far not smooth enough to get close to this ground truth depth map. So let us look more closely into a few existing approaches on which we based our research. So the first one is the adaptive support rate approach by Jun and Kwon. And they for the aggregation step they use a bilateral filter kind of approach where they filter costs values from neighboring pixels based on both the color difference with the central pixel and the distance to the central pixel. They use a weight for the left image, a weight for the right image and then these are the initial costs that were calculated. Based on this work Richard and others have developed in 2010 a dual cross bilateral grid approach where their main goal was to get to a real time implementation of something similar to the previous approach. What they did for that is they changed the cost or the aggregation function somewhat by using a number of Gaussian functions. This one is the luminance difference in the left image between your two pixels. This is the luminance difference in the right image between the two pixels and then there's the distance to the center pixel and again your original cost. Now once they got to this formulation it lends itself very well to applying a cross bilateral grid implementation and that can be easily translated to processing on a graphics card and therefore can be achieved in real time. So they get to a four dimensional cross bilateral grid where the four dimensions are for the horizontal and vertical position and for the luminance difference in left and right image. Then we get to our approach. In our approach for the initial cost calculation I'll go down these four steps that I indicated before. We used the absolute color difference as well as the gradient difference and made a weight at some of those two because they seem to contribute somewhat different information. In the gradient difference you get some more edge information and that's exactly where you can also do more precise matches. Then for the cost aggregation we use a similar function as the dual cross bilateral grid approach but instead of having the luminance difference in the left and the right image where it was clearly seen both by us and some other researchers that the luminance difference from the right image doesn't contribute that much anymore. We wanted to take advantage of this four dimension in the bilateral grid to use some other data and what we did is we performed a principal component analysis on our color left image and we used the first two principal components of the left image disregarding the luminance and color data of the right image for our cost aggregation and used that for cross bilateral grid. Then afterwards for the disparity computation we used the winner takes all approach meaning that for every pixel value you just choose the disparity value with the lowest cost and you perform the quadratic interpolation to get some sub pixel precision out of it. Then afterwards in the refinement step we also did some existing approaches first performing a left right consistency check and filling in those occlusions and afterwards we performed a weighted medium filter. As I said I was very concerned also with temporal behavior and instead of filtering our disparity maps we felt that it was much better or the results were much better when we performed the recursive filter on our bilateral grid itself. So what we did is we for to calculate our grid for the current frame we weighted the bilateral grid for the previous frame and added to that the new data that are available in the current frame and in that way automatically got some more consistency and while still keeping the edge alignment and so on which are very important in our very nice features of the cross bilateral grid. Now there was one caveat in there because as I said we are using a PCA for the dimension of our bilateral grid. If you just blindly update at every frame you are kind of combining different unrelated data because your PCA basis changes from frame to one frame to another. So what we did is we keep track of how much change there is in the PCA analysis and only when there was a larger change than let's say 5% we updated our grid with the new data from the new basis and otherwise we kept on working with the old basis because the changes would be minimal anyway. So that brings me to some results and here we compare on the left the dual cross bilateral grid approach in the center the proposed approach that we developed and on the right the adaptive support weight approach which was the first one that I presented and I think both from the images here at the bottom and from the counts of bad pixels you will be able to see that our approach while not exactly matching the adaptive support weight I think it gets pretty close to that and it's definitely way better than the performance of this approach while our approach still keeps roughly the same computational complexity. In my opinion the differences that we still have here between those approaches are at the level where we should start using other measures as well to distinguish between the different approaches. Here are some more images where we can see again the difference the different approaches on each of the images. I think you'll probably agree that these results look pretty smooth and pretty similar to the adaptive support weight approach and then as we also emphasize this video behavior we also performed an experiment on two videos unfortunately I don't have the video data themselves with me but we did one animation movie or one segment of an animation movie of about one minute and one piece of some live action video and we asked a number of viewers to watch it on an autostereoscopic display and to give their feedback on three different topics one is the overall 3D impression which is given here to the left second is the temporal coherence and third one are the depth artifacts and we asked them to evaluate it and the blue color is labeled excellent red is good and it goes down all the way to the purple here. So I think my impression from these results is that for the animation movie all videos performed fairly equal and fairly well. For the real live action video you see more variation and there you can see that our algorithm performed somewhat better and gave somewhat more stable results than most of the other algorithms. So to conclude I have presented you our new automated disparity estimation algorithm which is based on the dual cross bilateral grid approach by Richard and others and on the adaptive support weight method. One of the main differences is that we changed the left and right luminance image by the PCA on the left image where we keep those two principal components and I hope I've been able to convince you that we achieved pretty good results both on still images and video while still keeping something which while we did not implement it in real time ourselves should be feasible in real time. So that concludes my talk. So question or comment please. Hi, David Carrigan from Trinity College Dublin. I just want to ask how does your temporal consistency framework account for local motion in a video? So how is it robust to local motion? It is somewhat inherently dealt with in the bilateral grid approach in the sense that if there are some let's say if there it is an issue if you have very fast motion but if it's kind of smooth motion not going too fast then typically by applying some filtering on your bilateral grid you already spread the disparity values to the neighboring areas of the image and you can extract those disparity data already. Maybe one more? No? Present to the speaker. Thank you. Thank you. Thank you, dangit. Thank you.
 Okay, hello everyone and thanks for coming to my talk.  I will present you our work on temporarily consistent disparity estimation using a PCA  dual cross bilateral grid approach.  Now that's a whole mouthful of words but through the presentation I hope I'll be able  to clear that up.  This is work done by Jin Zhu while she did her graduation project at Philips and was  done together with Kierar Tahan and myself.  So by disparity we mean the shift between the left and right image of a stereo pair  that you get when you take two pictures from a slightly different point of view and there's  a strong resemblance with the depth maps given by the previous presenter.  Here they actually represent actual pixel shifts between the left and right image but  just as in the previous presentation you can see that white surfaces come more to the foreground,  black surfaces are more in the background.  Now why would we need disparity maps if we already have stereo information available?  Well it can be for multiple targets.  One of them is if you have multi-view displays like for example most auto stereoscopic displays  there are multi-view and then you need to generate like a multitude of views both between  the original stereo pair and outside the original stereo pair and then you would typically  use the depth information.  Similarly if you want to adjust the depth effect on stereoscopic screens for many movies  I think the depth effect for me personally is often too strong when I watch them on a  TV and I would like to be able to reduce it.  That's also an application where you would use disparity maps.  Now a large number of disparity estimation algorithms already exist and many of them  are gathered and evaluated on the Middlebury website which is well known to many researchers  in the field.  The best algorithms there now achieve an accuracy of around 4% of bad pixels which is quite  good.  On the other hand if you get to those numbers it often starts becoming difficult I think  to quantify the quality in terms of a percentage of bad pixels.  But yeah it is a measure and you have ground truth data available for that for those data  so that's the best we can do as evaluation for now I think.  Another indication that the quality of those algorithms is achieving an acceptable level  is that more and more algorithms get implemented in hardware and are released in actual products.  Many stereoscopic TVs now allow you to change the depth effect which means that some kind  of disparity estimation is done in most of them.  On the other hand my...  So why would we still do yet another disparity estimation algorithm?  My impression is that the quality is still lacking in some respects.  One of them is as I said the location of those bad pixels is starting to get very important.  If you do a poor estimation within let's say a blue sky it doesn't matter that much.  If you do a poor estimation on the edges of an object you immediately see the effect if  you watch the generated views out of that.  And then this Middlebury dataset is a set of still images and most people work only  on still images.  I was very happy to see the previous speaker also talk about video.  I think the temporal consistency in video disparity maps is very important because otherwise  you start seeing flickering and a lot of artifacts that are not visible if you just show a number  of frames one by one.  So if you look into those 150 methods that are evaluated now on the Middlebury website  you can...  They are generally classified in global and local methods.  And I will look here mostly into the local methods because they are generally more easily  transformed into something that can be implemented in real time.  Most of those algorithms are subdivided in four main parts.  First there is some calculation of an initial cost which can be seen here below.  This is...  This leads to an initial cost volume where you have a matching cost for every pixel and  every potential disparity value.  And then those data are aggregated or filtered to get somewhat more smoothness and try to  extend certain information to the neighboring pixels.  And then from there the disparity map is calculated.  So until here we work with a cost volume.  And then in the third step we actually reduce this to a two-dimensional disparity map.  And then after that most algorithms still perform some refinement step which could do  for example some correction in areas that are occluded around edges or similar things.  So if we would just from the initial cost go directly to a disparity computation we  would get an image like this one which I think you'll all agree is by far not smooth enough  to get close to this ground truth depth map.  So let us look more closely into a few existing approaches on which we based our research.  So the first one is the adaptive support rate approach by Jun and Kwon.  And they for the aggregation step they use a bilateral filter kind of approach where  they filter costs values from neighboring pixels based on both the color difference  with the central pixel and the distance to the central pixel.  They use a weight for the left image, a weight for the right image and then these are the  initial costs that were calculated.  Based on this work Richard and others have developed in 2010 a dual cross bilateral grid  approach where their main goal was to get to a real time implementation of something  similar to the previous approach.  What they did for that is they changed the cost or the aggregation function somewhat  by using a number of Gaussian functions.  This one is the luminance difference in the left image between your two pixels.  This is the luminance difference in the right image between the two pixels and then there's  the distance to the center pixel and again your original cost.  Now once they got to this formulation it lends itself very well to applying a cross  bilateral grid implementation and that can be easily translated to processing on a graphics  card and therefore can be achieved in real time.  So they get to a four dimensional cross bilateral grid where the four dimensions are for the  horizontal and vertical position and for the luminance difference in left and right image.  Then we get to our approach.  In our approach for the initial cost calculation I'll go down these four steps that I indicated  before.  We used the absolute color difference as well as the gradient difference and made a weight  at some of those two because they seem to contribute somewhat different information.  In the gradient difference you get some more edge information and that's exactly where  you can also do more precise matches.  Then for the cost aggregation we use a similar function as the dual cross bilateral grid  approach but instead of having the luminance difference in the left and the right image  where it was clearly seen both by us and some other researchers that the luminance difference  from the right image doesn't contribute that much anymore.  We wanted to take advantage of this four dimension in the bilateral grid to use some other data  and what we did is we performed a principal component analysis on our color left image  and we used the first two principal components of the left image disregarding the luminance  and color data of the right image for our cost aggregation and used that for cross bilateral  grid.  Then afterwards for the disparity computation we used the winner takes all approach meaning  that for every pixel value you just choose the disparity value with the lowest cost and  you perform the quadratic interpolation to get some sub pixel precision out of it.  Then afterwards in the refinement step we also did some existing approaches first performing  a left right consistency check and filling in those occlusions and afterwards we performed  a weighted medium filter.  As I said I was very concerned also with temporal behavior and instead of filtering our disparity  maps we felt that it was much better or the results were much better when we performed  the recursive filter on our bilateral grid itself.  So what we did is we for to calculate our grid for the current frame we weighted the  bilateral grid for the previous frame and added to that the new data that are available  in the current frame and in that way automatically got some more consistency and while still keeping  the edge alignment and so on which are very important in our very nice features of the  cross bilateral grid.  Now there was one caveat in there because as I said we are using a PCA for the dimension  of our bilateral grid.  If you just blindly update at every frame you are kind of combining different unrelated  data because your PCA basis changes from frame to one frame to another.  So what we did is we keep track of how much change there is in the PCA analysis and only  when there was a larger change than let's say 5% we updated our grid with the new data  from the new basis and otherwise we kept on working with the old basis because the changes  would be minimal anyway.  So that brings me to some results and here we compare on the left the dual cross bilateral  grid approach in the center the proposed approach that we developed and on the right the adaptive  support weight approach which was the first one that I presented and I think both from  the images here at the bottom and from the counts of bad pixels you will be able to see  that our approach while not exactly matching the adaptive support weight I think it gets  pretty close to that and it's definitely way better than the performance of this approach  while our approach still keeps roughly the same computational complexity.  In my opinion the differences that we still have here between those approaches are at  the level where we should start using other measures as well to distinguish between the  different approaches.  Here are some more images where we can see again the difference the different approaches  on each of the images.  I think you'll probably agree that these results look pretty smooth and pretty similar to the  adaptive support weight approach and then as we also emphasize this video behavior we  also performed an experiment on two videos unfortunately I don't have the video data  themselves with me but we did one animation movie or one segment of an animation movie  of about one minute and one piece of some live action video and we asked a number of  viewers to watch it on an autostereoscopic display and to give their feedback on three  different topics one is the overall 3D impression which is given here to the left second is  the temporal coherence and third one are the depth artifacts and we asked them to evaluate  it and the blue color is labeled excellent red is good and it goes down all the way  to the purple here.  So I think my impression from these results is that for the animation movie all videos  performed fairly equal and fairly well.  For the real live action video you see more variation and there you can see that our algorithm  performed somewhat better and gave somewhat more stable results than most of the other  algorithms.  So to conclude I have presented you our new automated disparity estimation algorithm  which is based on the dual cross bilateral grid approach by Richard and others and on  the adaptive support weight method.  One of the main differences is that we changed the left and right luminance image by the  PCA on the left image where we keep those two principal components and I hope I've  been able to convince you that we achieved pretty good results both on still images and  video while still keeping something which while we did not implement it in real time  ourselves should be feasible in real time.  So that concludes my talk.  So question or comment please.  Hi, David Carrigan from Trinity College Dublin.  I just want to ask how does your temporal consistency framework account for local motion  in a video?  So how is it robust to local motion?  It is somewhat inherently dealt with in the bilateral grid approach in the sense that  if there are some let's say if there it is an issue if you have very fast motion but  if it's kind of smooth motion not going too fast then typically by applying some filtering  on your bilateral grid you already spread the disparity values to the neighboring areas  of the image and you can extract those disparity data already.  Maybe one more?  No?  Present to the speaker.  Thank you.  Thank you.  Thank you, dangit.  Thank you.
Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_0_83.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_84_155.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_156_875.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_876_1811.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_1812_3071.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_3072_4283.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_4284_4979.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_4980_6611.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_6612_8915.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_8916_9251.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_9252_9335.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_9336_9755.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_9756_10559.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_10560_12635.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_12636_13331.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_13332_14687.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_14688_14831.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_14832_15251.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_15252_15707.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_15708_16943.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_16944_18131.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_18132_19679.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_19680_20171.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_20172_21455.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_21456_21575.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_21576_22307.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_22308_22499.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_22500_24155.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_24156_24227.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_24228_24275.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_24276_24323.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_24324_25367.png Temporally_consistent_disparity_estimation_using_PCA_dual_cross-bilateral_grid_25368_25436.png