title: How to generate a valid point cloud representation of a pair of stereo images using OpenCV 3.0 StereoSGBM and PCL
body: <p>I have recently started working with OpenCV 3.0 and my goal is to capture a pair of stereo images from a set of stereo cameras, create a proper disparity map, convert the disparity map to a 3D point cloud and finally show the resulting point cloud in a point-cloud viewer using PCL.</p>
  
  <p>I have already performed the camera calibration and the resulting calibration RMS is 0.4</p>
  
  <p>You can find my image pairs (Left Image)<a href="http://i.stack.imgur.com/v6TUR.jpg" rel="noreferrer">1</a>  and (Right Image)<a href="http://i.stack.imgur.com/hfGCs.jpg" rel="noreferrer">2</a> in the links below. I am using StereoSGBM in order to create disparity image. I am also using  track-bars to adjust StereoSGBM function parameters in order to obtain better disparity image. Unfortunately I can't post my disparity image since I am new to StackOverflow and don't have enough reputation to post more than two image links!</p>
  
  <p>After getting the disparity image ("disp" in the code below), I use the reprojectImageTo3D() function to convert the disparity image information to XYZ 3D coordinate, and then I convert the results into an array of "pcl::PointXYZRGB" points so they can be shown in a PCL point cloud viewer. After performing the required conversion, what I get as a point cloud is a silly pyramid shape point-cloud which does not make any sense. I have already read and tried all of the suggested methods in the following links:</p>
  
  <p>1- http: //blog.martinperis.com/2012/01/3d-reconstruction-with-opencv-and-point.html</p>
  
  <p>2- http: //stackoverflow.com/questions/13463476/opencv-stereorectifyuncalibrated-to-3d-point-cloud</p>
  
  <p>3- http: //stackoverflow.com/questions/22418846/reprojectimageto3d-in-opencv</p>
  
  <p>and non of them worked!!!</p>
  
  <p>Below I provided the conversion portion of my code, it would be greatly appreciated if you could tell me what I am missing:</p>
  
  <pre><code>pcl::PointCloud&lt;pcl::PointXYZRGB&gt;::Ptr pointcloud(new   pcl::PointCloud&lt;pcl::PointXYZRGB&gt;());
      Mat xyz;
      reprojectImageTo3D(disp, xyz, Q, false, CV_32F);
      pointcloud-&gt;width = static_cast&lt;uint32_t&gt;(disp.cols);
      pointcloud-&gt;height = static_cast&lt;uint32_t&gt;(disp.rows);
      pointcloud-&gt;is_dense = false;
      pcl::PointXYZRGB point;
      for (int i = 0; i &lt; disp.rows; ++i)
          {
              uchar* rgb_ptr = Frame_RGBRight.ptr&lt;uchar&gt;(i);
              uchar* disp_ptr = disp.ptr&lt;uchar&gt;(i);
              double* xyz_ptr = xyz.ptr&lt;double&gt;(i);
  
              for (int j = 0; j &lt; disp.cols; ++j)
              {
                  uchar d = disp_ptr[j];
                  if (d == 0) continue;
                  Point3f p = xyz.at&lt;Point3f&gt;(i, j);
  
                  point.z = p.z;   // I have also tried p.z/16
                  point.x = p.x;
                  point.y = p.y;
  
                  point.b = rgb_ptr[3 * j];
                  point.g = rgb_ptr[3 * j + 1];
                  point.r = rgb_ptr[3 * j + 2];
                  pointcloud-&gt;points.push_back(point);
              }
          }
      viewer.showCloud(pointcloud);
  </code></pre>