title: OpenCV - Project image plane point to 3d
body: <p>I've used OpenCV to calibrate my camera from different views and I obtained intrinsics, rvec and tvec with a reprojection error of .03 px (I thus think the calibration is fine).</p>
  
  <p>Now, given one view of my scene, I want to be able to click on a point and find its projection on the other views.
  To do so, I se the following functions:</p>
  
  <pre><code>void Camera::project(const vector&lt;cv::Point2f&gt; &amp;pts_2d, vector&lt;cv::Point3f&gt; &amp;pts_3d) {
  
       std::cout &lt;&lt; "Start proj 2d -&gt; 3d" &lt;&lt; std::endl;
  
       cv::Mat pts_2d_homo;
       convertPointsToHomogeneous(pts_2d, pts_2d_homo);
  
       std::cout &lt;&lt; "Cartesian to Homogeneous done!" &lt;&lt; std::endl;
  
       // Project point to camera normalized coordinates    
       cv::Mat unproj;
  
       cv::transform(pts_2d_homo, unproj, intrinsics().inv());
  
       std::cout &lt;&lt; "Point unprojected: " &lt;&lt; unproj.at&lt;cv::Point3f&gt;(0) &lt;&lt; std::endl;
  
       // Undo model view transform
       unproj -= transVec();
  
       cv::Mat rot;
       cv::Rodrigues(rotVec(), rot);
       cv::transform(unproj, unproj, rot.t());
  
       unproj *= 1.f/cv::norm(unproj);
  
       std::cout &lt;&lt; "Model view undone: " &lt;&lt; unproj.at&lt;cv::Point3f&gt;(0) &lt;&lt; std::endl;
  
      for (int i = 0; i &lt; unproj.rows; ++i) {
          std::cout &lt;&lt; "Inside for :" &lt;&lt; unproj.at&lt;cv::Point3f&gt;(i,0) &lt;&lt; std::endl;
          pts_3d.push_back(unproj.at&lt;cv::Point3f&gt;(i,0));
     }
  }
  
  void Camera::project(const vector&lt;cv::Point3f&gt; &amp;pts_3d, vector&lt;cv::Point2f&gt; &amp;pts_2d) {
      cv::projectPoints(pts_3d, rotVec(), transVec(), intrinsics(), dist_coeffs(), pts_2d);
  </code></pre>
  
  <p>} </p>
  
  <p>Now I have mixed feelings about what I get as an output. When I draw the point projected on each view, they all correspond BUT no matter where I clicked at first in the "canonical view", the projected point is always the same.</p>