title: 2D edge to 3D modelspace via glm::unproject - ray not intersecting with model
body: <p>I try to do ray picking with collision, which means a 2D position on the screen is transformed into a 3D position in model space.
  My problem is, that the ray doesn't always intersect with the model to result in a valid 3D position.</p>
  
  <p>Here's what I do:</p>
  
  <p>I create a depth texture in a framebuffer to <code>glReadPixels()</code> from it later.</p>
  
  <pre><code>glTexImage2D(GL_TEXTURE_2D, 0, GL_DEPTH_COMPONENT, screenWidth, screenHeight, 0, GL_DEPTH_COMPONENT, GL_FLOAT, NULL);
  
  cv::Mat_&lt;float&gt; depthCV(screenHeight, screenWidth);
  glReadPixels(0, 0, screenWidth, screenHeight, GL_DEPTH_COMPONENT, GL_FLOAT, depthCV.ptr());
  </code></pre>
  
  <p>I linearize the depth afterwards and store it into a <code>cv::Mat_&lt;uchar&gt;</code> for edge detection via <code>cv::Canny()</code>: (Don't forget to flip the image.)</p>
  
  <pre><code>cv::blur(gray, edge, cv::Size(3, 3));
  cv::Canny(edge, edge, edgeThresh, edgeThresh * ratio, 3);
  </code></pre>
  
  <p>... and get the contour via </p>
  
  <pre><code>cv::findContours(canny_output, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_NONE, cv::Point(0, 0));
  </code></pre>
  
  <p>The 2D coordinates (x,y) of the contour(s) are now being transformed in 3D model coordinates via:</p>
  
  <pre><code>GLint viewport[4];
  GLfloat winX, winY, winZ;
  glGetIntegerv(GL_VIEWPORT, viewport);
  winX = (float)x;
  winY = (float)viewport[3] - (float)(y);
  glReadPixels(x, int(winY), 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &amp;winZ);
  glm::vec4 viewportvec(viewport[0], viewport[1], viewport[2], viewport[3]);
  glm::vec3 winCoords(winX, winY, winZ);
  
  glm::vec3 modelCoords = glm::unProject(winCoords, view*model, proj, viewportvec);
  </code></pre>
  
  <p>After doing this and looking at the 3D points it shows, that some rays don't intersect with the model. The length of the ray is close to the value of the far-plane.
  I tried to broaden the input by looking also at the one pixel neighborhood around the input (x,y). Still some rays don't intersect with the model.</p>
  
  <p>Any idea why this is? Or how to solve it? Is the edge extraction with previous blurring responsible for it?</p>
  
  <p><a href="https://i.stack.imgur.com/Jr5dm.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Jr5dm.png" alt="Teapot"></a></p>
  
  <p>I attached an image where the distribution can be seen. The blue points are the ones not intersecting with the model (length of the ray close to the value of the far plane), the red ones are. I used the teapot model for my testing.</p>
  
  <p>I didn't want to paste my whole code here for ease of reading. If more code or information is needed to understand better what might be wrong, please let me know.</p>