title: Converting a 2D image point to a 3D world point
body: <p>I know that in the general case, making this conversion is impossible since depth information is lost going from 3d to 2d.</p>
  
  <p>However, I have a fixed camera and I know its camera matrix. I also have a planar calibration pattern of <strong>known dimensions</strong> - let's say that in world coordinates it has corners (0,0,0) (2,0,0) (2,1,0) (0,1,0). Using opencv I can estimate the pattern's pose, giving the translation and rotation matrices needed to project a point on the object to a pixel in the image.</p>
  
  <p>Now: this 3d to image projection is easy, but how about the other way? If I pick a pixel in the image that I <strong>know is part of the calibration pattern</strong>, how can I get the corresponding 3d point?</p>
  
  <p>I could iteratively choose some random 3d point on the calibration pattern, project to 2d, and refine the 3d point based on the error. But this seems pretty horrible.</p>
  
  <p>Given that this unknown point has world coordinates something like (x,y,0) -- since it must lie on the z=0 plane -- it seems like there should be some transformation that I can apply, instead of doing the iterative nonsense. My maths isn't very good though - can someone work out this transformation and explain how you derive it?</p>