title: get mouse in world coordinates with 2 gluUnproject calls to create ray
body: <p>I try to use what many people seem to find a good way, I call gluUnproject 2 times with different z-values and then try to calculate the direction vector for the ray from these 2 vectors.  </p>
  
  <p>I read <a href="https://stackoverflow.com/q/21345493/1972372">this question</a> and tried to use the structure there for my own code:  </p>
  
  <pre><code>        glGetFloat(GL_MODELVIEW_MATRIX, modelBuffer);
          glGetFloat(GL_PROJECTION_MATRIX, projBuffer);
          glGetInteger(GL_VIEWPORT, viewBuffer);
  
          gluUnProject(mouseX, mouseY, 0.0f, modelBuffer, projBuffer, viewBuffer, startBuffer);
          gluUnProject(mouseX, mouseY, 1.0f, modelBuffer, projBuffer, viewBuffer, endBuffer);
  
          start = vecmath.vector(startBuffer.get(0), startBuffer.get(1), startBuffer.get(2));
          end = vecmath.vector(endBuffer.get(0), endBuffer.get(1), endBuffer.get(2));
  
          direction = vecmath.vector(end.x()-start.x(), end.y()-start.y(), end.z()-start.z());
  </code></pre>
  
  <p>But this only returns the Homogeneous Clip Coordinates (I believe), since they only range from -1 to 1 on every axis.  </p>
  
  <p>How to actually get coordinates from which I can create a ray?  </p>
  
  <p><strong>EDIT:</strong> This is how I construct the matrices:  </p>
  
  <pre><code>Matrix projectionMatrix = vecmath.perspectiveMatrix(60f, aspect, 0.1f,
              100f);
  //The matrix of the camera = viewMatrix
  setTransformation(vecmath.lookatMatrix(eye, center, up));
  //And every object sets a ModelMatrix in it's display method  
  Matrix modelMatrix = parentMatrix.mult(vecmath
                      .translationMatrix(translation));
  modelMatrix = modelMatrix.mult(vecmath.rotationMatrix(1, 0, 1, angle));
  </code></pre>
  
  <p><strong>EDIT 2:</strong>  </p>
  
  <p>This is how the function looks right now:  </p>
  
  <pre><code>private void calcMouseInWorldPosition(float mouseX, float mouseY, Matrix proj, Matrix view) {
      Vector start = vecmath.vector(0, 0, 0);
      Vector end = vecmath.vector(0, 0, 0);
  
      FloatBuffer modelBuffer = BufferUtils.createFloatBuffer(16);
      modelBuffer.put(view.asArray());
      modelBuffer.rewind();
      FloatBuffer projBuffer = BufferUtils.createFloatBuffer(16);
      projBuffer.put(proj.asArray());
      projBuffer.rewind();
  
      FloatBuffer startBuffer = BufferUtils.createFloatBuffer(16);
      FloatBuffer endBuffer = BufferUtils.createFloatBuffer(16);
      IntBuffer viewBuffer = BufferUtils.createIntBuffer(16);
  
       //The two calls for projection and modelView matrix are disabled here,   
         as I use my own matrices in this case
  //  glGetFloat(GL_MODELVIEW_MATRIX, modelBuffer);
  //  glGetFloat(GL_PROJECTION_MATRIX, projBuffer);
      glGetInteger(GL_VIEWPORT, viewBuffer);
  
      //I know this is really ugly and bad, but I know that the height and width is always 600  
      // and this is just for testing purposes 
      mouseY = 600 - mouseY;
  
      gluUnProject(mouseX, mouseY, 0.0f, modelBuffer, projBuffer, viewBuffer, startBuffer);
      gluUnProject(mouseX, mouseY, 1.0f, modelBuffer, projBuffer, viewBuffer, endBuffer);
  
      start = vecmath.vector(startBuffer.get(0), startBuffer.get(1), startBuffer.get(2));
      end = vecmath.vector(endBuffer.get(0), endBuffer.get(1), endBuffer.get(2));
  
      direction = vecmath.vector(end.x()-start.x(), end.y()-start.y(), end.z()-start.z());
  </code></pre>
  
  <p>}</p>
  
  <p>I'm trying to use my own projection and view matrix, but this only seems to give weirder results.<br>
  With the GlGet... stuff I get this for a click in the upper right corner:<br>
  <em>start: (0.97333336, -0.98, -1.0)<br>
  end: (0.97333336, -0.98, 1.0)</em></p>
  
  <p>When I use my own stuff I get this for the same position:<br>
  <em>start: (-2.4399707, -0.55425626, -14.202201)<br>
  end: (-2.4399707, -0.55425626, -16.198204)</em></p>
  
  <p>Now I actually need a modelView matrix instead of just the view matrix, but I don't know how I am supposed to get it, since it is altered and created anew in every display call of every object.<br>
  But is this really the problem? In <a href="http://antongerdelan.net/opengl/raycasting.html" rel="nofollow noreferrer">this</a> tutorial he says <em>"Normally, to get into clip space from eye space we multiply the vector by a projection matrix. We can go backwards by multiplying by the inverse of this matrix."</em> and in the next step he multiplies again by the inverse of the view matrix, so I thought this is what I should actually do?</p>
  
  <p><strong>EDIT 3:</strong><br>
  Here I tried what user42813 suggested:  </p>
  
  <pre><code>    Matrix view = cam.getTransformation();
      view = view.invertRigid();
  
      mouseY = height - mouseY - 1;
  
      //Here I only these values, because the Z and W values would be 0  
      //following your suggestion, so no use adding them here
      float tempX = view.get(0, 0) * mouseX + view.get(1, 0) * mouseY;
      float tempY = view.get(0, 1) * mouseX + view.get(1, 1) * mouseY;
      float tempZ = view.get(0, 2) * mouseX + view.get(1, 2) * mouseY;
  
      origin = vecmath.vector(tempX, tempY, tempZ);
      direction = cam.getDirection();
  </code></pre>
  
  <p>But now the direction and origin values are always the same:<br>
  origin: (-0.04557252, -0.0020000197, -0.9989586)<br>
  direction: (-0.04557252, -0.0020000197, -0.9989586)</p>