title: How to transform back-facing vertices in GLSL when creating a shadow volume
body: <p>I'm writing a game using OpenGL and I am trying to implement shadow volumes.</p>
  
  <p>I want to construct the shadow volume of a model on the GPU via a vertex shader.  To that end, I represent the model with a VBO where:</p>
  
  <ul>
  <li>Vertices are duplicated such that each triangle has its own unique three vertices</li>
  <li>Each vertex has the normal of its triangle
  
  <ul>
  <li>For reasons I'm not going to get into, I was actually doing the above two points anyway, so I'm not too worried about the vertex duplication</li>
  </ul></li>
  <li>Degenerate triangles are added to form quads inside the edges between each pair of "regular" triangles</li>
  </ul>
  
  <p>Using this model format, inside the vertex shader I am able to find vertices that are part of triangles that face away from the light and move them back to form the shadow volume.</p>
  
  <p>What I have left to figure out is what transformation exactly I should apply to the back-facing vertices.</p>
  
  <p>I am able to detect when a vertex is facing away from the light, but I am unsure what transformation I should apply to it.  This is what my vertex shader looks like so far:</p>
  
  <pre><code>uniform vec3 lightDir; // Parallel light.
                         // On the CPU this is represented in world
                         // space.  After setting the camera with
                         // gluLookAt, the light vector is multiplied by
                         // the inverse of the modelview matrix to get
                         // it into eye space (I think that's what I'm
                         // working in :P ) before getting passed to
                         // this shader.
  
  void main()
  {
      vec3 eyeNormal = normalize(gl_NormalMatrix * gl_Normal);
      vec3 realLightDir = normalize(lightDir);
  
      float dotprod = dot(eyeNormal, realLightDir);
  
      if (dotprod &lt;= 0.0)
      {
          // Facing away from the light
          // Need to translate the vertex along the light vector to
          // stretch the model into a shadow volume
  
          //---------------------------------//
          // This is where I'm getting stuck //
          //---------------------------------//
          // All I know is that I'll probably turn realLightDir into a
          // vec4
  
          gl_Position = ???;
      }
      else
      {
          gl_Position = ftransform();
      }
  }
  </code></pre>
  
  <p>I've tried simply setting <code>gl_position</code> to <code>ftransform() - (vec4(realLightDir, 1.0) * someConstant)</code>, but this caused some kind of depth-testing bugs (some faces seemed to be visible behind others when I rendered the volume with colour) and <code>someConstant</code> didn't seem to affect how far the back-faces are extended.</p>
  
  <p><strong>Update - Jan. 22</strong></p>
  
  <p>Just wanted to clear up questions about what space I'm probably in.  I must say that keeping track of what space I'm in is the greatest source of my shader headaches.</p>
  
  <p>When rendering the scene, I first set up the camera using <code>gluLookAt</code>.  The camera may be fixed or it may move around; it should not matter.  I then use translation functions like <code>glTranslated</code> to position my model(s).</p>
  
  <p>In the program (i.e. on the CPU) I represent the light vector in world space (three floats).  I've found during development that to get this light vector in the right space of my shader I had to multiply it by the inverse of the modelview matrix after setting the camera and before positioning the models.  So, my program code is like this:</p>
  
  <ul>
  <li>Position camera (<code>gluLookAt</code>)</li>
  <li>Take light vector, which is in world space, and multiply it by the inverse of the current modelview matrix and pass it to the shader</li>
  <li>Transformations to position models</li>
  <li>Drawing of models</li>
  </ul>
  
  <p>Does this make anything clearer?</p>