title: Matching in-world texture to overlay image using mesh UVs
body: <p>The last few days I have been working on programming the effect I saw in a game called <a href="https://ianmaclarty.itch.io/catacombs-of-solaris" rel="nofollow noreferrer">the Catacombs of Solaris</a>. To test the effect I created this demo:</p>
  
  <ol>
  <li>This image is overlayed on the screen: (Scaled to match resolution) <a href="https://i.stack.imgur.com/FzN1F.png" rel="nofollow noreferrer">Rainbow image overlay</a>.  </li>
  <li>At the press of the space bar, the on-screen coordinates of the in-game mesh's vertices are converted into UVs and applied to the mesh with the same texture.</li>
  </ol>
  
  <p>When the camera facing at the quad at a perpendicular angle the effect looks like this: <a href="https://i.stack.imgur.com/yL6XN.png" rel="nofollow noreferrer">Perpendicular camera example</a>.</p>
  
  <p>To explain in a different way, if the top left coordinate of the quad is 30% through the screen horizontally, and 40% through the screen vertically, I hope that making the UV for the vertex (0.3, 0.4) that the texture would match the overlay perfectly (as seen in the picture).</p>
  
  <p>Now, the effect only works if the camera can be looking from other angles as well, so here is a picture of the camera at a slight rotation (about the Y axis), after updating the texture: <a href="https://i.stack.imgur.com/wX82v.png" rel="nofollow noreferrer">Not-perpendicular camera example</a>.</p>
  
  <p>As you can see the texture almost lines up, but is a bit skewed, especially around the triangle borders of the mesh, shown here in white: <a href="https://i.stack.imgur.com/BCquK.png" rel="nofollow noreferrer">Triangles of the mesh</a>.</p>
  
  <p>Here is the code I am using to update the texture</p>
  
  <pre><code>void UpdateTexture() {
      // test is the rainbow Texture2D
      // r is the Renderer
      r.material.mainTexture = test;
  
      // m is the Quad mesh
      Vector3[] vert = m.vertices;
      Vector2[] uv = m.uv;
  
      // We will go through each vertex of the mesh
      for(int i = 0; i &lt; uv.Length; i++) {
          // Find where the world position of that vertex is
          Vector3 worldPosition = transform.TransformPoint(vert[i]);
          // Convert that into a screen position based on our camera
          Vector2 screenPosition = CameraToTexture.cam.WorldToScreenPoint(worldPosition);
          // Put that within the range 0...1 as a UV
          Vector2 uvPosition = new Vector2(screenPosition.x / CameraToTexture.cam.pixelWidth, screenPosition.y / CameraToTexture.cam.pixelHeight);
          // Save that UV into our temp array
          uv[i] = uvPosition;
      }
  
      // Apply to the UV array of the mesh
      m.uv = uv;
  } 
  </code></pre>
  
  <p>My questions:</p>
  
  <ol>
  <li>Why is the texture warped around the triangle boundaries?</li>
  <li>Why does the effect work differently when the camera is not at a right angle?</li>
  <li>How can I fix this? / What can I look into to better understand what is happening?</li>
  </ol>
  
  <p>If any more information is needed I will likely be able to provide it.</p>