title: Mapping depth pixels to colour pixels using pyrealsense2
body: <p>I have an Intel Realsense Depth Camera D415. I am trying to use this method of mapping the pixels from the <strong>depth channel</strong> to <strong>colour channel</strong>. I am using the Python API and apparently the <code>pyrealsense2.align()</code> method is very CPU intensive and it dramatically slows down my application. So, I was thinking maybe mapping <strong>depth pixels</strong> to <strong>colour pixels</strong> would allow me make things run faster. However, I ran into the following problem: when I try to map depth pixel <strong>[200, 200]</strong> to its corresponding colour pixel I get returned <strong>[15294.3466796875, -113.68994140625]</strong>. I doubt this is correct since my colour channel resolution is <strong>1280x720</strong> and my colour coordinates are clearly out of <strong>[0, 1280]</strong>, <strong>[0, 720]</strong>. The code (<a href="https://github.com/IntelRealSense/librealsense/pull/1118" rel="nofollow noreferrer">link</a>) I am using is the following:</p>
  
  <pre><code>import pyrealsense2 as rs
  pipeline = rs.pipeline()
  pipe_profile = pipeline.start()
  frames = pipeline.wait_for_frames()
  depth_frame = frames.get_depth_frame()
  color_frame = frames.get_color_frame()
  
  # Intrinsics &amp; Extrinsics
  depth_intrin = depth_frame.profile.as_video_stream_profile().intrinsics
  color_intrin = color_frame.profile.as_video_stream_profile().intrinsics
  depth_to_color_extrin = depth_frame.profile.get_extrinsics_to(color_frame.profile)
  color_to_depth_extrin = color_frame.profile.get_extrinsics_to(depth_frame.profile)
  
  print "\n Depth intrinsics: " + str(depth_intrin)
  print "\n Color intrinsics: " + str(color_intrin)
  print "\n Depth to color extrinsics: " + str(depth_to_color_extrin)
  
  # Depth scale - units of the values inside a depth frame, i.e how to convert the value to units of 1 meter
  depth_sensor = pipe_profile.get_device().first_depth_sensor()
  depth_scale = depth_sensor.get_depth_scale()
  print depth_scale
  # Map depth to color
  depth_pixel = [200, 200]   # Random pixel
  print "\n\t depth_pixel: " + str(depth_pixel)
  depth_point = rs.rs2_deproject_pixel_to_point(depth_intrin, depth_pixel, depth_scale)
  color_point = rs.rs2_transform_point_to_point(depth_to_color_extrin, depth_point)
  color_pixel = rs.rs2_project_point_to_pixel(color_intrin, color_point)
  print "\n\t color_pixel: " + str(color_pixel)
  pipeline.stop()
  </code></pre>
  
  <p>In addition, this is what gets printed:</p>
  
  <pre><code> Depth intrinsics: width: 1280, height: 720, ppx: 644.664, ppy: 406.596, fx: 965.563, fy: 965.563, model: Brown Conrady, coeffs: [0, 0, 0, 0, 0]
  
   Color intrinsics: width: 640, height: 480, ppx: 330.929, ppy: 233.255, fx: 610.102, fy: 609.615, model: Brown Conrady, coeffs: [0, 0, 0, 0, 0]
  
   Depth to color extrinsics: rotation: [0.99999, 0.00428028, 0.0010846, -0.00427981, 0.999991, -0.000430169, -0.00108643, 0.000425523, 0.999999]
  translation: [0.0151529, -0.00012542, -0.000400548]
  0.0010000000475
  
       depth_pixel: [200, 200]
  
       color_pixel: [15294.3466796875, -113.68994140625]
  </code></pre>
  
  <p><strong>1.Is anyone aware of a workaround in this case?</strong></p>
  
  <p><strong>2.Are there any other transformations I need to carry out?</strong> 
  Any help/guidance is very much appreciated.</p>