title: MediaCodec and Camera: colorspaces don't match
body: <p>I have been trying to get H264 encoding to work with input captured by the camera on an Android tablet using the new low-level <a href="http://developer.android.com/reference/android/media/MediaCodec.html" rel="noreferrer">MediaCodec</a>. I have gone through some difficulties with this, since the MediaCodecAPI is poorly documented, but I've gotten something to work at last.</p>
  
  <p>I'm setting up the camera as follows:</p>
  
  <pre><code>        Camera.Parameters parameters = mCamera.getParameters();
          parameters.setPreviewFormat(ImageFormat.YV12); // &lt;1&gt;
          parameters.setPreviewFpsRange(4000,60000);
          parameters.setPreviewSize(640, 480);            
          mCamera.setParameters(parameters);
  </code></pre>
  
  <p>For the encoding part, I'm instantiating the MediaCodec object as follows:</p>
  
  <pre><code>    mediaCodec = MediaCodec.createEncoderByType("video/avc");
      MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/avc", 640, 480);
      mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, 500000);
      mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 15);
      mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT,
              MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar); // &lt;2&gt;
      mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5);
      mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
      mediaCodec.start();
  </code></pre>
  
  <p>The final goal is to create an RTP-stream (and correspond with Skype), but so far I am only streaming the raw H264 directly to my desktop. There I use the following GStreamer-pipeline to show the result:</p>
  
  <pre><code>gst-launch udpsrc port=5555 ! video/x-h264,width=640,height=480,framerate=15/1 ! ffdec_h264 ! autovideosink
  </code></pre>
  
  <p>All works well, except for the colors. I need to set 2 colorformats in the computer: one for the camera-preview (line tagged with <code>&lt;1&gt;</code>) and one for the MediaCodec-object (tagged with <code>&lt;2&gt;</code>)</p>
  
  <p>To determine the acceptable values for the lines <code>&lt;1&gt;</code> I used <code>parameters.getSupportedPreviewFormats()</code>. From this, I know that the only supported formats on the camera are <a href="http://developer.android.com/reference/android/graphics/ImageFormat.html#NV21" rel="noreferrer">ImageFormat.NV21</a> and <a href="http://developer.android.com/reference/android/graphics/ImageFormat.html#YV12" rel="noreferrer">ImageFormat.YV2</a>.</p>
  
  <p>For <code>&lt;2&gt;</code>, I retrieved the <a href="http://developer.android.com/reference/android/media/MediaCodecInfo.CodecCapabilities.html" rel="noreferrer">MediaCodecInfo.CodecCapabilities</a>-object for type <em>video/avc</em>, being the integer values 19 (corresponding with <a href="http://developer.android.com/reference/android/media/MediaCodecInfo.CodecCapabilities.html#COLOR_FormatYUV420Planar" rel="noreferrer">MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar</a> and 2130708361 (which doesn't correspond with any value of <a href="http://developer.android.com/reference/android/media/MediaCodecInfo.CodecCapabilities.html" rel="noreferrer">MediaCodecInfo.CodecCapabilities</a>). </p>
  
  <p>Any other value than the above results in a crash.</p>
  
  <p>Combining these settings gives different results, which I'll show below. Here's the screenshot on Android (i.e. the "real" colors):
  <img src="https://i.stack.imgur.com/fjFXp.png" alt="Input on Android-tablet">
  Here are the results as shown by Gstreamer:</p>
  
  <p><code>&lt;1&gt;</code> = NV21, <code>&lt;2&gt;</code> = COLOR_FormatYUV420Planar
  <img src="https://i.stack.imgur.com/kyrMZ.png" alt="Gstreamer-output for NV21-COLOR_FormatYUV420Planar"></p>
  
  <p><code>&lt;1&gt;</code> = NV21, <code>&lt;2&gt;</code> = 2130708361
  <img src="https://i.stack.imgur.com/C2MZN.png" alt="Gstreamer-output for NV21-2130708361"></p>
  
  <p><code>&lt;1&gt;</code> = YV2, <code>&lt;2&gt;</code> = COLOR_FormatYUV420Planar
  <img src="https://i.stack.imgur.com/FdJZN.png" alt="Gstreamer-output for YV2-COLOR_FormatYUV420Planar"></p>
  
  <p><code>&lt;1&gt;</code> = YV2, <code>&lt;2&gt;</code> = 2130708361
  <img src="https://i.stack.imgur.com/rl8xb.png" alt="Gstreamer-output for YV2-2130708361"></p>
  
  <p>As can be seen, none of these are satisfying. The YV2-colorspace looks the most promising, but it looks like red (Cr) and blue (Cb) are inverted. The NV21 looks interlaced I guess (however, I'm no expert in this field).</p>
  
  <p>Since the purpose is to communicate with Skype, I assume I shouldn't change the decoder (i.e. the Gstreamer-command), right? Is this to be solved in Android and if so: how? Or can this be solved by adding certain RTP payload information? Any other suggestion?</p>