title: How to play raw NAL units in Android MediaCodec
body: <p>I initially tried <a href="https://stackoverflow.com/questions/50217980/how-to-play-raw-nal-units-in-andoid-exoplayer">How to play raw NAL units in Andoid exoplayer?</a> but I noticed I'm gonna have to do things in low level.</p>
  
  <p>I've found this <a href="https://github.com/taehwandev/MediaCodecExample/blob/35c3ca9111a184d9f6aca44f027cb72b286f14a1/src/net/thdev/mediacodecexample/decoder/VideoDecoderThread.java" rel="nofollow noreferrer">simple MediaCodec example</a>. As you can see, it's a thread that plays a file on a surface passed to it.  </p>
  
  <p>Notice the lines</p>
  
  <pre><code>mExtractor = new MediaExtractor();
  mExtractor.setDataSource(filePath);
  </code></pre>
  
  <p>It looks like that I should create my own <code>MediaExtractor</code> which, instead of extracting the video units from a file, it'll use the h264 NAL units from a buffer I'll provide.</p>
  
  <p>I can then call <code>mExtractor.setDataSource(MediaDataSource dataSource)</code>, see <a href="https://developer.android.com/reference/android/media/MediaDataSource" rel="nofollow noreferrer">MediaDataSource</a></p>
  
  <p>It has <code>readAt(long position, byte[] buffer, int offset, int size)</code> </p>
  
  <p>This is where it reads the NAL units. However, how should I pass them? I have no information on the structure of the buffer that needs to be read. </p>
  
  <p><strong>Should I pass a <code>byte[] buffer</code> with the NAL units in it, and if so, in which format? What is the offset for? If it's a buffer, shouldn't I just erase the lines that were read and thus have no offset or size?</strong></p>
  
  <p>By the way, the h264 NAL units are streaming ones, they come from RTP packets, not files. I'm gonna get them through C++ and store them on a buffer an try to pass to the mediaExtractor.</p>
  
  <p><strong>UPDATE:</strong></p>
  
  <p>I've been reading a lot about MediaCodec and I think I understand it better. According to <a href="https://developer.android.com/reference/android/media/MediaCodec" rel="nofollow noreferrer">https://developer.android.com/reference/android/media/MediaCodec</a>, everything relies on something of this type:</p>
  
  <pre><code> MediaCodec codec = MediaCodec.createByCodecName(name);
   MediaFormat mOutputFormat; // member variable
   codec.setCallback(new MediaCodec.Callback() {
     @Override
     void onInputBufferAvailable(MediaCodec mc, int inputBufferId) {
       ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferId);
       // fill inputBuffer with valid data
       …
       codec.queueInputBuffer(inputBufferId, …);
     }
  
     @Override
     void onOutputBufferAvailable(MediaCodec mc, int outputBufferId, …) {
       ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
       MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
       // bufferFormat is equivalent to mOutputFormat
       // outputBuffer is ready to be processed or rendered.
       …
       codec.releaseOutputBuffer(outputBufferId, …);
     }
  
     @Override
     void onOutputFormatChanged(MediaCodec mc, MediaFormat format) {
       // Subsequent data will conform to new format.
       // Can ignore if using getOutputFormat(outputBufferId)
       mOutputFormat = format; // option B
     }
  
     @Override
     void onError(…) {
       …
     }
   });
   codec.configure(format, …);
   mOutputFormat = codec.getOutputFormat(); // option B
   codec.start();
   // wait for processing to complete
   codec.stop();
   codec.release();
  </code></pre>
  
  <p>As you can see, I can pass input buffers and get decoded output buffers. The exact byte formats are still a mystery, but I think that's how it works. Also according to the same article, the usage of <code>ByteBuffer</code>s is slow, and <code>Surface</code>s are preferred. They consume the output buffers automatically. Although there's no tutorial on how to do it, there's a section in the article that says it's almost identical, so I guess I just need to add the additional lines </p>
  
  <pre><code> codec.setInputSurface(Surface inputSurface) 
   codec.setOutputSurface(Surface outputSurface) 
  </code></pre>
  
  <p>Where <code>inputSurface</code> and <code>outputSurface</code> are <a href="https://developer.android.com/reference/android/view/Surface" rel="nofollow noreferrer">Surface</a>s which I pass to a <a href="https://developer.android.com/reference/android/media/MediaPlayer#setSurface(android.view.Surface)" rel="nofollow noreferrer">MediaPlayer</a> which I use (how) to display the video in an activity. And the output buffers will simply not come on <code>onOutputBufferAvailable</code> (because the <code>surface</code> consumes them first), neither <code>onInputBufferAvailable</code>.</p>
  
  <p>So the questions now are: <strong>how exactly do I construct a <code>Surface</code> which contains the video buffer, and how do I display a <code>MediaPlayer</code> into an activity</strong></p>
  
  <p>For output I can simply create a <code>Surface</code> and pass to a <code>MediaPlayer</code> and <code>MediaCodec</code>, but what about input? Do I need <code>ByteBuffer</code> for the input anyways, and <code>Surface</code> would just be for using other outputs as inputs?</p>