title: PCM -> AAC (Encoder) -> PCM(Decoder) in real-time with correct optimization
body: <p>I'm trying to implement  </p>
  
  <pre><code>AudioRecord (MIC) -&gt;
  
  PCM -&gt; AAC Encoder
  AAC -&gt; PCM Decode
  
  -&gt; AudioTrack??  (SPEAKER)
  </code></pre>
  
  <p>with <code>MediaCodec</code> on Android 4.1+ (API16).</p>
  
  <p>Firstly, I successfully (but not sure correctly optimized) implemented <code>PCM -&gt; AAC Encoder</code> by  <code>MediaCodec</code> as intended as below</p>
  
  <pre><code>private boolean setEncoder(int rate)
  {
      encoder = MediaCodec.createEncoderByType("audio/mp4a-latm");
      MediaFormat format = new MediaFormat();
      format.setString(MediaFormat.KEY_MIME, "audio/mp4a-latm");
      format.setInteger(MediaFormat.KEY_CHANNEL_COUNT, 1);
      format.setInteger(MediaFormat.KEY_SAMPLE_RATE, 44100);
      format.setInteger(MediaFormat.KEY_BIT_RATE, 64 * 1024);//AAC-HE 64kbps
      format.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectHE);
      encoder.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
      return true;
  }
  </code></pre>
  
  <p><strong>INPUT: PCM Bitrate = 44100(Hz) x 16(bit) x 1(Monoral) =  705600 bit/s</strong></p>
  
  <p><strong>OUTPUT: AAC-HE Bitrate = 64 x 1024(bit) = 65536 bit/s</strong></p>
  
  <p><strong>So, the data size is approximately compressed <code>x11</code></strong> ,and I confirmed this working by observing a log</p>
  
  <ul>
  <li>AudioRecoder﹕ 4096 bytes read</li>
  <li>AudioEncoder﹕ 369 bytes encoded</li>
  </ul>
  
  <p>the data size is approximately compressed <code>x11</code>, so far so good.</p>
  
  <p>Now, I have a UDP server to receive the encoded data, then decode it.</p>
  
  <p>The decoder profile is set as follows:</p>
  
  <pre><code>private boolean setDecoder(int rate)
  {
      decoder = MediaCodec.createDecoderByType("audio/mp4a-latm");
      MediaFormat format = new MediaFormat();
      format.setString(MediaFormat.KEY_MIME, "audio/mp4a-latm");
      format.setInteger(MediaFormat.KEY_CHANNEL_COUNT, 1);
      format.setInteger(MediaFormat.KEY_SAMPLE_RATE, 44100);
      format.setInteger(MediaFormat.KEY_BIT_RATE, 64 * 1024);//AAC-HE 64kbps
      format.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectHE);
      decoder.configure(format, null, null, 0);
  
      return true;
  }
  </code></pre>
  
  <p>Since UDPserver packet buffer size is <code>1024</code></p>
  
  <ul>
  <li>UDPserver   ﹕ 1024 bytes received</li>
  </ul>
  
  <p>and since this is the compressed AAC data, I would expect the decoding size will be </p>
  
  <p>approximately 1024 <code>x11</code>, however the actual result is</p>
  
  <ul>
  <li>AudioDecoder﹕ 8192 bytes decoded</li>
  </ul>
  
  <p>It's approximately <code>x8</code>, and I feel something wrong.</p>
  
  <p>The decoder code is as follows:</p>
  
  <pre><code>    IOudpPlayer = new Thread(new Runnable()
      {
          public void run()
          {
              SocketAddress sockAddress;
              String address;
  
              int len = 1024;
              byte[] buffer2 = new byte[len];
              DatagramPacket packet;
  
              byte[] data;
  
              ByteBuffer[] inputBuffers;
              ByteBuffer[] outputBuffers;
  
              ByteBuffer inputBuffer;
              ByteBuffer outputBuffer;
  
              MediaCodec.BufferInfo bufferInfo;
              int inputBufferIndex;
              int outputBufferIndex;
              byte[] outData;
              try
              {
                  decoder.start();
                  isPlaying = true;
                  while (isPlaying)
                  {
                      try
                      {
                          packet = new DatagramPacket(buffer2, len);
                          ds.receive(packet);
  
                          sockAddress = packet.getSocketAddress();
                          address = sockAddress.toString();
  
                          Log.d("UDP Receiver"," received !!! from " + address);
  
                          data = new byte[packet.getLength()];
                          System.arraycopy(packet.getData(), packet.getOffset(), data, 0, packet.getLength());
  
                          Log.d("UDP Receiver",  data.length + " bytes received");
  
                          //===========
                          inputBuffers = decoder.getInputBuffers();
                          outputBuffers = decoder.getOutputBuffers();
                          inputBufferIndex = decoder.dequeueInputBuffer(-1);
                          if (inputBufferIndex &gt;= 0)
                          {
                              inputBuffer = inputBuffers[inputBufferIndex];
                              inputBuffer.clear();
  
                              inputBuffer.put(data);
  
                              decoder.queueInputBuffer(inputBufferIndex, 0, data.length, 0, 0);
                          }
  
                          bufferInfo = new MediaCodec.BufferInfo();
                          outputBufferIndex = decoder.dequeueOutputBuffer(bufferInfo, 0);
  
                          while (outputBufferIndex &gt;= 0)
                          {
                              outputBuffer = outputBuffers[outputBufferIndex];
  
                              outputBuffer.position(bufferInfo.offset);
                              outputBuffer.limit(bufferInfo.offset + bufferInfo.size);
  
                              outData = new byte[bufferInfo.size];
                              outputBuffer.get(outData);
  
                              Log.d("AudioDecoder", outData.length + " bytes decoded");
  
                              decoder.releaseOutputBuffer(outputBufferIndex, false);
                              outputBufferIndex = decoder.dequeueOutputBuffer(bufferInfo, 0);
  
                          }
  
  
  
                          //===========
  
                      }
                      catch (IOException e)
                      {
                      }
                  }
  
                  decoder.stop();
  
              }
              catch (Exception e)
              {
              }
          }
      });
  </code></pre>
  
  <p>the full code:</p>
  
  <p><a href="https://gist.github.com/kenokabe/9029256" rel="noreferrer">https://gist.github.com/kenokabe/9029256</a></p>
  
  <p>also need Permission:</p>
  
  <pre><code> &lt;uses-permission android:name="android.permission.INTERNET"&gt;&lt;/uses-permission&gt;
   &lt;uses-permission android:name="android.permission.RECORD_AUDIO"&gt;&lt;/uses-permission&gt;
  </code></pre>
  
  <hr>
  
  <p>A member <a href="https://stackoverflow.com/users/294248/fadden">fadden</a> who works for Google told me </p>
  
  <p>Looks like I'm not setting position &amp; limit on the output buffer. </p>
  
  <p>I have read 
  <a href="https://stackoverflow.com/questions/21626156/vp8-encoding-nexus-5-returns-empty-0-frames/21630351#21630351">VP8 Encoding Nexus 5 returns empty/0-Frames</a> , but not sure how to implement correctly.</p>
  
  <hr>
  
  <p><strong>UPDATE: I sort of understood where to modify for</strong></p>
  
  <blockquote>
    <p>Looks like I'm not setting position &amp; limit on the output buffer.</p>
  </blockquote>
  
  <p><strong>, so add 2 lines within the while loop of Encoder and Decoder as follows:</strong></p>
  
  <pre><code> outputBuffer.position(bufferInfo.offset);
   outputBuffer.limit(bufferInfo.offset + bufferInfo.size);
  </code></pre>
  
  <p><a href="https://gist.github.com/kenokabe/9029256/revisions" rel="noreferrer">https://gist.github.com/kenokabe/9029256/revisions</a></p>
  
  <p>However the result is the same.</p>
  
  <p><strong>and now, I think, the errors:
   <code>W/SoftAAC2﹕ AAC decoder returned error 16388, substituting silence.</code> indicates this decoder fails completely from the first. It's again <code>the data is not seekable</code> issue.  <a href="https://stackoverflow.com/questions/3933387/seeking-in-aac-streams-on-android">Seeking in AAC streams on Android</a> It's very disappointing if the AAC decoder cannot handle the streaming data in this way but only with adding some header.</strong> </p>
  
  <hr>
  
  <p><strong>UPDATE2: UDP receiver did wrong, so modified</strong></p>
  
  <p><a href="https://gist.github.com/kenokabe/9029256" rel="noreferrer">https://gist.github.com/kenokabe/9029256</a></p>
  
  <p>Now, the error</p>
  
  <p><strong><code>W/SoftAAC2﹕ AAC decoder returned error 16388, substituting silence.</code>
  disappeared!!</strong></p>
  
  <p>So, it indicates the decoder works without an error, at least,</p>
  
  <p>however, this is the log of 1 cycle:</p>
  
  <pre><code>D/AudioRecoder﹕ 4096 bytes read
  D/AudioEncoder﹕ 360 bytes encoded
  D/UDP Receiver﹕ received !!! from /127.0.0.1:39000
  D/UDP Receiver﹕ 360 bytes received
  D/AudioDecoder﹕ 8192 bytes decoded
  </code></pre>
  
  <p>PCM(4096)->AACencoded(360)->UDP-AAC(360)->(supposed to be )PCM(8192)</p>
  
  <p>The final result is about 2x size of the original PCM, something is still wrong.</p>
  
  <hr>
  
  <p>So my Question here would be</p>
  
  <ol>
  <li><p>Can you properly optimize my sample code to work correctly?</p></li>
  <li><p>Is it a right way to use <code>AudioTrack</code> API to play the decoded PCM raw data on the fly, and can you show me the proper way to do that? A example code is appreciated.</p></li>
  </ol>
  
  <p>Thank you.</p>
  
  <p>PS. My project targets on Android4.1+(API16), I've read things are easier on API18(Andeoid 4.3+), but for obvious compatibility reasons, unfortunately, I have to skip MediaMuxer etc. here...</p>