title: LibAV - what approach to take for realtime audio and video capture?
body: <p>I'm using libav to encode raw RGB24 frames to h264 and muxing it to flv. This works
  all fine and I've streamed for more then 48 hours w/o any problems!  My next step
  is to add audio to the stream. I'll be capturing live audio and I want to encode it 
  in real time using speex, mp3 or nelly moser.  </p>
  
  <p><strong>Background info</strong>  </p>
  
  <p>I'm new to digital audio and therefore I might be doing things wrong. But basically my application gets a "float" buffer with interleaved audio. This "audioIn" function gets called by the application framework I'm using. The buffer contains 256 samples per channel, 
  and I have 2 channels. Because I might be mixing terminology, this is how I use the 
  data:</p>
  
  <pre><code>// input = array with audio samples
  // bufferSize = 256
  // nChannels = 2
  void audioIn(float * input, int bufferSize, int nChannels) {
      // convert from float to S16
          short* buf = new signed short[bufferSize * 2];
      for(int i = 0; i &lt; bufferSize; ++i) {  // loop over all samples
          int dx = i * 2;
          buf[dx + 0] = (float)input[dx + 0] * numeric_limits&lt;short&gt;::max();  // convert frame  of the first channel
          buf[dx + 1] = (float)input[dx + 1] * numeric_limits&lt;short&gt;::max();  // convert frame  of the second channel
      }
  
          // add this to the libav wrapper. 
      av.addAudioFrame((unsigned char*)buf, bufferSize, nChannels);
  
      delete[] buf;
  }
  </code></pre>
  
  <p>Now that I have a buffer, where each sample is 16 bits, I pass this <code>short* buffer</code>,  to my 
  wrapper <code>av.addAudioFrame()</code>  function.  In this function I create a buffer, before I encode 
  the audio. From what I read, the <code>AVCodecContext</code> of the audio encoder sets the <code>frame_size</code>. This frame_size must match the number of samples in the buffer when calling <code>avcodec_encode_audio2()</code>.  Why I think this, is because of what is <a href="http://libav.org/doxygen/master/group__lavc__encoding.html#ga93a49fbd0973b216dcb8a8c5dffe1d82" rel="nofollow">documented here</a>. </p>
  
  <p>Then, especially the line: 
  If it is not set, <code>frame-&gt;nb_samples</code> must be equal to <code>avctx-&gt;frame_size</code> for all frames except the last.*(Please correct me here if I'm wrong about this).</p>
  
  <p>After encoding I call <code>av_interleaved_write_frame()</code> to actually write the frame. 
  When I use mp3 as codec my application runs for about 1-2 minutes and then my server, which is receiving the video/audio stream (flv, tcp), disconnects with a message "<code>Frame too large: 14485504</code>".  This message is generated because the rtmp-server is getting a frame which is way to big. And this is probably due to the fact I'm not interleaving correctly with libav. </p>
  
  <p><strong>Questions:</strong></p>
  
  <ul>
  <li><p>There quite some bits I'm not sure of, even when going through the source code of libav and therefore I hope if someone has an working example of encoding audio which comes from a buffer which which comes from "outside" libav (i.e. your own application). i.e. how do you create a buffer which is large enough for the encoder? How do you make the "realtime" streaming work when you need to wait on this buffer to fill up? </p></li>
  <li><p>As I wrote above I need to keep track of a buffer before I can encode. Does someone else has some code which does this? I'm using AVAudioFifo now. The functions which encodes the audio and fills/read the buffer is here too: <a href="https://gist.github.com/62f717bbaa69ac7196be" rel="nofollow">https://gist.github.com/62f717bbaa69ac7196be</a></p></li>
  <li><p>I compiled with --enable-debug=3 and disable optimizations, but I'm not seeing any
  debug information. How can I make libav more verbose? </p></li>
  </ul>
  
  <p>Thanks!</p>