title: Glitchy audio or broken video in fragmented MP4
body: <p>I'm working on small C++ library for muxing audio and video. This is basically a facade for FFMPEG functions and structures. The code is <a href="https://github.com/PookyFan/AVMuxer/tree/5658b6e4cf1cdef3d639e6ab482c4affc19b202b" rel="nofollow noreferrer">here</a> with minimal reproduction testing code <a href="https://gist.github.com/PookyFan/32ddd7317deb1c2d781538d7aff9ac60" rel="nofollow noreferrer">here</a> and as of now it seems like it's working fine... almost.</p>
  <p>For the record - my MP4 file is so-called &quot;fragmented MP4&quot;, with headers moved at the beginning of the file in a way that would allow to stream that file (ie. play it in a browser while it's being buffered). That's what these <code>movflags</code> I'm setting in <code>Mp4Muxer::writeHeader()</code> are for.</p>
  <p>While testing this library with raw H264 video stream and MP3 file (video is &lt;1 min long, MP3 - a few minutes long), I observed that:</p>
  <ul>
  <li>if I don't limit muxing audio stream when it's way &quot;ahead&quot; of the video (and it will be since MP3 is longer, so eventually video frames stop coming but audio frames still come in), all muxes just fine with no errors, but playing output MP4 with <code>ffplay</code> after just a few seconds results in the following log (and also frozen video, while audio keeps playing):</li>
  </ul>
  <pre><code>[h264 @ 0x7f90a40ae2c0] Invalid NAL unit size (2162119 &gt; 76779).0
  [h264 @ 0x7f90a40ae2c0] Error splitting the input into NAL units.
  [mp3float @ 0x7f90a4009540] Header missing  515KB sq=    0B f=0/0
  [h264 @ 0x7f90a40cb0c0] Invalid NAL unit size (-860010620 &gt; 17931).
  [h264 @ 0x7f90a40cb0c0] Error splitting the input into NAL units.
  [h264 @ 0x7f90a42bf440] Invalid NAL unit size (-168012642 &gt; 8000).
  [h264 @ 0x7f90a42bf440] Error splitting the input into NAL units.
  [h264 @ 0x7f90a42fa780] Invalid NAL unit size (-1843711407 &gt; 5683).
  [ and it repeats...]
  </code></pre>
  <ul>
  <li>even if I limit how much a stream can be &quot;ahead&quot; of the other, limiting it too much results in no muxed data in the output</li>
  <li>any other intermediate level of limiting how much one stream can be buffered in muxer relative to the other stream results in glitchy audio, with the following errors popping out every now and then in <code>ffplay</code> (the more strict limit is, the more often they are printed):</li>
  </ul>
  <pre><code>[mp3float @ 0x7f744c01b640] overread, skip -6 enddists: -1 -1=0/0 
  </code></pre>
  <p>Not limitting muxed audio (at all or enough) relative to muxed video also results in following messages in my muxing application:</p>
  <pre><code>[mp4 @ 0x55d0c6c21940] Delay between the first packet and last packet in the muxing queue is 10004898 &gt; 10000000: forcing output
  </code></pre>
  <p>For now, the fix is quite ugly and I don't even understand <em>why</em> it works, but before writting MP4 header I manually set a limit for frames buffered by muxer, like so:</p>
  <pre><code>formatCtxt-&gt;max_interleave_delta = 10000000LL * 10LL;
  </code></pre>
  <p>This way the muxer can store more packets of one stream that's way &quot;ahead&quot; of the other (maximum difference between DTS of the packets at the beginning and at the end of queue is set to 10x larger than default; it also gets rid of information log mentioned above). Obviously, I'd like to resolve it more properly, without hacking things like that.</p>
  <p>I was trying various things, including manual skipping of ID3 tags in MP3 file (but seems like FFMPEG handles them just fine and it didn't change anything). I was also experimenting with FLAC in MP4 instead of MP3. and while I know it's rather experimental thing, I encountered very similar problems with glitching audio (no problem with video being frozen when lots of audio data gets muxed, though). It also seems that problem with glitching audio or frozen video varies in scale depending on how large are input data chunks that I feed muxer with. For now, honestly, I'm out of ideas.</p>