title: Dual H264 videos inside single MPEG-TS stream with GStreamer: audio/video sync?
body: <p>For a research prototype, I need to have two synchronized HD video streams and one audio stream inside a streamable container for real-time communication (basically Skype with an additional secondary video stream). The only container format I've found that works as intended is MPEG-TS. I have two working pipelines which I put together based on various examples, but there's a number of things I haven't fully understood yet. First of all, the pipelines:</p>
  
  <p>Sender pipeline (with videotestsrc as a stand-in):</p>
  
  <pre><code>gst-launch-1.0 -ve mpegtsmux name=mux ! rtpmp2tpay ! udpsink host=$1 port=5000 \
      v4l2src                   ! video/x-raw,width=1280,height=720                ! videoconvert ! queue leaky=downstream ! x264enc noise-reduction=10000 tune=zerolatency byte-stream=true threads=2 key-int-max=15 ! mux. \
      videotestsrc is-live=true ! video/x-raw,width=1280,height=720,framerate=10/1 ! videoconvert ! queue leaky=downstream ! x264enc noise-reduction=10000 tune=zerolatency byte-stream=true threads=2 key-int-max=15 ! mux. \
      pulsesrc                  ! audio/x-raw,channels=1,rate=8000                 ! queue leaky=downstream ! opusenc bitrate=8000 ! mux.
  </code></pre>
  
  <p>Receiver pipeline:</p>
  
  <pre><code>gst-launch-1.0 -ve udpsrc port=5000 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=MP2T ! rtpjitterbuffer ! rtpmp2tdepay ! tsparse ! tsdemux name=mux \
      mux. ! h264parse ! queue leaky=downstream ! avdec_h264 ! videoconvert ! fpsdisplaysink sync=false \
      mux. ! h264parse ! queue leaky=downstream ! avdec_h264 ! videoconvert ! fpsdisplaysink sync=false \
      mux. ! queue leaky=downstream ! opusdec ! autoaudiosink sync=false
  
  </code></pre>
  
  <p>This setup generates about 500 kB/s of network traffic, which is pretty close to the expected value, and works just fine on localhost. I'm using the echo plugin for Pulseaudio for echo cancellation, so that's also covered.</p>
  
  <p>Most of my previous issues (see below) have now been resolved by a) using MPEG2-TS inside RTP on the sender side, and <code>rtpjitterbuffer</code> on the receiver side, and b) by using <code>queue leaky=downstream</code> everywhere. As per Florian Zwoch's suggestion, I also removed intra-refresh from x264enc.</p>
  
  <p>The only issue that's now left is that I have a constant, but unpredictable offset between audio and video. Occasionally, it's almost in sync and also stays that way, but on other occasions, it's off by 1-2 seconds, which is enough to be annoying. I assume I could somehow get rid of this by tweaking the queue parameters?</p>
  
  <hr>
  
  <p>Original set of issues, preserved for archiving:</p>
  
  <ol>
  <li><p>When actually using a real network between two separate machines (high-bandwidth WiFi with nearly 100% signal strength), the quality degrades massively. There doesn't seem to be any noticeable packet loss, and 500 kB/s isn't exactly taxing for this WiFi connection. What gives?</p></li>
  <li><p>What exactly is the function of the queue elements on the receiver side? AFAICT if I set max-size... properties to 0, then they would just buffer indefinitely?</p></li>
  <li><p>Even on a fairly powerful machine, the sender keeps issuing warnings - how can an 8 kBit/s audio encoder "fall behind"?</p></li>
  </ol>
  
  <pre><code>WARNING: from element /GstPipeline:pipeline0/GstPulseSrc:pulsesrc0: Can't record audio fast enough
  Additional debug info:
  gstaudiobasesrc.c(849): gst_audio_base_src_create (): /GstPipeline:pipeline0/GstPulseSrc:pulsesrc0:
  Dropped 1600 samples. This is most likely because downstream can't keep up and is consuming samples too slowly.
  </code></pre>