title: streaming H.264 over RTP with libavformat
body: <p>I've been trying over the past week to implement H.264 streaming over RTP, using x264 as an encoder and libavformat to pack and send the stream. Problem is, as far as I can tell it's not working correctly.</p>
  
  <p>Right now I'm just encoding random data (x264_picture_alloc) and extracting NAL frames from libx264. This is fairly simple:</p>
  
  <pre><code>x264_picture_t pic_out;
  x264_nal_t* nals;
  int num_nals;
  int frame_size = x264_encoder_encode(this-&gt;encoder, &amp;nals, &amp;num_nals, this-&gt;pic_in, &amp;pic_out);
  
  if (frame_size &lt;= 0)
  {
      return frame_size;
  }
  
  // push NALs into the queue
  for (int i = 0; i &lt; num_nals; i++)
  {
      // create a NAL storage unit
      NAL nal;
      nal.size = nals[i].i_payload;
      nal.payload = new uint8_t[nal.size];
      memcpy(nal.payload, nals[i].p_payload, nal.size);
  
      // push the storage into the NAL queue
      {
          // lock and push the NAL to the queue
          boost::mutex::scoped_lock lock(this-&gt;nal_lock);
          this-&gt;nal_queue.push(nal);
      }
  }
  </code></pre>
  
  <p><code>nal_queue</code> is used for safely passing frames over to a Streamer class which will then send the frames out. Right now it's not threaded, as I'm just testing to try to get this to work. Before encoding individual frames, I've made sure to initialize the encoder.</p>
  
  <p>But I don't believe x264 is the issue, as I can see frame data in the NALs it returns back.
  Streaming the data is accomplished with libavformat, which is first initialized in a Streamer class:</p>
  
  <pre><code>Streamer::Streamer(Encoder* encoder, string rtp_address, int rtp_port, int width, int height, int fps, int bitrate)
  {
      this-&gt;encoder = encoder;
  
      // initalize the AV context
      this-&gt;ctx = avformat_alloc_context();
      if (!this-&gt;ctx)
      {
          throw runtime_error("Couldn't initalize AVFormat output context");
      }
  
      // get the output format
      this-&gt;fmt = av_guess_format("rtp", NULL, NULL);
      if (!this-&gt;fmt)
      {
          throw runtime_error("Unsuitable output format");
      }
      this-&gt;ctx-&gt;oformat = this-&gt;fmt;
  
      // try to open the RTP stream
      snprintf(this-&gt;ctx-&gt;filename, sizeof(this-&gt;ctx-&gt;filename), "rtp://%s:%d", rtp_address.c_str(), rtp_port);
      if (url_fopen(&amp;(this-&gt;ctx-&gt;pb), this-&gt;ctx-&gt;filename, URL_WRONLY) &lt; 0)
      {
          throw runtime_error("Couldn't open RTP output stream");
      }
  
      // add an H.264 stream
      this-&gt;stream = av_new_stream(this-&gt;ctx, 1);
      if (!this-&gt;stream)
      {
          throw runtime_error("Couldn't allocate H.264 stream");
      }
  
      // initalize codec
      AVCodecContext* c = this-&gt;stream-&gt;codec;
      c-&gt;codec_id = CODEC_ID_H264;
      c-&gt;codec_type = AVMEDIA_TYPE_VIDEO;
      c-&gt;bit_rate = bitrate;
      c-&gt;width = width;
      c-&gt;height = height;
      c-&gt;time_base.den = fps;
      c-&gt;time_base.num = 1;
  
      // write the header
      av_write_header(this-&gt;ctx);
  }
  </code></pre>
  
  <p>This is where things seem to go wrong. <code>av_write_header</code> above seems to do absolutely nothing; I've used wireshark to verify this. For reference, I use <code>Streamer streamer(&amp;enc, "10.89.6.3", 49990, 800, 600, 30, 40000);</code> to initialize the Streamer instance, with <code>enc</code> being a reference to an <code>Encoder</code> object used to handle x264 previously.</p>
  
  <p>Now when I want to stream out a NAL, I use this:</p>
  
  <pre><code>// grab a NAL
  NAL nal = this-&gt;encoder-&gt;nal_pop();
  cout &lt;&lt; "NAL popped with size " &lt;&lt; nal.size &lt;&lt; endl;
  
  // initalize a packet
  AVPacket p;
  av_init_packet(&amp;p);
  p.data = nal.payload;
  p.size = nal.size;
  p.stream_index = this-&gt;stream-&gt;index;
  
  // send it out
  av_write_frame(this-&gt;ctx, &amp;p);
  </code></pre>
  
  <p>At this point, I can see RTP data appearing over the network, and it looks like the frames I've been sending, even including a little copyright blob from x264. <strong>But</strong>, no player I've used has been able to make any sense of the data. VLC quits wanting an SDP description, which <a href="https://stackoverflow.com/questions/6023668/rtp-video-stream-is-presence-of-sdp-file-mandatory">apparently isn't required</a>.</p>
  
  <p>I then tried to play it through <code>gst-launch</code>:</p>
  
  <p><code>gst-launch udpsrc port=49990 ! rtph264depay ! decodebin ! xvimagesink</code></p>
  
  <p>This will sit waiting for UDP data, but when it is received, I get:</p>
  
  <blockquote>
    <p>ERROR: element /GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0: No RTP
    format was negotiated. Additional debug info:
    gstbasertpdepayload.c(372): gst_base_rtp_depayload_chain ():
    /GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0: Input buffers
    need to have RTP caps set on them. This is usually achieved by setting
    the 'caps' property of the upstream source element (often udpsrc or
    appsrc), or by putting a capsfilter element before the depayloader and
    setting the 'caps' property on that. Also see
    <a href="http://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/gst/rtp/README" rel="nofollow noreferrer">http://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/gst/rtp/README</a></p>
  </blockquote>
  
  <p>As I'm not using GStreamer to stream itself, I'm not quite sure what it means with RTP caps. But, it makes me wonder if I'm not sending enough information over RTP to describe the stream. I'm pretty new to video and I feel like there's some key thing I'm missing here. Any hints?</p>