title: Decode an H264 stream using the VideoToolbox API (kVTVideoDecoderBadDataErr)
body: <p>my goal is to encode the main framebuffer of my Windows machine using nvenc and stream its content to my iPad using the VideoToolbox API</p>
  <p>The code I use to encode the h264 stream is basically a copy/paste of <a href="https://github.com/NVIDIA/video-sdk-samples/tree/master/nvEncDXGIOutputDuplicationSample" rel="nofollow noreferrer">https://github.com/NVIDIA/video-sdk-samples/tree/master/nvEncDXGIOutputDuplicationSample</a> the only change is that instead of writing to a file, I do send the data</p>
  <p>For the decoding I do use <a href="https://github.com/zerdzhong/SwfitH264Demo/blob/master/SwiftH264/ViewController.swift#L71" rel="nofollow noreferrer">https://github.com/zerdzhong/SwfitH264Demo/blob/master/SwiftH264/ViewController.swift#L71</a></p>
  <p>The encoding work perfectly when I write all the contents to a file, I am able to use a h264-&gt;mp4 online converter without issue, the problem is that the decoder gives me the error kVTVideoDecoderBadDataErr in the function decompressionSessionDecodeFrameCallback</p>
  <p>So for what I tried:</p>
  <ul>
  <li>Firsly using an h264 analyzer I found that the frame order are: 7/8/5/5/5/5/1...</li>
  <li>I found that nvenc does encode the frames 7/8/5/5/5/5 in only one packet</li>
  <li>I did try to separate this packet into multiple ones using the sequence (0x00 0x00 0x00 0x01), it gave me the frames 7/8/5 separately</li>
  <li>As you can see I only got one 5 frame which is around 100KB, the H264 analyzer said that there are four 5 frames (which are something like 40KB, 20KB, 30KB, 10KB)</li>
  <li>Using a hex file viewer I saw that the sequence separating those 5 frames were (0x00 0x00 0x01) instead, tried to also separate them but I got the exact same VideoToolbox error while decompressing</li>
  </ul>
  <p>here is the code I use to separate and send the frames:
  The protocol is simply PACKET_SIZE-&gt;PACKET_DATA
  The swift code is able to read the NALU types so I am confident that this is not the issue</p>
  <pre class="lang-rust prettyprint-override"><code>    unsafe {
          Setup();
          loop {
              CaptureFrame();
  
              let frame_count = GetDataCount();
              if frame_count == 0 {
                  continue;
              }
  
              for i in 0..frame_count {
                  let size = RetrieveDataSize(i as i32);
                  let size_slice = &amp;(u32::to_le_bytes(size as u32));
  
                  let data = RetrieveData(i as i32);
                  let data_slice = std::slice::from_raw_parts(data, size);
  
                  let mut last_frame = 0;
  
                  for x in 0..size {
                      if data_slice[x] == 0 &amp;&amp;
                          data_slice[x + 1] == 0 &amp;&amp;
                          data_slice[x + 2] == 0 &amp;&amp;
                          data_slice[x + 3] == 1 {
                          let frame_size = x - last_frame;
                          if frame_size &gt; 0 {
                              let frame_data = &amp;data_slice[last_frame..x];
                              stream.write(&amp;(u32::to_le_bytes(frame_size as u32))).unwrap();
                              stream.write(frame_data).unwrap();
                              println!(&quot;SEND MULTIPLE {}&quot;, frame_size);
                          }
  
                          last_frame = x;
                          println!(&quot;NALU {}&quot;, data_slice[x + 4] &amp; 0x1F);
                          //println!(&quot;TEST {} {}&quot;,i, size);
                          continue;
                      }
                  }
                  // Packet was a single frame
                  let frame_size = size - last_frame;
                  let frame_data = &amp;data_slice[last_frame..size];
                  stream.write(&amp;(u32::to_le_bytes(frame_size as u32))).unwrap();
                  stream.write(frame_data).unwrap();
                  println!(&quot;SEND SINGLE {} {}&quot;, last_frame, size);
              }
          }
      }
  </code></pre>
  <p>It could be concerning the texture format, VideoToolbox makes mentioning of kCVPixelFormatType_420YpCbCr8BiPlanarFullRange, and the NVENC codes mentions YUV420 and NV12, I am unsure if both are the same or not</p>
  <p>Here is my format description:</p>
  <pre><code>Optional(&lt;CMVideoFormatDescription 0x2823dd410 [0x1e0921e20]&gt; {
      mediaType:'vide' 
      mediaSubType:'avc1' 
      mediaSpecific: {
          codecType: 'avc1'       dimensions: 3840 x 2160 
      } 
      extensions: {{
      CVFieldCount = 1;
      CVImageBufferChromaLocationBottomField = Left;
      CVImageBufferChromaLocationTopField = Left;
      CVPixelAspectRatio =     {
          HorizontalSpacing = 1;
          VerticalSpacing = 1;
      };
      FullRangeVideo = 0;
      SampleDescriptionExtensionAtoms =     {
          avcC = {length = 41, bytes = 0x01640033 ffe10016 67640033 ac2b401e ... 68ee3cb0 fdf8f800 };
      };
  }}
  })
  </code></pre>