title: Decoding h264 / avc with OpenCV
body: <p>I am trying to decode the h264 stream from <a href="https://github.com/NetrisTV/ws-scrcpy/" rel="nofollow noreferrer">ws-scrcpy</a> frame by frame to apply some computer vision operations. (ScrCpy is basically a mobile device application that broadcasts the device screen via h264. WS-ScrCpy is a web application that is able to receive and display the stream in a browser).</p>
  <p>Since I need to do the computer vision operations in RGB space and on a server (via nodejs), I need to convert frames first from YUV to RGB format. Since JS-only implementations are slow, I am trying to get this done with <a href="https://github.com/justadudewhohacks/opencv4nodejs/" rel="nofollow noreferrer">OpenCV</a>.</p>
  <p>Unfortunately, the decoded frames look like this:</p>
  <p><a href="https://i.stack.imgur.com/5JQlJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5JQlJ.png" alt="yuv -&gt; RGB" /></a></p>
  <p>Here is what I do in more detail:</p>
  <p>I use <a href="https://github.com/mbebenita/Broadway" rel="nofollow noreferrer">Broadway</a> <a href="https://github.com/NetrisTV/ws-scrcpy/blob/master/vendor/Broadway/Decoder.js" rel="nofollow noreferrer">Acv decoder</a> to decode data (NALUs?- received via websocket) into frames. Then I use OpenCV to convert YUV frames to RGB.</p>
  <pre><code>import Avc from './vendor/Broadway/Decoder'
  //...
  this.avc.onPictureDecoded = (buffer: Uint8Array, width: number, height: number) =&gt; {
    const mYUV = new cv.Mat(h + h / 2, w, cv.CV_8UC1, buffer)
    const mRgb = mYUV.cvtColor(cv.COLOR_YUV420p2BGR)
    cv.imwrite('./del/test.png', mRgb)
  }
  </code></pre>
  <p>The fact that the image size of the resulting Mat object is correct, tells me that I am doing something right.
  But as you can see there also is obviously something wrong.</p>
  <p>Any idea what the problem might be?</p>