title: Video as texture for openGLES
body: <p>I've a local video that I want to pass as a texture to an openGL shader.
  I'm aware of a number of posts covering related topics, some being old or weird, and some I could not get to work.</p>
  
  <p>It sounds that the way to go is:</p>
  
  <ul>
  <li>load the video</li>
  <li>get the video output as a <code>CVPixelBuffer</code></li>
  <li>then approaches vary around yuv vs rgb, <code>CVOpenGLESTextureCacheCreateTextureFromImage</code> vs <code>glTexImage2D</code> etc. If there's no specific reason to use yuv, I'd rather stick to rgb.</li>
  </ul>
  
  <p>My code is able to render <code>UIImages</code> but I could not adapt it to video.</p>
  
  <p>It seems that <code>CVOpenGLESTextureCacheCreateTextureFromImage</code> is now recommended over <code>glTexImage2D</code> to pass the video frame to the openGL program. <a href="https://stackoverflow.com/questions/27268663/video-as-texture-in-opengles2-0">Some</a> convert the video output buffer as an image and then pass it down the pipeline, but this sounds inefficient.</p>
  
  <p>As a start, here is how I get the video pixel buffer that I pass to the view managing the GL program (you can probably skip this as I think it works ok):</p>
  
  <pre><code>import UIKit
  import AVFoundation
  
  class ViewController: UIViewController {
      // video things
      var videoOutput: AVPlayerItemVideoOutput!
      var player: AVPlayer!
      var playerItem: AVPlayerItem!
      var isVideoReady = false
  
      override func viewDidLoad() {
          super.viewDidLoad()
          self.setupVideo()
      }
  
      func setupVideo() -&gt; Void {
          let url = Bundle.main.urlForResource("myVideoName", withExtension: "mp4")!
  
          let outputSettings: [String: AnyObject] = ["kCVPixelBufferPixelFormatTypeKey": Int(kCVPixelFormatType_32BGRA)]
          self.videoOutput = AVPlayerItemVideoOutput.init(pixelBufferAttributes: outputSettings)
          self.player = AVPlayer()
          let asset = AVURLAsset(url: url)
  
  
          asset.loadValuesAsynchronously(forKeys: ["playable"]) {
              var error: NSError? = nil
              let status = asset.statusOfValue(forKey: "playable", error: &amp;error)
              switch status {
              case .loaded:
                  self.playerItem = AVPlayerItem(asset: asset)
                  self.playerItem.add(self.videoOutput)
                  self.player.replaceCurrentItem(with: self.playerItem)
                  self.isVideoReady = true
              case .failed:
                  print("failed")
              case .cancelled:
                  print("cancelled")
              default:
                  print("default")
              }
          }
      }
  
      // this function is called just before that the openGL program renders
      // and can be used to update the texture. (all the GL program is already initialized at this point)
      func onGlRefresh(glView: OpenGLView) -&gt; Void {
          if self.isVideoReady {
              let pixelBuffer = self.videoOutput.copyPixelBuffer(forItemTime: self.playerItem.currentTime(), itemTimeForDisplay: nil)
              glView.pixelBuffer = pixelBuffer
          }
      }
  }
  </code></pre>
  
  <p>This seems to work fine, even though I'm not able to really test it :)</p>
  
  <p>So now I've an <code>CVPixelBuffer</code> available (as soon as the video is loaded)
  How can I pass it to a GL program?</p>
  
  <p>This code works for an <code>CGImage?</code></p>
  
  <pre><code>    // textureSource is an CGImage?
      guard let textureSource = textureSource else { return }
      let width: Int = textureSource.width
      let height: Int = textureSource.height
  
      let spriteData = UnsafeMutablePointer&lt;GLubyte&gt;(calloc(Int(UInt(CGFloat(width) * CGFloat(height) * 4)), sizeof(GLubyte.self)))
  
      let colorSpace = textureSource.colorSpace!
  
      let spriteContext: CGContext = CGContext(data: spriteData, width: width, height: height, bitsPerComponent: 8, bytesPerRow: width*4, space: colorSpace, bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue)!
      spriteContext.draw(in: CGRect(x: 0, y: 0, width: CGFloat(width), height: CGFloat(height)), image: textureSource)
  
      glBindTexture(GLenum(GL_TEXTURE_2D), _textureId!)
      glTexImage2D(GLenum(GL_TEXTURE_2D), 0, GL_RGBA, GLsizei(width), GLsizei(height), 0, GLenum(GL_RGBA), UInt32(GL_UNSIGNED_BYTE), spriteData)
  
      free(spriteData)
  </code></pre>
  
  <p>but I could not get my head around how to adapt it efficiently to a <code>CVPixelBuffer</code></p>
  
  <p>I'm happy to share more code if needed, but I thought this post was already long enough :)</p>
  
  <p>========== EDIT ==========</p>
  
  <p>I've looked at a bunch of repos (that all copy from Apple's <a href="https://developer.apple.com/library/ios/samplecode/GLCameraRipple/Introduction/Intro.html" rel="nofollow noreferrer">CameraRipple</a> and Ray Wenderlich's <a href="https://www.raywenderlich.com/3664/opengl-tutorial-for-ios-opengl-es-2-0" rel="nofollow noreferrer">tutorial</a>) and <a href="https://github.com/gsabran/Swift-openGL-Video/tree/d7f7487bff2ca1d2a5d016bf0731d34ef5cb6f28" rel="nofollow noreferrer">here</a> is the github repo of what I've so far (I'll keep it alive to preserve the link) It's not ideal but I don't want to paste too much code here. I've been able to get some video texturing to work but:</p>
  
  <ul>
  <li>the color are wrong</li>
  <li>the display in the simulator is different from the one on a device. In the simulator only the left half of the video is displayed (and covers the entire screen) and there are some vertical aberrations.</li>
  </ul>
  
  <p>The simulator issues looks like they might be related to XCode 8 being in beta, but I'm not sure about that...</p>