title: OpenCV: reading frames from VideoCapture advances the video to bizarrely wrong location
body: <p>(I will put a 500 reputation bounty on this question as soon as it's eligible - unless the question got closed.)</p>
  
  <p><strong>Problem in one sentence</strong></p>
  
  <p>Reading frames from a <code>VideoCapture</code> advances the video much further than it's supposed to.</p>
  
  <p><strong>Explanation</strong></p>
  
  <p>I need to read and analyze frames from a 100 fps (according to <code>cv2</code> and VLC media player) video between certain time-intervals. In the minimal example that follows I am trying to read all the frames for the first ten seconds of a three minute video.</p>
  
  <p>I am creating a <code>cv2.VideoCapture</code> object from which I read frames until the  desired position in milliseconds is reached. In my actual code each frame is analyzed, but that fact is irrelevant in order to showcase the error.</p>
  
  <p>Checking the current frame and millisecond position of the <code>VideoCapture</code> after reading the frames yields correct values, so the <code>VideoCapture</code> <strong>thinks</strong> it is at the right position - but it is not. Saving an image of the last read frame reveals that my iteration is grossly overshooting the destination time <strong>by over two minutes</strong>.</p>
  
  <p>What's even more bizarre is that if I manually set the millisecond position of the capture with <code>VideoCapture.set</code> to 10 seconds (the same value <code>VideoCapture.get</code> returns after reading the frames) and save an image, the video is at (almost) the right position!</p>
  
  <p><strong>Demo video file</strong></p>
  
  <p>In case you want to run the MCVE, you need the demo.avi video file.
  You can download it <a href="https://workupload.com/file/Q4ELvBC" rel="noreferrer"><strong>HERE</strong></a>.</p>
  
  <p><strong>MCVE</strong></p>
  
  <p>This MCVE is carefully crafted and commented. Please leave a comment under the question if anything remains unclear.</p>
  
  <p>If you are using OpenCV 3 you have to replace all instances of <code>cv2.cv.CV_</code> with <code>cv2.</code>. (The problem occurs in both versions for me.)</p>
  
  <pre><code>import cv2
  
  # set up capture and print properties
  print 'cv2 version = {}'.format(cv2.__version__)
  cap = cv2.VideoCapture('demo.avi')
  fps = cap.get(cv2.cv.CV_CAP_PROP_FPS)
  pos_msec = cap.get(cv2.cv.CV_CAP_PROP_POS_MSEC)
  pos_frames = cap.get(cv2.cv.CV_CAP_PROP_POS_FRAMES)
  print ('initial attributes: fps = {}, pos_msec = {}, pos_frames = {}'
        .format(fps, pos_msec, pos_frames))
  
  # get first frame and save as picture
  _, frame = cap.read()
  cv2.imwrite('first_frame.png', frame)
  
  # advance 10 seconds, that's 100*10 = 1000 frames at 100 fps
  for _ in range(1000):
      _, frame = cap.read()
      # in the actual code, the frame is now analyzed
  
  # save a picture of the current frame
  cv2.imwrite('after_iteration.png', frame)
  
  # print properties after iteration
  pos_msec = cap.get(cv2.cv.CV_CAP_PROP_POS_MSEC)
  pos_frames = cap.get(cv2.cv.CV_CAP_PROP_POS_FRAMES)
  print ('attributes after iteration: pos_msec = {}, pos_frames = {}'
        .format(pos_msec, pos_frames))
  
  # assert that the capture (thinks it) is where it is supposed to be
  # (assertions succeed)
  assert pos_frames == 1000 + 1 # (+1: iteration started with second frame)
  assert pos_msec == 10000 + 10
  
  # manually set the capture to msec position 10010
  # note that this should change absolutely nothing in theory
  cap.set(cv2.cv.CV_CAP_PROP_POS_MSEC, 10010)
  
  # print properties  again to be extra sure
  pos_msec = cap.get(cv2.cv.CV_CAP_PROP_POS_MSEC)
  pos_frames = cap.get(cv2.cv.CV_CAP_PROP_POS_FRAMES)
  print ('attributes after setting msec pos manually: pos_msec = {}, pos_frames = {}'
        .format(pos_msec, pos_frames))
  
  # save a picture of the next frame, should show the same clock as
  # previously taken image - but does not
  _, frame = cap.read()
  cv2.imwrite('after_setting.png', frame)
  </code></pre>
  
  <p><strong>MCVE output</strong></p>
  
  <p>The <code>print</code> statements produce the following output.</p>
  
  <blockquote>
    <p>cv2 version = 2.4.9.1<br>
    initial attributes: fps = 100.0, pos_msec = 0.0, pos_frames = 0.0<br>
    attributes after reading: pos_msec = 10010.0, pos_frames = 1001.0<br>
    attributes after setting msec pos manually: pos_msec = 10010.0, pos_frames = 1001.0<br></p>
  </blockquote>
  
  <p>As you can see, all properties have the expected values.</p>
  
  <p><code>imwrite</code> saves the following pictures.</p>
  
  <p>first_frame.png
  <a href="https://i.stack.imgur.com/ED7Fl.png" rel="noreferrer"><img src="https://i.stack.imgur.com/ED7Fl.png" alt="first_frame.png"></a></p>
  
  <p>after_iteration.png
  <a href="https://i.stack.imgur.com/X2zG3.png" rel="noreferrer"><img src="https://i.stack.imgur.com/X2zG3.png" alt="after_iteration.png"></a></p>
  
  <p>after_setting.png
  <a href="https://i.stack.imgur.com/HIrxH.png" rel="noreferrer"><img src="https://i.stack.imgur.com/HIrxH.png" alt="after_setting.png"></a></p>
  
  <p>You can see the problem in the second picture. The target of 9:26:15 (real time clock in picture) is missed by over two minutes. Setting the target time manually (third picture) sets the video to (almost) the correct position. </p>
  
  <p>What am I doing wrong and how do I fix it?</p>
  
  <p><strong>Tried so far</strong></p>
  
  <p>cv2 2.4.9.1 @ Ubuntu 16.04<br>
  cv2 2.4.13 @ Scientific Linux 7.3 (three computers)<br>
  cv2 3.1.0 @ Scientific Linux 7.3 (three computers)<br></p>
  
  <p>Creating the capture with </p>
  
  <pre><code>cap = cv2.VideoCapture('demo.avi', apiPreference=cv2.CAP_FFMPEG)
  </code></pre>
  
  <p>and</p>
  
  <pre><code>cap = cv2.VideoCapture('demo.avi', apiPreference=cv2.CAP_GSTREAMER)
  </code></pre>
  
  <p>in OpenCV 3 (version 2 does not seem to have the <code>apiPreference</code> argument).
  Using <code>cv2.CAP_GSTREAMER</code> takes extremely long (about 2-3 minutes to run the MCVE) but both api-preferences produce the same incorrect images.</p>
  
  <p>When using <code>ffmpeg</code>  directly to read frames (credit to <a href="http://zulko.github.io/blog/2013/09/27/read-and-write-video-frames-in-python-using-ffmpeg/" rel="noreferrer">this</a> tutorial) the correct output images are produced.</p>
  
  <pre><code>import numpy as np
  import subprocess as sp
  import pylab
  
  # video properties
  path = './demo.avi'
  resolution = (593, 792)
  framesize = resolution[0]*resolution[1]*3
  
  # set up pipe
  FFMPEG_BIN = "ffmpeg"
  command = [FFMPEG_BIN,
             '-i', path,
             '-f', 'image2pipe',
             '-pix_fmt', 'rgb24',
             '-vcodec', 'rawvideo', '-']
  pipe = sp.Popen(command, stdout = sp.PIPE, bufsize=10**8)
  
  # read first frame and save as image
  raw_image = pipe.stdout.read(framesize)
  image = np.fromstring(raw_image, dtype='uint8')
  image = image.reshape(resolution[0], resolution[1], 3)
  pylab.imshow(image)
  pylab.savefig('first_frame_ffmpeg_only.png')
  pipe.stdout.flush()
  
  # forward 1000 frames
  for _ in range(1000):
      raw_image = pipe.stdout.read(framesize)
      pipe.stdout.flush()
  
  # save frame 1001
  image = np.fromstring(raw_image, dtype='uint8')
  image = image.reshape(resolution[0], resolution[1], 3)
  pylab.imshow(image)
  pylab.savefig('frame_1001_ffmpeg_only.png')
  
  pipe.terminate()
  </code></pre>
  
  <p>This produces the correct result! (Correct timestamp 9:26:15)</p>
  
  <p>frame_1001_ffmpeg_only.png:
  <a href="https://i.stack.imgur.com/VVJqA.png" rel="noreferrer"><img src="https://i.stack.imgur.com/VVJqA.png" alt="frame_1001_ffmpeg_only.png"></a></p>
  
  <p><strong>Additional information</strong></p>
  
  <p>In the comments I was asked for my <code>cvconfig.h</code> file. I only seem to have this file for cv2 version 3.1.0 under <code>/opt/opencv/3.1.0/include/opencv2/cvconfig.h</code>. </p>
  
  <p><a href="https://pastebin.com/dK9AFfan" rel="noreferrer"><strong>HERE</strong></a> is a paste of this file.</p>
  
  <p>In case it helps, I was able to extract the following video information with <code>VideoCapture.get</code>.</p>
  
  <blockquote>
    <p>brightness             0.0<br>
    contrast               0.0<br>
    convert_rgb            0.0<br>
    exposure               0.0<br>
    format                 0.0<br>
    fourcc                 1684633187.0<br>
    fps                    100.0<br>
    frame_count            18000.0<br>
    frame_height           593.0<br>
    frame_width            792.0<br>
    gain                   0.0<br>
    hue                    0.0<br>
    mode                   0.0<br>
    openni_baseline        0.0<br>
    openni_focal_length    0.0<br>
    openni_frame_max_depth 0.0<br>
    openni_output_mode     0.0<br>
    openni_registration    0.0<br>
    pos_avi_ratio          0.01<br>
    pos_frames             0.0<br>
    pos_msec               0.0<br>
    rectification          0.0<br>
    saturation             0.0<br></p>
  </blockquote>