title: Problem to remove distortion (after camera-calibration) from image
body: <p>The <a href="https://ibb.co/b1hqDsZ" rel="nofollow noreferrer">image</a> is a single frame of a 90-minute video.</p>
  <p>If one draws a line from the top left corner (see, <a href="https://ibb.co/FnT90WP" rel="nofollow noreferrer">left-distorted</a>) to the right (see, <a href="https://ibb.co/6sJc9tz" rel="nofollow noreferrer">right-distorted</a>), the distortion is visible by looking at the top-line (see, <a href="https://ibb.co/7kVdW5W" rel="nofollow noreferrer">middle-distorted</a>).</p>
  <p>The white line between the points to the left and the right should be straight but has a slight U-shape compared to the straight line in red.</p>
  <p>Performing a camera calibration with open-cv (<a href="https://docs.opencv.org/3.4.15/dc/dbb/tutorial_py_calibration.html" rel="nofollow noreferrer">reference</a>) using a sample of entire frames from calibration videos (for some examples see, <a href="https://ibb.co/JvgkYw5" rel="nofollow noreferrer">indoor</a>, <a href="http://ibb.co/zmVbN88" rel="nofollow noreferrer">outdoor-1</a>, and <a href="https://ibb.co/KDPLMmP" rel="nofollow noreferrer">outdoor-2</a>) results in this <a href="https://ibb.co/X7ZHK0b" rel="nofollow noreferrer">undistorted</a> image.</p>
  <p>Here, the line from the top left corner (see, <a href="https://ibb.co/1zxKrkN" rel="nofollow noreferrer">left-undistorted</a>) to the right (see, <a href="https://ibb.co/KFfk9dD" rel="nofollow noreferrer">right-undistorted</a>) overcorrects the distortion as the difference between the red-line and the white has become an inverse U shape (see, <a href="https://ibb.co/tmsdqQ4" rel="nofollow noreferrer">middle-undistorted</a>).</p>
  <p>I took the recording with an iPhone 11, and I am using python 3.8.8 and open-cv 4.5.3.</p>
  <p>I followed the advice I could find at StackOverflow (and through the most popular search results), but using any checkerboard (variation of size, camera angle, distance, and setting) does not correct distortion correction. I fail to understand why.</p>
  <p><strong>UPDATE:</strong></p>
  <p>Based on below conversation the calibration footage needs to</p>
  <ul>
  <li>have a focus, similar to the reference scenario (in my case, objects are far away),</li>
  <li>the board should cover a minimum of say 20% of the image.</li>
  </ul>
  <p>Here the <a href="https://drive.google.com/file/d/1CUVMATE8xLs13yod01BC5myqjbLVcCoD/view?usp=sharing" rel="nofollow noreferrer">video</a> that I used (focus is fixed on objects being far away) to extract a sample of 50 frames to calculate the camera calibration:</p>
  <pre><code>SQUARE_SIZE = 0.012
  CHECKERBOARD_WIDTH = 22
  CHECKERBOARD_HEIGHT = 15
  
  objpoints = []
  imgpoints = []
  
  criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
  val = cv2.CALIB_CB_ADAPTIVE_THRESH + cv2.CALIB_CB_FAST_CHECK + cv2.CALIB_CB_NORMALIZE_IMAGE
  SAMPLE_SIZE = 50
  
  objp = np.zeros((CHECKERBOARD_HEIGHT * CHECKERBOARD_WIDTH, 3), np.float32)
  objp[:, :2] = np.mgrid[0:CHECKERBOARD_WIDTH, 0:CHECKERBOARD_HEIGHT.T.reshape(-1, 2)
  objp = objp * SQUARE_SIZE
  
  # for loop going over the sample of images
  img = cv2.imread('path to image')
  gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
  ret, corners = cv2.findChessboardCorners(gray, (CHECKERBOARD_WIDTH[i], CHECKERBOARD_HEIGHT[i]), val)
  if ret == True:
      objpoints.append(objp)
      # refining pixel coordinates for given 2d points.
      corners2 = cv2.cornerSubPix(gray, corners, (11,11), (-1,-1), criteria)
      imgpoints.append(corners2)
  
  # Calculate camera calibration
  img = cv2.imread('path to reference image')
  h,w = img.shape[:2]
  ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints,
                                                     imgpoints,
                                                     (w,h),
                                                     None,
                                                     None)                                          
  
  newcameramtx, roi = cv2.getOptimalNewCameraMatrix(mtx, dist, (w,h), 1, (w,h))
  
  # Undistort
  mapx,mapy = cv2.initUndistortRectifyMap(mtx, dist, None, newcameramtx, (w,h), 5)
  dst = cv2.remap(img,mapx,mapy,cv2.INTER_LINEAR)
  cv2.imwrite('path to output image', dst)
  </code></pre>
  <p>The <a href="https://drive.google.com/file/d/1pS_j_gc1vZvys-0hO2clWKirM4Ou2Oey/view?usp=sharing" rel="nofollow noreferrer">result</a> is unfortunately not correct as shown by the deviation of the red and white line at the top of the pitch.</p>