# Transform points from pitch to texture and vice-versa
### Matrix P - Projection
The P matrix is a 3x4 matrix that given a 3D point in the world reference frame, it is projected into the 2D image in **texture coordinate** reference frame, i.e:

\begin{align}
\mathbf{pt_{world}} = (x, y, z, 1) \\
\mathbf{pt_{pitch}} = (a, b, c) = \mathbf{P_{texture}} * \mathbf{pt_{world}} \\
\mathbf{pt_{texture}} = (i, j) = (a/c, b/c)
\end{align}

**Texture coordinate range** is [0.0, 1.0]



### Matrix H - Homography
The H matrix is a 3x3 Matrix that transforms points from one plane to another. In our case from pitch to texture and from texture to pitch which is the inverse of the first (pitch to texture homography). **Pitch is the world plane where z=0.** Hence, get matrix **H*pith2texture*** is as simple as discard 3th matrix P column:




### How to convert from pitch (world) to texture coordinate system
\begin{align}
\mathbf{pt_{texture}} = \mathbf{H_{pitch2texture}} * \mathbf{pt_{pitch}}
\end{align}

### How to convert from texture to pitch (world) coordinate system
\begin{align}
\mathbf{pt_{pitch}} = \mathbf{H_{texture2pitch}} * \mathbf{pt_{texture}}
\end{align}

Where,

\begin{align}
\mathbf{H_{texture2pitch}} = \mathbf{H_{pitch2texture}^{-1}}
\end{align}

### How to convert from texture to video image coordinate system

\begin{equation*}
\mathbf{P_{image}} = \begin{vmatrix}
Width & 0 & 0 \\ 0 & Height & 0 \\ 0 & 0 & 1
\end{vmatrix} * \mathbf{P_{texture}}
\end{equation*}

Where **`Width`** and **`Height`** refer to the video frame size. 


```python
# import what we need
import numpy as np
import math
```


```python
def rodrigues(r):
    '''
    Rodrigues formula
    :param r: 1x3 array of rotations about x, y, and z
    :return: the 3x3 rotation matrix
    '''
    def S(n):
        Sn = np.array([
            [0.0, -n[2], n[1]],
            [n[2], 0.0, -n[0]],
            [-n[1], n[0], 0]])
        return Sn
    theta = np.linalg.norm(r)
    if theta > 1e-30:
        n = r/theta
        Sn = S(n)
        R = np.eye(3) + np.sin(theta) * Sn + (1.0 - np.cos(theta)) * np.dot(Sn, Sn)
    else:
        Sr = S(r)
        theta2 = theta ** 2.0
        R = np.eye(3) + (1.0 - theta2 / 6.0)*Sr + (0.5 - theta2 / 24.0) * np.dot(Sr, Sr)
    return np.mat(R)
```


```python
def world_to_texture(pw, P):
    """
    Projects a world point to texture projection plane
    :param pw: world point (x, y, z)
    :param P: projection matrix P
    :return: texture point (i, j)
    """
    pw_h = np.append(pw, 1.0)
    pp = P.dot(pw_h)
    return np.array([pp[0]/pp[2], pp[1]/pp[2]])
```


```python
def homography_pitch_to_texture(P):
    """
    Returns the homografy to transform points from the pitch (world Z=0) to texture image
    :param matrixP: Full matrix P
    :return: homography_pitch_to_texture matrix
    """
    # delete the 3th column, the z component
    return np.delete(P, 2, axis=1) 
```


```python
def pitch_to_texture(pitch, P):
    """
    Project a point from pitch plane to texture plane
    :param pitch: pitch point (x, y)
    :param P: projection matrix P
    :return: texture point (i, j)
    """
    pitch_h = np.append(pitch, 1.0)
    H = homography_pitch_to_texture(P)
    transformed = H.dot(pitch_h)
    #print(projected)
    return np.array([transformed[0]/transformed[2], transformed[1]/transformed[2]])
    
```


```python
def texture_to_pitch(texture, P):
    """
    Project a point from texture plane to pitch plane
    :param texture: texture point (i, j)
    :param P: projection matrix P
    :return: pitch point (x, y)
    """
    texture_h = np.append(texture, 1.0)
    H = homography_pitch_to_texture(P)
    Hinv = np.linalg.inv(H)
    transformed = Hinv.dot(texture_h)
    return np.array([transformed[0]/transformed[2], transformed[1]/transformed[2]])
```


```python
def texture_to_image(texture, width, height):
    """
    Converts a texture coordinate to image coordinate system
    """
    return  np.array([int(float(width)*texture[0]), int(float(height)*texture[1])])

def image_to_texture(image, width, height):
    """
    Converts a image coordinate to texture coordinate system
    """
    return  np.array([float(image[0])/float(width), float(image[1])/float(height)])
```

# Projection matrix P from an individual camera
The funciton `camera_full_projection_matrix` is only valid for individual cameras. 

## How to get camera parameters: *aspect_ratio, zoom, skew, pan, tilt, roll, Tx, Ty, Tz* 
You can find camera parameters **`aspect_ratio, zoom, skew, pan, tilt, roll, Tx, Ty, Tz`** in `C:\AutomaticTV\data\cameras\{id}.xml`. In XML `<modelcalibration>` tag, for instance:

```
<modelsCalibration>
    <modelCalibration computed="true" sportFieldName="Football11">
      <Zoom>1.2124214</Zoom>
      <AspectRatio>1.7777778</AspectRatio>
      <Skew>0</Skew>
      <Pan>-28.826538</Pan>
      <Tilt>110.37401</Tilt>
      <Roll>-10.530287</Roll>
      <Tx>34.07756</Tx>
      <Ty>-3.4855517</Ty>
      <Tz>74.498503</Tz>
    </modelCalibration>
  </modelsCalibration>
```


```python
def camera_full_projection_matrix(aspect_ratio, zoom, skew, pan, tilt, roll, Tx, Ty, Tz):
    """
    Creates projection matrix P from camera model parameters
    :param aspect_ratio: camera aspect ratio = image width / image height
    :param zoom: camera focal
    :param skew:
    :param pan:
    :param tilt:
    :param roll:
    :param Tx:
    :param Ty:
    :param Tz:
    :return: the projection Matrix P
    """
    K = np.array([[zoom, skew, 0.5], [0.0, zoom * aspect_ratio, 0.5], [0.0, 0.0, 1.0]])

    # Rotation matrix
    Rpan = rodrigues(np.array([0.0, 0.0, pan*math.pi/180.0]))
    Rtilt = rodrigues(np.array([tilt*math.pi/180.0, 0.0, 0.0]))
    Rroll = rodrigues(np.array([0.0, 0.0, roll*math.pi/180.0]))
    Mrot = Rroll * Rtilt * Rpan

    # Translation vector
    t = np.array([Tx, Ty, Tz])
    KR = K * Mrot
    Kt = np.dot(K, t)

    # Projection Martix P
    P = np.zeros((3, 4))
    P[:, 0] = KR[:, 0].T
    P[:, 1] = KR[:, 1].T
    P[:, 2] = KR[:, 2].T
    P[:, 3] = Kt
    return P
```

### Individual camera matrix P Testing


```python
# test
# test matrixP_from_camera_model
camera_model = {
    "aspect_ratio": 5.333333333,
    "zoom": 0.45361389,
    "skew": 0.0,
    "pan": -2.0953152,
    "tilt": 108.76381,
    "roll": 0.0,
    "Tx": 0.48063888,
    "Ty": -0.30635475,
    "Tz": 87.349004}
P = camera_full_projection_matrix(**camera_model)
P_expected = np.array([[0.436, 0.4897, -0.1608, 43.893],
                      [0.01114, -0.3046, -2.4515, 42.933],
                      [-0.03462, 0.9462, -0.3217, 87.349]])
assert(np.allclose(P, P_expected, rtol=1e-3))

# ML
camera_model = {
    "aspect_ratio": 1.7777778,
    "zoom": 1.2124214,
    "skew": 0.0,
    "pan": -28.826538,
    "tilt": 110.37401,
    "roll": -10.530287,
    "Tx": 34.07756,
    "Ty": -3.4855517,
    "Tz": 74.498503}
P = camera_full_projection_matrix(**camera_model)
print(P)
P_expected = np.array([[0.8555, 0.9178, -0.3818, 78.566],
                      [-0.2154, -0.4256, -2.1606, 29.736],
                      [-0.452, 0.8213, -0.3481, 74.499]])
assert(np.allclose(P, P_expected, rtol=1e-3))

print()
# project world point
x = -10.0
y = 3.0
world = np.array([x, y, 0.0])
print("world: {}".format(world))
texture = world_to_texture(world, P)
print("texture: {}".format(texture))
"""
if (texture > 1.0).any() or (texture < 0.0).any():
    print("point is out the texture limits")
else:
    print("point is in the texture limits")
"""

print()
# transform point from pitch to texture plane
pitch = np.array([x, y])
print("pitch: {}".format(pitch))
texture = pitch_to_texture(pitch, P)
print("texture: {}".format(texture))

print()
# transform point from texture to pitch
print("texture: {}".format(texture))
pitch = texture_to_pitch(texture, P)
print("pitch: {}".format(pitch))


```

    [[  0.85549003   0.91779104  -0.38178799  78.5656145 ]
     [ -0.21537938  -0.42563356  -2.16061687  29.73643812]
     [ -0.45199561   0.82127568  -0.34814685  74.498503  ]]
    
    world: [-10.   3.   0.]
    texture: [ 0.89300498  0.37570535]
    
    pitch: [-10.   3.]
    texture: [ 0.89300498  0.37570535]
    
    texture: [ 0.89300498  0.37570535]
    pitch: [-10.   3.]


## Matrix P for Panorama

## How to get Panorama camera parameters: *src_width, src_height, zoom, skew, pan, tilt, roll, Tx, Ty, Tz* 
You can find camera parameters **`zoom, skew, pan, tilt, roll, Tx, Ty, Tz`** in `C:\AutomaticTV\data\virtual_cameras\{id}.xml`. In XML `<CameraModel name="PANORAMA">` tag, for instance:

```
    <CameraModel name="PANORAMA">
      <Width>5760</Width>
      <Height>1080</Height>
      <Zoom>0.45361389</Zoom>
      <AspectRatio>5.3333333</AspectRatio>
      <Skew>0</Skew>
      <Pan>-2.0953152</Pan>
      <Tilt>108.76381</Tilt>
      <Roll>10</Roll>
      <Tx>0.48063888</Tx>
      <Ty>-0.30635475</Ty>
      <Tz>87.349004</Tz>
    </CameraModel>
```
* **src_width** is `5760`
* **src_height** is `1080`

**NOTE: Do not use this `AspectRatio`.**

## How to get Panorama camera *offset_x* and *offset_y*
You can find **`offset_x`** and **`offset_y`** in `C:\AutomaticTV\data\virtual_cameras\{id}.xml` file tag `<PanoramaOffsetX>` and `<PanoramaOffsetY>`
```
    <PanoramaOffsetX>0</PanoramaOffsetX>
    <PanoramaOffsetY>0</PanoramaOffsetY>
```

## How to get Panorama camera *src_width* and *src_height*
It can be found in `C:\AutomaticTV\data\productions\{id}.xml`. In XML `<panorama><realization>` tag, for instance:`
```
    <width>3840</width>
    <height>1080</height>
```

## How to get Panorama camera *aspect_ratio, state_x, state_y, state_zoom*
It can be found in `C:\AutomaticTV\data\productions\{id}.xml`. In XML `<panorama><realization><operators><operator name="panorama"><currentState>` tag, for instance:

```
          <currentState>
            <elem>0.500000</elem>
            <elem>0.500000</elem>
            <elem>1.000000</elem>
            <elem>0.453614</elem>
            <elem>3.555556</elem>
            <elem>0.000000</elem>
            <elem>-2.095315</elem>
            <elem>108.763810</elem>
            <elem>0.000000</elem>
            <elem>0.480639</elem>
            <elem>-0.306355</elem>
            <elem>87.349004</elem>
          </currentState>
```
where:
* ***aspect_ratio*** is the 4th elem `3.555556`
* ***state_x*** is the 1st elem `0.500000`
* ***state_y*** is the 2nd elem `0.500000`
* ***state_zoom*** is the 3rd elem `1.000000`


```python
def camera_full_projection_matrix_with_crop(aspect_ratio, zoom, skew, pan, tilt, roll, Tx, Ty, Tz, crop_tx, crop_ty, crop_zoom):
    """
    Creates projection matrix P from camera model parameters
    :param aspect_ratio: camera aspect ratio = dst_width / dst_height
    :param zoom: camera focal
    :param skew:
    :param pan:
    :param tilt:
    :param roll:
    :param Tx:
    :param Ty:
    :param Tz:
    :param crop_tx:
    :param crop_ty:
    :param crop_zoom:
    :return: the projection Matrix P
    """
    K = np.array([[crop_zoom * zoom, crop_zoom * skew, 0.5 + crop_tx], 
                  [0.0, crop_zoom * zoom * aspect_ratio, 0.5 + crop_ty],
                  [0.0, 0.0, 1.0]])

    # Rotation matrix
    Rpan = rodrigues(np.array([0.0, 0.0, pan*math.pi/180.0]))
    Rtilt = rodrigues(np.array([tilt*math.pi/180.0, 0.0, 0.0]))
    Rroll = rodrigues(np.array([0.0, 0.0, roll*math.pi/180.0]))
    Mrot = Rroll * Rtilt * Rpan

    # Translation vector
    t = np.array([Tx, Ty, Tz])
    KR = K * Mrot
    Kt = np.dot(K, t)

    # Projection Martix P
    P = np.zeros((3, 4))
    P[:, 0] = KR[:, 0].T
    P[:, 1] = KR[:, 1].T
    P[:, 2] = KR[:, 2].T
    P[:, 3] = Kt
    return P

def get_crop_params(src_width, src_height, offset_x, offset_y, dst_width, dst_height, state_x, state_y, state_zoom):
    """
    Computes crop parameters taking in account source and destination dimensions
    : return crop_tx: 
    : return crop_ty:
    : return crop_zoom:
    """
    width_ratio_inv = float(src_width) / float(dst_width)
    height_ratio_inv = float(src_height) / float(dst_height)
    crop_tx = (0.5 - state_x + offset_y) * (state_zoom * width_ratio_inv)
    crop_ty = (0.5 - state_x + offset_y) * (state_zoom * height_ratio_inv)
    crop_zoom = state_zoom * max(width_ratio_inv, height_ratio_inv)
    
    return crop_tx, crop_ty, crop_zoom
```

### Panorama matrix P Testing


```python
camera_model = {
    "aspect_ratio": 3.5555555820465088,
    "zoom": 0.45361389,
    "skew": 0.0,
    "pan": -2.0953152,
    "tilt": 108.76381,
    "roll": 10.0,
    "Tx": 0.48063888,
    "Ty": -0.30635475,
    "Tz": 87.349004}

crop_params = {
    "src_width": 5760,
    "src_height": 1080,
    "offset_x": 0.0,
    "offset_y": 0.0,
    "dst_width": 3840,
    "dst_height": 1080,
    "state_x": 0.5,
    "state_y": 0.5,
    "state_zoom": 1.0}

# compute crop from params
crop_tx, crop_ty, crop_zoom = get_crop_params(**crop_params)
print("crop: {}, {}, {}".format(crop_tx, crop_ty, crop_zoom))

# compute matrix P
P = camera_full_projection_matrix_with_crop(**camera_model, crop_tx=crop_tx, crop_ty=crop_ty, crop_zoom=crop_zoom)
print(P)
P_expected = np.array([[0.6509, 0.53559, -0.04896, 44.0015],
                      [0.4305, -0.2774, -2.4167, 42.933],
                      [-0.034618, 0.946219, -0.321667, 87.3490]])
assert(np.allclose(P, P_expected, rtol=1e-3))


```

    crop: 0.0, 0.0, 1.5
    [[  6.50936689e-01   5.35590234e-01  -4.89595755e-02   4.40015387e+01]
     [  4.30532613e-01  -2.77397706e-01  -2.41672906e+00   4.29333459e+01]
     [ -3.46188241e-02   9.46219547e-01  -3.21667695e-01   8.73490040e+01]]


## How to transform points from an image A to an image B



### How to get frames from video
```
ffmpeg -i "CleanFeed.mp4" "frames/out-%03d.jpg"
```


```python

```

d0ac032193d305d7e1e448ddded39adc54b4b250
ipynb
Jupyter Notebook
MatrixP.ipynb
marceljanerfont/MatrixP
94fc881e3696f79a1ee6d8ec78535810c26f5b10
MIT
MatrixP.ipynb
marceljanerfont/MatrixP
94fc881e3696f79a1ee6d8ec78535810c26f5b10
MIT
MatrixP.ipynb
marceljanerfont/MatrixP
94fc881e3696f79a1ee6d8ec78535810c26f5b10
MIT
Qwen/Qwen-72B
1. YES
2. YES