<s>[INST] Separating scaling, rotation and sheer coefficients correctly in this case This question follows-on from Correct the Fourier transform of a shear-distorted image? ($x_{new} = x + \alpha y$). I have a group of distorted $x, y$ points and have found a matrix $A$ that corrects their positions:
$$
    \begin{bmatrix}
    x' \\
    y' \\
    \end{bmatrix} = 
    \begin{bmatrix}
    a_{xx} & a_{xy}  \\
    a_{yx} & a_{yy} \\
    \end{bmatrix} 
    \begin{bmatrix}
    x \\
    y \\
    \end{bmatrix}
$$
I now want to break this up into three separate terms, one for pure scaling or magnification $M$, one for pure rotation $R$, and one for shear $S$. I think they will look like this:
$$M = \begin{bmatrix}
    m_x & 0\\
    0 & m_y \\
    \end{bmatrix}
$$
$$R = \begin{bmatrix}
    \cos \theta & -\sin \theta \\
    \sin \theta & \cos \theta \\
    \end{bmatrix} 
$$
$$S = \begin{bmatrix}
    1 & s_{xy}  \\
    s_{yx} & 1 \\
    \end{bmatrix} 
$$
I see there are answers to

*

*Separating Out Parts of a Matrix (Translation, Rotation, Scaling)

*Is it true that any matrix can be decomposed into product of rotation,reflection,shear,scaling and projection matrices?
and I recognize that those answers may apply to my problem, but they are beyond me; I need a simpler, more applied answer, or at least assistance getting there.
Q1: I can imagine setting $A$ equal to the product, but I'm worried if the order matters. Will this order work? Would any order work equally as well?
$$A = R \ S \ M $$
Q2: I now have five parameters instead of four. Since I've separated out rotation, should I now use only one shear, i.e. I have to choose only one of $s_xy$ or $s_yx$ and set the other to zero? Or just use $s_{xy} = s_{yx} = s$?
$$S = \begin{bmatrix}
    1 & s  \\
    s & 1 \\
    \end{bmatrix} 
$$
 [/INST] You have already observed the bad effect if you use $-s$ and $s$ in the off-diagonal terms of your "shear" matrix (making an antisymmetric matrix) -- the matrix doesn't just look like a rotation matrix, it actually is a combined rotation and scaling.
Changing the form of the matrix to a symmetric matrix does not help, because
$$
\begin{bmatrix} 1 & s  \\ s & 1 \end{bmatrix} =
\begin{bmatrix} \cos\left(-\frac\pi4\right) & -\sin\left(-\frac\pi4\right)  \\
            \sin\left(-\frac\pi4\right) & \cos\left(-\frac\pi4\right) \end{bmatrix}
\begin{bmatrix} 1 - s & 0  \\ 0 & 1 + s \end{bmatrix} 
\begin{bmatrix} \cos\left(\frac\pi4\right) & -\sin\left(\frac\pi4\right)  \\
             \sin\left(\frac\pi4\right) & \cos\left(\frac\pi4\right) \end{bmatrix}.
$$
That is, the effect of the symmetric matrix is what we get by rotating $\frac\pi4$ radians counterclockwise, dilating by a factor $1-s$ in the direction of the $x$ axis and $1+s$ in the direction of the $y$ axis, and rotating back by $\frac\pi4$ radians clockwise.
Equivalently, it is a dilation by $1-s$ in the direction of the vector
$[1,-1]^T$ and a dilation by $1+s$ in the direction of the vector $[1,1]^T.$
As a more concrete way of looking at it, just observe what the transformation does to the unit square: it distorts it into a rhombus for any value of $s.$ This is just stretching, compressing, or even reflecting it along its two diagonals; there is no shear at all.
On the other hand, if you look at the effect on the unit square of a transformation by the matrix
$$ \begin{bmatrix} 1 & s  \\ 0 & 1 \end{bmatrix}, $$
it is clearly a shear transformation: the vertices $(0,0)$ and $(1,0)$ are unaffected, while the vertices $(0,1)$ and $(1,1)$ are both moved $s$ units to the right.

Now let's see how we can decompose an arbitrary $2\times2$ matrix into matrices for rotation, scaling, and shear. We start with
$$ A = \begin{bmatrix} a & b  \\ c & d \end{bmatrix}. $$
(Since there are only four entries, I find it simple enough to keep track of them by name rather than by subscripts, and the formulas will be less "busy" this way.)
Note what happens if we apply a rotation of $\theta$ radians to this matrix:
$$
\begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}
 \begin{bmatrix} a & b  \\ c & d \end{bmatrix} =
\begin{bmatrix} a \cos\theta - c \sin\theta & b \cos\theta - d \sin\theta \\
c \cos\theta + a \sin\theta & d \cos\theta + b \sin\theta \end{bmatrix}.
$$
Let $\theta = -\operatorname{atan2}(c, a)$ where $\operatorname{atan2}$ is the two-parameter arc tangent function
(see this answer for a definition).
Then $\sin\theta = -\frac{c}{\sqrt{a^2+c^2}}$ and
$\cos\theta = \frac{a}{\sqrt{a^2+c^2}}$,
so $c \cos\theta + a \sin\theta = 0.$
Define a rotation matrix $Q$ (not $R$; let's reserve the name $R$ for later) by
$$
Q = \begin{bmatrix}\cos\theta & -\sin\theta \\ \sin\theta & \cos\theta\end{bmatrix}.
$$
Then
$$
Q A = 
\begin{bmatrix} a \cos\theta - c \sin\theta & b \cos\theta - d \sin\theta \\
            0 & d \cos\theta + b \sin\theta \end{bmatrix}.
$$
Suppose $a \cos\theta - c \sin\theta \neq 0$
and $d \cos\theta + b \sin\theta \neq 0.$
(The alternative is that the matrix $QA$ is a projection and therefore so is the original matrix $A$.)
Define a scaling matrix $N$ (saving the name $M$ for later) by
$$
N = \begin{bmatrix} \frac{1}{a \cos\theta - c \sin\theta} & 0 \\
                    0 & \frac{1}{d \cos\theta + b \sin\theta} \end{bmatrix}.
$$
Then
$$
NQA =
 \begin{bmatrix} 1 & \frac{b\cos\theta - d\sin\theta}{a\cos\theta - c\sin\theta}\\
                 0 & 1 \end{bmatrix}.
$$
This is a shear. So let $NQA = S$ and multiply both sides of the equation by
$Q^{-1}N^{-1}$; then
$$
A = Q^{-1} N^{-1} S = RMS
$$
where
\begin{align}
R &= Q^{-1} =
\begin{bmatrix} \cos\theta & \sin\theta \\ -\sin\theta & \cos\theta \end{bmatrix} =
\begin{bmatrix} \frac{a}{\sqrt{a^2 + c^2}} & \frac{-c}{\sqrt{a^2 + c^2}} \\
   \frac{c}{\sqrt{a^2 + c^2}} & \frac{a}{\sqrt{a^2 + c^2}} \end{bmatrix}, \\
M &= N^{-1} = \begin{bmatrix} a \cos\theta - c \sin\theta & 0 \\
                    0 & d \cos\theta + b \sin\theta \end{bmatrix}
= \begin{bmatrix} \sqrt{a^2 + c^2} & 0 \\
                    0 & \frac{ad-bc}{\sqrt{a^2 + c^2}} \end{bmatrix}, \\
S &=
\begin{bmatrix} 1 &
        \frac{b\cos\theta - d\sin\theta}{a\cos\theta - c\sin\theta}\\
                 0 & 1 \end{bmatrix} =
\begin{bmatrix} 1 & \frac{ab + cd}{a^2 + c^2} \\ 0 & 1 \end{bmatrix}.
\end{align}
This is good for every matrix $A$ except the case where $a = c = 0,$
which is a projection and therefore presumably not what you would use to "correct" the positions of points.
For completeness, however, if $a = c = 0$ and $b^2 + d^2 \neq 0$ we can write
$$
A = \begin{bmatrix} 0 & b \\ 0 & d \end{bmatrix} =
\begin{bmatrix} \frac{d}{\sqrt{b^2 + d^2}} & \frac{b}{\sqrt{b^2 + d^2}} \\
   \frac{-b}{\sqrt{b^2 + d^2}} & \frac{d}{\sqrt{b^2 + d^2}} \end{bmatrix}
\begin{bmatrix} 0 & 0 \\ 0 & \sqrt{b^2 + d^2} \end{bmatrix}
\begin{bmatrix} 1 & s  \\ 0 & 1 \end{bmatrix}
$$
where $s$ is whatever you like, that is, $A=RMS$ where $R$ is a rotation, $M$ is a scaling matrix (where $m_x = 0$) and $S$ is a shear.
And in the final case, $a = b = c = d,$ obviously you can just set $M$ to the zero matrix (that is, $m_x = m_y = 0$) and use any shear and rotation you like.
So it is indeed always possible to decompose $A$ into a product of the form $RMS,$
even in the cases you probably don't care about.

This suggests that it is possible to use another sequence such as $RSM,$
and indeed that particular sequence is possible.
In the $RMS$ derivation, the next step after zeroing out the lower left corner (via rotation) was to scale so that the diagonals are $1$
(which immediately gives you a shear matrix);
for $RSM,$ let's use a shear to produce a diagonal matrix.
That is, constructing $Q$ as before we have
$$
Q A =
\begin{bmatrix} a \cos\theta - c \sin\theta & b \cos\theta - d \sin\theta \\
            0 & d \cos\theta + b \sin\theta \end{bmatrix}.
$$
A shear parallel to the $x$ axis will have no affect on the first column of this matrix, but the effect on the second column can be seen in
$$
\begin{bmatrix} 1 & t  \\ 0 & 1 \end{bmatrix}
\begin{bmatrix} b \cos\theta - d \sin\theta \\
                d \cos\theta + b \sin\theta \end{bmatrix} =
\begin{bmatrix} t(d\cos\theta + b\sin\theta) + b\cos\theta - d\sin\theta \\
                d \cos\theta + b \sin\theta \end{bmatrix}.
$$
We need
$t(d\cos\theta + b\sin\theta) + b\cos\theta - d\sin\theta = 0,$
which we get by setting
$t = \frac{d\sin\theta - b\cos\theta}{d\cos\theta + b\sin\theta}.$
but the derivation might be a little more difficult to find (I have not tried)
and we should expect the resulting formulas to be quite different.
Then
$$
TQA =
\begin{bmatrix} a \cos\theta - c \sin\theta & 0 \\
            0 & d \cos\theta + b \sin\theta \end{bmatrix}
$$
where $T$ is the shear matrix
$$
T =
\begin{bmatrix} 1 &
        \frac{d\sin\theta - b\cos\theta}{d\cos\theta + b\sin\theta}\\
          0 & 1 \end{bmatrix} =
\begin{bmatrix} 1 & -\frac{ab + cd}{ad - bc} \\ 0 & 1 \end{bmatrix}
$$
We see that $M = TQA$ is a scaling matrix, so we have
$A = RSM$ where
\begin{align}
R &= Q^{-1} =
\begin{bmatrix} \cos\theta & \sin\theta \\ -\sin\theta & \cos\theta \end{bmatrix} =
\begin{bmatrix} \frac{a}{\sqrt{a^2 + c^2}} & \frac{-c}{\sqrt{a^2 + c^2}} \\
   \frac{c}{\sqrt{a^2 + c^2}} & \frac{a}{\sqrt{a^2 + c^2}} \end{bmatrix}, \\
S &= T^{-1} =
\begin{bmatrix} 1 & \frac{ab + cd}{ad - bc} \\ 0 & 1 \end{bmatrix}, \\
M &=
\begin{bmatrix} a \cos\theta - c \sin\theta & 0 \\
            0 & d \cos\theta + b \sin\theta \end{bmatrix} =
\begin{bmatrix} \sqrt{a^2 + c^2} & 0 \\
                    0 & \frac{ad - bc}{\sqrt{a^2 + c^2}} \end{bmatrix}.
\end{align}
That is, the $RSM$ decomposition is just like the $RMS$ decomposition except for the shear factor.
</s>