<s>[INST] Determining the transformation matrix for Jordan normal form. Let $$A=
\begin{pmatrix}
2 & 0 & -4 & -4 \\
0 &4&2&3\\
2&0&8&4\\
-1&0&-2&2\\
\end{pmatrix}$$
I want to find Jordan normal form of $A$ and the transformation matrix $P$. $(P^{-1}AP=J).$
I could find the Jordan form of $A$ is $$\begin{pmatrix}
4&0&0&0\\
0&4&1&0\\
0&0&4&1\\
0&0&0&4\\
\end{pmatrix}$$
But I have difficulty finding the transformation matrix.
I know the procedure for finding the transformation matrix.
Let $P=(p_1 \quad p_2 \quad p_3 \quad p_4)\ (p_i\in \mathbb R^4)$ and $AP=PJ$.
$$AP=(Ap_1 \quad Ap_2 \quad Ap_3 \quad Ap_4)$$
and
\begin{align}
PJ&=
(p_1 \quad p_2 \quad p_3 \quad p_4)
\begin{pmatrix}
4&0&0&0\\
0&4&1&0\\
0&0&4&1\\
0&0&0&4\\
\end{pmatrix}\\
&=(4p_1 \quad 4p_2 \quad p_2+4p_3 \quad p_3+4p_4)
\end{align}
Thus,
$$\begin{cases}
Ap_1=4p_1 \\
Ap_2=4p_2 \\
Ap_3=p_2+4p_3 \\
Ap_4=p_3+4p_4 \\
\end{cases}$$
$p_1$ and $p_2$ are the solutions of $Ax=4x.$ Solving $Ax=4x,$ I get
$$x=
s
\begin{pmatrix}
2 \\
0 \\
-3 \\
2 \\
\end{pmatrix}
+
t
\begin{pmatrix}
0 \\
1 \\
0\\
0\\
\end{pmatrix}.$$
I have to determine $p_1$ and $p_2$ so that they can be linearly independent, thus I decide $p_1=\begin{pmatrix}
2 \\
0 \\
-3 \\
2 \\
\end{pmatrix}$, $p_2=\begin{pmatrix}
0 \\
1 \\
0\\
0\\
\end{pmatrix}$.
But I don't know how I determine $p_3$.
$p_3$ is the solution of $Ay=p_2+4y$, i.e., $(A-4E)y=\begin{pmatrix}
0 \\
1 \\
0\\
0\\
\end{pmatrix}$
I found the solution of this is $y=\begin{pmatrix}
u-1\\
v \\
\frac{1}{2}-\frac{3}{2}u\\
u 
\end{pmatrix}$
I have to decide $u,v$ so that $p_1, p_2, p_3$ can be linely independent and
the forth equation $Az=p_3+4z$ can have solutions for $p_4$.
For example, if I decide $u=v=0,$ then I get $p_3=\begin{pmatrix}
-1 \\
0 \\
\frac{1}{2}\\
0
\end{pmatrix}$ and thus $p_1, p_2, p_3$ are linely independent but $Az=p_3+4z$ doesn't have solutions so I cannot find $p_4.$
How should I determine $u,v$ ?
 [/INST] You need to be careful with your choice of $p_2$; not just any vector will do. Note that
$$p_2 = (A - 4I)p_3 = (A - 4I)^2 p_4,$$
which means we need $p_2$ to be in the columnspace (or range, if you like) of $(A - 4I)^2$. It also needs to be an eigenvector, i.e. an element of $\operatorname{null} (A - 4I)$ so we ought to compute the intersection $$\operatorname{null}(A - 4I) \cap \operatorname{colspace}(A - 4I)^2.$$
First, we compute:
$$(A - 4I)^2=
\begin{pmatrix}
-2 & 0 & -4 & -4 \\
0 &0&2&3\\
2&0&4&4\\
-1&0&-2&-2
\end{pmatrix}^2 =
\begin{pmatrix}
0 & 0 & 0 & 0 \\
1 & 0 & 2 & 2 \\
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0
\end{pmatrix}$$
The columnspace is one-dimensional:
$$\operatorname{span} \begin{pmatrix} 0 \\ 1 \\ 0 \\ 0\end{pmatrix},$$
and as you no doubt determined, this is also an eigevector, i.e. it belongs to $\operatorname{null} (A - 4I)$. So, choosing
$$p_2 = \begin{pmatrix} 0 \\ 1 \\ 0 \\ 0\end{pmatrix}$$
will work.
Note that you had this right, but this was something of a fluke. You could have mixed up the order of $p_1$ or $p_2$, or worse, chosen a basis not containing a multiple of $(0, 1, 0, 0)^\top$, in which case neither choice would work. That is the problem you are running into when finding $p_4$ from $p_3$.
Next, to get $p_3$, note that, by similar reasoning, we need it to be both in $\operatorname{null}(A - 4I)^2$ and $\operatorname{colspace}(A - 4I)$. Indeed, choosing $p_3$ such that $(A - 4I)p_3 = p_2$ will guarantee the former. We just need to choose a solution satisfying the latter.
As you showed, possible solutions to the equation $(A - 4I)p_3 = p_2$ are:
$$p_3=\begin{pmatrix}
u-1\\
v \\
\frac{1}{2}-\frac{3}{2}u\\
u 
\end{pmatrix} = \begin{pmatrix} -1 \\ 0 \\ 1/2 \\ 0 \end{pmatrix} + u\begin{pmatrix} 1 \\ 0 \\ -3/2 \\ 1 \end{pmatrix} + v\begin{pmatrix} 0 \\ 1 \\ 0 \\ 0 \end{pmatrix}.$$
Which ones of these belong to the columnspace of $A - 4I$? Note that:
$$\operatorname{colspace}(A - 4I) = \operatorname{span} \left\{\begin{pmatrix} -2 \\ 0 \\ 2 \\ -1 \end{pmatrix}, \begin{pmatrix} 0 \\ 1 \\ 0 \\ 0 \end{pmatrix}\right\}.$$
Clearly, choosing $u = -1$, and $v$ being whatever we want (e.g. $v = 0$) would work, giving us a good choice of $p_3$:
$$p_3 = \begin{pmatrix} -2 \\ 0 \\ 2 \\ -1 \end{pmatrix}.$$
Finally, we just need to solve $(A - 4I)p_3 = p_4$. This will guarantee that $p_4 \in \operatorname{null}(A - 4I)^3$, and this is the only requirement (the requirement that $p_4 \in \operatorname{colspace}(A - 4I)^0 = \operatorname{colspace} I = \Bbb{R}^4$) is trivial!). This time, we should have a solution, if our construction has been correct so far. We then get the augmented matrix
$$\left(\begin{array}{cccc|c}
-2 & 0 & -4 & -4 & -2 \\
0 & 0 & 2 & 3 & 0 \\
2 & 0 & 4 & 4 & 2\\
-1 & 0 & -2 & -2 & -1
\end{array}\right).$$
This has a clear solution right off the bat:
$$p_4 = \begin{pmatrix} 1 \\ 0 \\ 0 \\ 0 \end{pmatrix}.$$
So, our constructed Jordan basis should be
$$\left(\begin{pmatrix} 2 \\ 0 \\ -3 \\ 2 \end{pmatrix}, \begin{pmatrix} 0 \\ 1 \\ 0 \\ 0 \end{pmatrix}, \begin{pmatrix} -2 \\ 0 \\ 2 \\ -1 \end{pmatrix}, \begin{pmatrix} 1 \\ 0 \\ 0 \\ 0 \end{pmatrix}\right).$$
You should verify that this basis does indeed produce the JNF given.
</s>