<s>[INST] Short way for upper triangularization 
We are given a matrix $$A = 
\begin{bmatrix}
3 & 0 & 1 \\
-1 & 4 & -3 \\
-1 & 0 & 5 \\
\end{bmatrix}$$
  and we are asked to find a matrix $P$ such that $P^{-1}AP$ is upper triangular.

Here, we first find one eigenvalue as $\lambda= 4$. Then the matrix $$
A-4I = 
\begin{bmatrix}
-1 & 0 & 1 \\
-1 & 0 & -3 \\
-1 & 0 & 1 \\
\end{bmatrix}$$
has basis formed from $f_1 = (-1,-1,-1)^T$, $f_2 = (1,-3,1)^T$. We extend this to a basis of the whole space by adjoining $f_3 = (1,0,0)^T$, and so we have base-change matrix
$$ P = 
\begin{bmatrix}
-1 & 1 & 1 \\
-1 & -3 & 0 \\
-1 & 1 & 0 \\
\end{bmatrix}.$$
Then, by using some computational tools, we find that
$$ P^{-1}AP = 
\begin{bmatrix}
3 & 1 & 1 \\
-1 & 5 & 0 \\
0 & 0 & 4 \\
\end{bmatrix}$$
Now, we need to look at $$B = \begin{bmatrix}
3 & 1 \\
-1 & 5 \\
\end{bmatrix},$$ which has eigenvalue $\lambda = 4$. So we have $$B-4I = \begin{bmatrix}
-1 & 1 \\
-1 & 1 \\
\end{bmatrix}.$$
So basis for the image of this is $(1,1)^T$. We extend this to the basis $(1,1)^T$, $(1,0)^T$ of $\mathbb{R}^2$. Now, going back to $\mathbb{R}^3$, we have the matrix $$Q = \begin{bmatrix}
1 & 1 & 0 \\
1 & 0 & 0 \\
0 & 0 & 1 \\
\end{bmatrix}.$$
Then, by using calculation tools, we get
$$Q^{-1}P^{-1}APQ = \begin{bmatrix}
4 & -1 & 0 \\
0 & 4 & 1 \\
0 & 0 & 4 \\
\end{bmatrix},$$
which is in upper triangular form.
Now, what I wanted to ask is that is there a way to directly find the matrix $R = PQ$ such that $R^{-1}AR$ is upper triangular, without going through these steps?
 [/INST] Suppose we begin with the matrix $$A = 
\begin{bmatrix}
3 & 0 & 1 \\
-1 & 4 & -3 \\
-1 & 0 & 5 \\
\end{bmatrix}$$
and use "double Gaussian elimination". That is, whenever we multiply by an elementary row operation $E$ on the left, we must also multiply by the corresponding column operation $E^{-1}$ on the right. We begin by removing the $-1$ in the bottom left corner by doing a row operation of the form $R_3 \leftarrow R_3 + \lambda R_1$. We must then also perform the column operation $C_1 \leftarrow C_1 - \lambda C_3$. Since we want this to cancel the bottom-left cell, we see that $\lambda$ must satisfy
$$-1 + 3 \lambda - \lambda(5 + \lambda) = 0 \iff \lambda^2+2\lambda+1 = (\lambda+1)^2 = 0$$
So we take $\lambda = -1$, and apply those operations to get
$$A_1 = 
\begin{bmatrix}
4 & 0 & 1 \\
-4 & 4 & -3 \\
0 & 0 & 4 \\
\end{bmatrix}$$
Now, this is almost in upper-triangular form: if we perform the paired operations $(R_1 \leftrightarrow R_2, C_1 \leftrightarrow C_2)$ then we end up with
$$A_2 = 
\begin{bmatrix}
4 & -4 & -3 \\
0 & 4 & 1 \\
0 & 0 & 4 \\
\end{bmatrix}$$
which is upper-triangular. Let $P$ be the matrix tracking the column operations, so that
$$P = 
\begin{bmatrix}
0 & 1 & 0 \\
1 & 0 & 0 \\
0 & 1 & 1 \\
\end{bmatrix}$$
Then we have $P^{-1} A P = A_2$.
</s>