<s>[INST] Short way for upper triangularization 
We are given a matrix $$A = 
\begin{bmatrix}
3 & 0 & 1 \\
-1 & 4 & -3 \\
-1 & 0 & 5 \\
\end{bmatrix}$$
  and we are asked to find a matrix $P$ such that $P^{-1}AP$ is upper triangular.

Here, we first find one eigenvalue as $\lambda= 4$. Then the matrix $$
A-4I = 
\begin{bmatrix}
-1 & 0 & 1 \\
-1 & 0 & -3 \\
-1 & 0 & 1 \\
\end{bmatrix}$$
has basis formed from $f_1 = (-1,-1,-1)^T$, $f_2 = (1,-3,1)^T$. We extend this to a basis of the whole space by adjoining $f_3 = (1,0,0)^T$, and so we have base-change matrix
$$ P = 
\begin{bmatrix}
-1 & 1 & 1 \\
-1 & -3 & 0 \\
-1 & 1 & 0 \\
\end{bmatrix}.$$
Then, by using some computational tools, we find that
$$ P^{-1}AP = 
\begin{bmatrix}
3 & 1 & 1 \\
-1 & 5 & 0 \\
0 & 0 & 4 \\
\end{bmatrix}$$
Now, we need to look at $$B = \begin{bmatrix}
3 & 1 \\
-1 & 5 \\
\end{bmatrix},$$ which has eigenvalue $\lambda = 4$. So we have $$B-4I = \begin{bmatrix}
-1 & 1 \\
-1 & 1 \\
\end{bmatrix}.$$
So basis for the image of this is $(1,1)^T$. We extend this to the basis $(1,1)^T$, $(1,0)^T$ of $\mathbb{R}^2$. Now, going back to $\mathbb{R}^3$, we have the matrix $$Q = \begin{bmatrix}
1 & 1 & 0 \\
1 & 0 & 0 \\
0 & 0 & 1 \\
\end{bmatrix}.$$
Then, by using calculation tools, we get
$$Q^{-1}P^{-1}APQ = \begin{bmatrix}
4 & -1 & 0 \\
0 & 4 & 1 \\
0 & 0 & 4 \\
\end{bmatrix},$$
which is in upper triangular form.
Now, what I wanted to ask is that is there a way to directly find the matrix $R = PQ$ such that $R^{-1}AR$ is upper triangular, without going through these steps?
 [/INST] There is a misprint in the announcement. The $(1,3)$ element of $A$ should be $+1$ rather than $-1$ in order for $A$ to be consistent with the calculations that follow in your posted question.
So let $A=\begin{bmatrix} 3&0&1\\-1&4&-3\\-1&0&5 \end{bmatrix}$ and set
$N=A-4I= \begin{bmatrix} -1&0&1\\-1&0&-3\\-1&0&1 \end{bmatrix}$.
Then $N^2= \begin{bmatrix} 0&0&0\\4&0&-4\\0&0&0 \end{bmatrix}$ and $N^3=0$. So $N$ is nilpotent and $4$ is a triple eigenvalue of $A$ having a `Jordan chain' of length 3. A simple way to triangularize $A$ is then to calculate 3  vectors that generate this chain. For this you may note that $\ker N =\langle v_1\rangle $ is generated by  $v_1=\begin{bmatrix} 0 & 1 & 0 \end{bmatrix}^T$ and $\ker N^2 =\langle v_1,v_2\rangle$ with e.g. $v_2=\begin{bmatrix} -1 & 1 & -1\end{bmatrix}^T$. Finally, set e.g. $v_3=\begin{bmatrix} 2 & 0 & -2\end{bmatrix}^T$ so that $\ker(N^3)={\Bbb R}^3=\langle v_1,v_2,v_3\rangle$.
Now set $R= [v_1 \; v_2\; v_3] = \begin{bmatrix} 0&-1&2\\1&1&0\\0&-1&-2\end{bmatrix}$ and we get: $A R = R  \begin{bmatrix} 4 & 4 & 0\\ 0&4&4\\0&0&4\end{bmatrix}$ as wanted.
You may, of course, replace $v_2$ by $a v_2+b v_1$ with $a\neq 0$ which will change e.g. the $(1,2)$ element of the Jordan form (and similarly for $v_3$). My choice was just in order to get easy numbers and a nice end-result.
</s>