<s>[INST] How to solve for P in a similarity when matrices aren't diagonalizable When matrices $A,B$ are similar and they are both diagonalizable, as
$$
\left\{
\begin{array}{l}
P_1^{-1}AP_1=Λ\\
P_2^{-1}BP_2=Λ
\end{array}
\right.
\Longrightarrow (P_1P_2^{-1})^{-1}A(P_1P_2^{-1})=B$$
then we know when $P=P_1P_2^{-1}$, we can get $P^{-1}AP=B$.  We just solve for the eigenvectors of $A$ and $B$, and then we can easily get the $P_1$ and $P_2$.

My question is about how to proceed when $A$ and $B$ are similar yet not both diagonalizable. How can we solve for a matrix $P$ so that $B = P^{-1}A P$? 
I have two example in following:  
First
$A=\left(
\begin{array}{cc}
 0 & 1 \\
 0 & 0 \\
\end{array} 
\right)$ and $B=\left(
\begin{array}{cc}
 -\frac{12}{7} & -\frac{32}{7} \\
 \frac{9}{14} & \frac{12}{7} \\
\end{array}
\right)$.  How to get the $P$?
Actually I know the $P= ( \begin{smallmatrix} 2&10\\3&8 \end{smallmatrix} ) $
Second
$A=\left(
\begin{array}{ccc}
 27 & 48 & 81 \\
 -6 & 0 & 0 \\
 1 & 0 & 3 \\
\end{array}
\right)$ and $
B=\left(
\begin{array}{ccc}
 52 & 62 & 24 \\
 4 & 8 & 0 \\
 -80 & -\frac{197}{2} & -30 \\
\end{array}
\right)$
.How to solve the $P$?Actually the $P=\left(
\begin{smallmatrix}
 8 & 10 & 0 \\
 5 & 3 & 4 \\
 0 & 2 & 0 \\
\end{smallmatrix}
\right)$
 [/INST] Lets take this the other way
First of all we find one eigenvector the same way we normally would
$B\mathbf v_1 = \lambda \mathbf v_1$
In this case $\lambda = 0$ and $\mathbf v_1 = \begin{bmatrix}8\\-3\end{bmatrix}$
we have one vector in P, we need to find the other such that.
$BP = P\Lambda + P\begin{bmatrix}0&1\\0&0\end{bmatrix}\\
(B-\lambda I) P = P\begin{bmatrix}0&1\\0&0\end{bmatrix}\\
(B-\lambda I) \mathbf v_2 = \mathbf v_1$
Since $\lambda$ equals $0,$ you have made life easier on us... but not that much easier..
$B \mathbf v_2 = \mathbf v_1\\
B^2 \mathbf v_2 = B\mathbf v_1 = 0\\$
But $B^2 = 0$ which gives us a lot of latitude to find suitable $\mathbf v_2$
There is not a unique solution.  Any solution will suffice.
$-\frac {12}7  v_{2,1} -\frac{32}{7} v_{2,2} = 8\\
3 v_{2,1} + 8 v_{2,2} = -14\\
3\cdot6 + 8 \cdot -4 = -14$
$P = \begin{bmatrix}8& 6\\-3&-4\end{bmatrix}$ 
$BP = PA$ or $P^{-1} B P = A$
Update....
If you can't diagonalize the matrix, can you put it into "Jordan Normal" form?
$P^{-1}A P
=  \begin{bmatrix}
 \lambda_1 & 0 & 0 \\
 0 & \lambda_2 & 1 \\
 0 & 0 & \lambda_2 \\
\end{bmatrix}$
If you can put two matrices into the same Jordan Normal form, they are similar.
$A = \begin{bmatrix}
 27 & 48 & 81 \\
 -6 & 0 & 0 \\
 1 & 0 & 3 \\
\end{bmatrix}$
The characteristic equation is: $(\lambda - 6)(\lambda-12)^2 = 0$
We find the first two eigenvectors using the normal approach.
$A \begin{bmatrix} 3\\-3\\1\end{bmatrix} = \begin{bmatrix} 18\\-18\\6\end{bmatrix}$
$A \begin{bmatrix} 18\\-9\\2\end{bmatrix} = \begin{bmatrix} 216\\-108\\24\end{bmatrix}$
the $3^{rd}$ eigenvector....
$(A -12 I)^2 = \begin{bmatrix} 18&144&486\\-18&-144&-486\\6&48&162\end{bmatrix}$
Any vector in the kernel of the above is parallel to the second eigenvector.  Now you just need to scale it!
$\begin{bmatrix} 3\\3\\-1\end{bmatrix}$ is in the kernel.
$A \begin{bmatrix} 3\\3\\-1\end{bmatrix} = \begin{bmatrix}108\\-54\\12\end{bmatrix}= 6 \begin{bmatrix} 18\\-9\\2\end{bmatrix}$
$P = \begin{bmatrix} 3&18&\frac 12\\-3&9&\frac 12\\1&2&\frac 16\end{bmatrix}$
</s>