<s>[INST] Is the matrix $A^4+A^3-3A^2-3A$ invertible? 
Let $A$ be a real $5 \times 5$ matrix satisfying $A^3-4A^2+5A-2I=O$. Is the matrix $A^4+A^3-3A^2-3A$ invertible?

Consider the polynomial $t^3-4t^2+5t-2=(t-1)^2(t-2)$. The minimal polynomial of $A$ divides $(t-1)^2(t-2)$. Let $p(t)$ be the minimal polynomial of $A$.
If $p(x)=t-1$ or $t-2$, then $A$ is a scalar multiple of the identity and $A^4+A^3-3A^2-3A=-4I \text{ or }6I$, so $A^4+A^3-3A^2-3A$ is invertible.
If $p(t)=(t-1)^2(t-2)$ or $(t-1)(t-2)$, then $A$ is similar to a upper triangular matrix $J$ with diagonal entries $1,2$. Then $A^4+A^3-3A^2-3A=P(J^4+J^3-3J^2-3J)P^{-1}$ and $\det(A^4+A^3-3A^2-3A)=\det(J^4+J^3-3J^2-3J)$. Since $J^4+J^3-3J^2-3J$ is upper triangular and the diagonal entries do not vanish, $\det(J^4+J^3-3J^2-3J) \ne 0$.
If $p(t)=(t-1)^2$, then by the same reasoning, $A^4+A^3-3A^2-3A$ is invertible.
Is there more efficient way to solve this kind of problems or I have to discuss any possible situations every time?
 [/INST] The polynomials
$$ p(x) = x^3 -4 x^2 + 5x -2 = (x-1)^2(x-2) $$
$$ q(x) = x^4 + x^3 - 3 x^2 -3x = x(x+1)(x^2 - 3)$$
are coprime.
From Bézout's identitity, it follows that there exists two polynomials $u(x)$ and $v(x)$ such that
$$ u(x) p(x) + v(x) q(x) = 1$$
As $p(A) = 0$, it follows that $q(A)$ is invertible.
A pair of Bézout coefficients $u(x)$ and $v(x)$ can be computed by the extended Euclidean algorithm.
</s>