<s>[INST] Let $A=\begin{bmatrix} 1 & 2\\ 3& 4 \end{bmatrix}$ then det$(A^3-6A^2+5A+3I)=3$ 
Let $A=\begin{bmatrix}
1 & 2\\ 
 3& 4
\end{bmatrix}$ then det$(A^3-6A^2+5A+3I)=3$

det$(A^3-6A^2+5A+3I)=$det$((A^2-5A-2I)(A-I)+2A+I)= $det$(2A+I)=3$, Since a matrix satisfies its characteristic polynomial. Is this right? 
 [/INST] Here is another method. Assume that $\lambda_i\in\mathbb{C}$, $i=1,2$ are the two igenvalues of $A$. Let $f(x)\in \mathbb{C}[x]$ be the characteristic polynomial of $A$. Then $$f(x)=\det(A-xI)=x^2-5x-2.$$
The matrix $A$ can be diagonalized as 
$$A=P\begin{bmatrix} \lambda_1 & 0\\ 0& \lambda_2 \end{bmatrix}P^{-1},$$
where $P$ is an invertible matrix over $\mathbb{C}$. Let $g(x)=x^3-6x^2+5x+3$. We have $g(x)=f(x)(x-1)+2x+1$. It follows that 
$$\det(A^3-6A^2+5A+3I)=\det(g(A))=\det\left(g\left(\begin{bmatrix} \lambda_1 & 0\\ 0& \lambda_2 \end{bmatrix}\right)\right)=\det\left(\begin{bmatrix} g(\lambda_1) & 0\\ 0& g(\lambda_2) \end{bmatrix}\right)=g(\lambda_1)g(\lambda_2).$$
Note that $f(\lambda_i)=0$, $i=1,2$. Thus $g(\lambda_i)=f(\lambda_i)(\lambda_i-1)+2\lambda_i+1=2\lambda_i+1$. So
$$\det(A^3-6A^2+5A+3I)=(2\lambda_1+1)(2\lambda_2+1)=4\lambda_1\lambda_2+2(\lambda_1+\lambda_2)+1.$$
According to Vieta theorem, we have $$\lambda_1\lambda_2=-2, \lambda_1+\lambda_2=5.$$
Consequently, $$\det(A^3-6A^2+5A+3I)=4\times (-2)+2\times 5 +1=3.$$
</s>