<s>[INST] How to find the determinant of this $5 \times 5$ matrix? How can I find the determinant of this matrix?

I know in matrix $3 \times 3$ 
$$A= 1(5\cdot 9-8\cdot 6)-2 (4\cdot 9-7\cdot 6)+3(4\cdot 8-7\cdot 5) $$
but how to work with a $5\times 5$ matrix?
 [/INST] The Laplace expansion of the determinant can be done using any row or column of a square matrix. In each row, we multiply the $a_{ij}$ component of the matrix with the determinant of the matrix formed by deleting the $i$th row and $j$th column of our original matrix. This new matrix is called the $ij$-minor of $A$ and is commonly given the symbol $M_{ij}$. This term is then given a sign that depends on the sum of the two indices of the associated component: $(-1)^{i+j}$. So, in total, each term is 
$$(-1)^{i+j}a_{ij}M_{ij}$$
We add all of these terms for an entire row, or an entire column. No matter which we choose, we will get the same number, called the determinant of the matrix.
To decrease the amount of terms involved, we usually try to expand along a row or column with the most zeroes. In your matrix, the third row has the most zeroes, with only one non-zero component: $a_{33} = 4$.
So the only non-zero term in the Laplace expansion of the determinant of your matrix is
$$(-1)^{3+3}a_{33}\left|\begin{array}{cccc}1&2&4&1\\0&-1&4&2\\-3&-6&-12&4\\0&0&1&1\end{array}\right| = 4\left|\begin{array}{cccc}1&2&4&1\\0&-1&4&2\\-3&-6&-12&4\\0&0&1&1\end{array}\right|$$
Now we do the same exercise for the 4x4 matrix. Let's call it $B$. The 4th row has the most zeroes, so we choose that for our expansion. The only non-zero components are $b_{43} = 1$ and $b_{44} = 1$. So our expansion is:
$$4\left((-1)^{4+3}b_{43}\left|\begin{array}{ccc}1&2&1\\0&-1&2\\-3&-6&4\end{array}\right| + (-1)^{4+4}b_{44}\left|\begin{array}{ccc}1&2&4\\0&-1&4\\-3&-6&-12\end{array}\right|\right) = 4\left(-\left|\begin{array}{ccc}1&2&1\\0&-1&2\\-3&-6&4\end{array}\right| + \left|\begin{array}{ccc}1&2&4\\0&-1&4\\-3&-6&-12\end{array}\right|\right)$$ 
The first column is fine for both of these 3x3 determinant expansions. I'll omit the cofactor symbolism since you already know how to expand 3x3 determinants:
$$4\left(-\left(\left|\begin{array}{cc}-1&2\\-6&4\end{array}\right| - 3\left|\begin{array}{cc}2&1\\-1&2\end{array}\right|\right) + \left(\left|\begin{array}{cc}-1&4\\-6&-12\end{array}\right| - 3\left|\begin{array}{cc}2&4\\-1&4\end{array}\right|\right)\right)$$
And finally, we expand the 2x2 determinants:
$$4(-((-4-(-12)) - 3(4 - (-1))) + ((12-(-24)) - 3(8-(-4)))) = 28$$
</s>