<s>[INST] Vectors that form basis in $\mathbb{R}^3$ and transition matrix 
Consider the following set of vectors in $\mathbb{R}^3:$ 
$u_0 = (1,2,0),~ u_1 = (1,2,1), ~u_2 = (2,3,0), ~u_3 = (4,6,1)$
Explain why each of the two subsets $B_0 = \left\{u_0, u_2,
 u_3\right\}$ and $B_1 = \left\{u_1, u_2, u_3\right\}$ forms a basis of
   $\mathbb{R}^3$. If we write $[\mathbf{x}]_0$ and $[\mathbf{x}]_1$ for the coordinates of the vector $\mathbf{x}$ in terms of these two basis, find the precise transition matrix which inter-relates these two sets of coordinates. If $\mathbf{x} = 4\mathbf{e}_1+4\mathbf{e}_3$, what are $\mathbf [\mathbf{x}]_0$ and $[\mathbf{x}]_1$?

Writing the vectors of the subset $B_0$ as the columns of a matrix we have:
$\mathbf{A}_0: = \begin{pmatrix}
1&2&4  \\ 
2&3&6  \\ 
0&0 &1  
\end{pmatrix} \to \begin{pmatrix}
1&2&4  \\ 
0&-1&-4  \\ 
0&0 &1  
\end{pmatrix} \implies \det(\mathbf{A}_0) =(1)(-1)(1) =  -1 $
As the columns of a $3 \times 3$ matrix whose determinant is non-zero, these vectors form a basis for $\mathbb{R}^3.$
Similarly, writing the vectors of the subset $B_1$ as the columns of a matrix we have:
$\mathbf{A}_1: = \begin{pmatrix}
1&1&2  \\ 
3&2&6  \\ 
0&1 &1 
\end{pmatrix} \to \begin{pmatrix}
1&1&2  \\ 
0&-1&0  \\ 
0&0 &1  
\end{pmatrix} \implies \det(\mathbf{A}_1) =(1)(-1)(1) =  -1 $
As the columns of a $3 \times 3$ matrix whose determinant is non-zero, these vectors form a basis for $\mathbb{R}^3.$
I don't know what it wants me to do beyond this point however. Could someone please explain what a transition matrix is?
 [/INST] The given vector $x$
$$ 
x = 4 e_1+ 4 e_3
$$
can be represented according the first or the second basis too.
\begin{align}
x 
&= x_1^{(0)} b_1^{(0)} + x_2^{(0)} b_2^{(0)} + x_3^{(0)} b_3^{(0)}
= (x_1^{(0)},x_2^{(0)},x_3^{(0)})^T = [x]_0 \\
&= x_1^{(1)} b_1^{(1)} + x_2^{(1)} b_2^{(1)} + x_3^{(1)} b_3^{(1)}
= (x_1^{(1)},x_2^{(1)},x_3^{(1)})^T = [x]_1
\end{align}
This gives a set $[x]_i$ of coordinates, a coordinate vector, for each basis. You are asked to give the matrix $T$, which would transform the first set into the second set of coordinates.
$$
[x]_1 = T [x]_0
$$
Solution:
From $B_0$ to standard basis $e_i$ we get via the matrix
$$
A_0 = 
(u_0, u_2, u_3) = 
\begin{pmatrix}
1 & 2 & 4 \\
2 & 3 & 6 \\
0 & 0 & 1
\end{pmatrix}
$$
From $B_1$ to standard basis we get via the matrix
$$
A_1 = 
(u_1, u_2, u_3) = 
\begin{pmatrix}
1 & 2 & 4 \\
2 & 3 & 6 \\
1 & 0 & 1
\end{pmatrix}
$$
From $B_0$ to $B_1$ we get by going from $B_0$ to standard basis via $A_0$ and from standard basis to $B_1$ via $A_1^{-1}$:
\begin{align}
T &= A_1^{-1} A_0 \\
&=
\begin{pmatrix}
1 & 0 & 0 \\
2 & 1 & 0 \\
-1 & 0 & 1
\end{pmatrix}
\end{align}
We have 
\begin{align}
[x]_0 &= A_0^{-1} [x] \\
&=
\left(
\begin{array}{rrr}
-3 & 2 & 0 \\
2 & -1 & -2 \\
0 & 0 & 1
\end{array}
\right)
\begin{pmatrix}
4 \\
0 \\
4
\end{pmatrix} \\
&=
\left(
\begin{array}{rrr}
-12 \\
0 \\
4
\end{array}
\right)
\end{align}
and
\begin{align}
[x]_1 &= A_1^{-1} [x] \\
&=
\left(
\begin{array}{rrr}
-3 &  2 & 0 \\
-4 &  3 & -2 \\
 3 & -2 & 1
\end{array}
\right)
\left(
\begin{array}{rrr}
4 \\
0 \\
4
\end{array}
\right)
\\
&=
\left(
\begin{array}{rrr}
-12 \\
-24 \\
16
\end{array}
\right)
\end{align}
</s>