<s>[INST] Three points: $A=(4,3), B=(2,5)$, and $C=(3,6)$. Find the angle between vector $AB$ and vector $AC$. Okay, so where am I going wrong? Here is step by step what I tried.
First I did this to find vectors:
$a=[A_x-B_x, A_y-B_y]$ so $a=[2,-2]$
$b=[A_x-C_x, A_y-C_y]$ so $b=[1,-3]$
Then I found the magnitudes of both vectors:
$$\|a\|= \sqrt{x^2 + y^2} \text{ so } \|a\|= 0$$
$$\|b\|=\sqrt{x^2 + y^2} \text{ so } \|b\| = \sqrt{10}$$
Then with this new information I tried the dot product:
$$\frac{a\cdot b}{\|a\|\cdot \|b\|} \text{ so } \frac{(2\cdot -2)+(1\cdot -3)}{0\cdot \sqrt{10}}$$
My calculator says "DIVIDE BY 0 Error"
So I assume I needed to try this:
$$\frac{(2 \cdot -2)+(1 \cdot -3)}{\sqrt{10}}= \frac{-7\sqrt{10}}{10} = -2.213594362$$
inverse of cos(-2.213594362)=DOMAIN ERROR
 [/INST] Here are some mistakes you've made:
Note that $(-2)^2=4$, which is not equal to $-4$. Therefore, $\|\mathbf{a}\|\neq0$:
$$\|\mathbf{a}\|=\sqrt{2^2+(-2)^2}=\sqrt{4+4}=\sqrt{8}$$

Also, note that your dot product is incorrect. For vectors $\mathbf{a}=\begin{pmatrix} a_1 \\ a_2 \end{pmatrix}$ and $\mathbf{b}=\begin{pmatrix} b_1 \\ b_2 \end{pmatrix}$, we should have:
$$\mathbf{a}\cdot \mathbf{b}=a_1b_1+a_2b_2$$
Here is what you seem to have done instead:
$$\mathbf{a}\cdot \mathbf{b}\neq a_1a_2+b_1b_2$$
Hence for your vectors, we should have:
$$\mathbf{a}\cdot \mathbf{b}=\begin{pmatrix} 2 \\ -2 \end{pmatrix}\cdot \begin{pmatrix} 1 \\ -3 \end{pmatrix}=2+6=8$$

The correction:
Hence, you should have:
$$\cos{\theta}=\frac{\mathbf{a}\cdot \mathbf{b}}{\|\mathbf{a}\|\|\mathbf{b}\|}=\frac{8}{\sqrt{8}\cdot \sqrt{10}}=\cdots$$
After simplifying, you should be able to find the value of $\theta$.
</s>