<s>[INST] Overdetermined system with parameters $$\begin{cases}
ax & +y &=2a\\
x &+by  &= b \\
ax &+ (5-b^2)y &= 1 \tag{A,B,C (in order)}
\end{cases}$$
Where $(x,y)$ are variables and $a,b$ are constants. 


*

*What combination of $a$ and $b$ yields infinitely many solutions to our system?



By subtracting $A$ from $C$ we get:
$\begin{cases}
ax & +y &=2a\\
x &+by  &= b \\
&(5-b^2)y - y &= 1 - 2a \tag{A',B',C' (in order)}
\end{cases}$
$\iff\begin{cases}
ax & +y &=2a\\
x &+by  &= b \\
&y(4-b^2) &= 1 - 2a \tag{A'',B'',C'' (in order)}
\end{cases}$
$\iff\begin{cases}
ax & +y &=2a\\
x &+by  &= b \\
&y &= \frac{1 - 2a}{4-b^2} \tag{A''',B''',C''' (in order)}
\end{cases}$
By subtracting $C'''$ from $A'''$ we get:
$$\begin{cases}
ax &&=2a - \frac{1 - 2a}{4-b^2}\\
x &+by  &= b \\
&y &= \frac{1 - 2a}{4-b^2}
\end{cases}$$
$$\iff\begin{cases}
x &&=2 - \frac{1 - 2a}{4a-ab^2}\\
x &+by  &= b \\
&y &= \frac{1 - 2a}{(4-b^2)}
\end{cases}$$
Substituting $x$ and $y$ into $B$ gets us the following expression:
$$\left(2-\frac{1-2a}{4a-ab^2}\right) + b\cdot\left(\frac{1-2a}{4-b^2}\right) = b$$
Did I do anything right? Is this leading to the solution? Is there any more examples of this particular problem anywhere? Can't seem to find any.
NB! I ONLY SEEK THE VALUES OF $a$ AND $b$ WHEN THEY YIELD INFINITELY MANY SOLUTIONS TO OUR SYSTEM
 [/INST] My advice is to avoid dividing by variable expressions (since we might lose solutions or otherwise be implicitly making certain assumptions). We can row reduce the corresponding augmented matrix to get:
$$
\left[\begin{array}{cc|c}
a & 1 & 2a \\
1 & b & b \\
a & 5 - b^2 & 1
\end{array}\right]
\sim
\left[\begin{array}{cc|c}
1 & b & b \\
0 & 1-ab & 2a-ab \\
0 & 4 - b^2 & 1 - 2a
\end{array}\right]
$$
Now since the system must have infinitely many solutions, the second column cannot have a pivot. Thus, $1 - ab = 4 - b^2 = 0$. But then since the system must be consistent, the last two entries in the augmented column must be zero, so $2a - ab = 1 - 2a = 0$. Since:
$$
1 - 2a = 0 \iff \boxed{a = 1/2}
$$
we can substitute this into $1 - ab = 0$ to get that $\boxed{b = 2}$.
Indeed, this combination of $a$ and $b$ values makes the last two rows of the augmented matrix all zeroes, as desired. $~~\blacksquare$
</s>