<s>[INST] If the range of the function $f(x)=\frac{x^2+ax+b}{x^2+2x+3}$ is $[-5,4],a,b\in N$,then find the value of $a^2+b^2.$ If the range of the function $f(x)=\frac{x^2+ax+b}{x^2+2x+3}$ is $[-5,4],a,b\in N$,then find the value of $a^2+b^2.$

Let $y=\frac{x^2+ax+b}{x^2+2x+3}$
$$x^2y+2xy+3y=x^2+ax+b$$
$$x^2(y-1)+x(2y-a)+3y-b=0$$
As $x$ is real,so the discriminant of the above quadratic equation has to be greater than or equal to zero.
$$(2y-a)^2-4(y-1)(3y-b)\geq0$$
$$-8y^2+y(-4a+4b+12)+a^2-4b\geq0$$
As above quadratic inequality is greater than or equal to zero,so its discriminant is less than or equal to zero.
$$(-4a+4b+12)^2+32(a^2-4b)\leq0$$
I am stuck here,i cant find $a^2+b^2$.
 [/INST] Begin with
$$g(y)=-8y^2+y(-4a+4b+12)+a^2-4b\geq 0,\quad\forall y\in[-5,4].$$
Then we wish to solve $g(-5)=g(4)=0$, that is,
\begin{align}
-200-5(-4a+4b+12)+a^2-4b=0,\\
-128+4(-4a+4b+12)+a^2-4b=0,
\end{align}
or 
\begin{align}
a^2+20a-24b&=260,\\
a^2-16a+12b&=80.
\end{align}
Thus
\begin{align}
a-b=5\quad\mbox{and}\quad
0=a^2-4a-140=(a-14)(a+10)
\end{align}
and we obtain two sets of solutions 
$$a=14,\,b=9\quad\mbox{and}\quad
  a=-10,\,b=-15.$$
where the second solution does not fit because $a,$ $b$ are assumed to be positive integers. Hence $a^2+b^2=277.$
</s>