<s>[INST] If the range of the function $f(x)=\frac{x^2+ax+b}{x^2+2x+3}$ is $[-5,4],a,b\in N$,then find the value of $a^2+b^2.$ If the range of the function $f(x)=\frac{x^2+ax+b}{x^2+2x+3}$ is $[-5,4],a,b\in N$,then find the value of $a^2+b^2.$

Let $y=\frac{x^2+ax+b}{x^2+2x+3}$
$$x^2y+2xy+3y=x^2+ax+b$$
$$x^2(y-1)+x(2y-a)+3y-b=0$$
As $x$ is real,so the discriminant of the above quadratic equation has to be greater than or equal to zero.
$$(2y-a)^2-4(y-1)(3y-b)\geq0$$
$$-8y^2+y(-4a+4b+12)+a^2-4b\geq0$$
As above quadratic inequality is greater than or equal to zero,so its discriminant is less than or equal to zero.
$$(-4a+4b+12)^2+32(a^2-4b)\leq0$$
I am stuck here,i cant find $a^2+b^2$.
 [/INST] $$-5 \le \frac{x^2+ax+b}{x^2+2x+3} \le 4 $$
As the denominator is positive, this is equivalent to
$$-5x^2-10x-15 \le x^2+ax+b \le 4x^2+8x + 12$$
which can be considered as two quadratic inequalities,
$$6x^2+(a+10)x+(b+15) \ge 0, \quad 3x^2+(8-a)x+(12-b) \ge 0$$
Note that equality must happen as well for some $x$. For this to be true for all reals, the discriminants must be then zero, and $-15 \le b \le 12$ to ensure the quadratics are both above the x axis. So
$$(a+10)^2 = 24(b+15), \quad (8-a)^2 = 12(12-b)$$
Solving, $a=14$ and $b = 9$ giving $a^2+b^2 = 277$.
</s>