<s>[INST] Find the Range of $f(a,b)=2a+b-3ab$ Let $a,b>0$,and such $$a^2+b^2-ab=4$$ Find the range
$$f(a,b)=2a+b-3ab$$
I try let $a=x+y,b=x-y$,then
$$a^2+b^2-ab=4\Longrightarrow x^2+3y^2=16,x>y,x>-y$$
so we Let
$$\begin{align}
x &=4\cos{t}, \qquad y=\dfrac{4}{\sqrt{3}}\sin{t} \\
f(a,b) &= 2a+b-3ab \\
&= \dfrac{3x}{2}+\dfrac{y}{2}-\frac{3}{4}x^2+\dfrac{3}{4}y^2\\
&=6\cos{t}-\dfrac{2}{\sqrt{3}}\sin{t}-12\cos^2{t}+4\sin^2{t}
\end{align}$$
Then I stuck 
 [/INST] I think you have made a mistake in your computations as I did the arithmetic with my CAS. However, if we make the substitutions
$$\begin{align}
a &= 2 \cos t + \frac{2}{\sqrt{3}} \sin t = \frac{4}{\sqrt{3}} \sin(t+\frac{\pi}{3}) \\
b &= 2 \cos t - \frac{2}{\sqrt{3}} \sin t = \frac{4}{\sqrt{3}} \sin(t+\frac{2\pi}{3})
\end{align}$$
then the constraint equation
$$a^2+b^2-ab=4$$
will be satisfied identically $4=4$. Also, to satisfy $a \gt 0$ and $b \gt 0$ we require that
$$\begin{cases}
0 \lt t+\frac{\pi}{3} \lt \pi \\
0 \lt t+\frac{2\pi}{3} \lt \pi
\end{cases}
\to 
\begin{cases}
-\frac{\pi}{3} \lt t \lt \frac{2\pi}{3} \\
-\frac{2\pi}{3} \lt t \lt \frac{\pi}{3}
\end{cases}
\to
-\frac{\pi}{3} \lt t \lt \frac{\pi}{3}$$
So, it remains to work on $f(a,b)=2a+b- 3ab$. Now, if we do the substitution in $f(a,b)$ we will get
$$\begin{align}g(t) &= f(a(t),b(t)) \\
&=\frac{2}{\sqrt{3}}\sin t + 6 \cos t - 16 \cos^2 t +4  \\
\end{align}$$
Finally, we can use any method in calculus to find the Maximum and Minimum of this continuous function on the interval $(-\frac{\pi}{3},\frac{\pi}{3})$. So the range of $f(a,b)$ with the desired constraints on $a$ and $b$ will be obtained. In fact, we just reduced a constrained multi-variable optimization problem to a usual single variable optimization problem in calculus.
</s>