<s>[INST] Not so easy optimization of variables? What is the maximum value of $x^2+y^2$, where $(x,y)$ are solutions to $2x^2+5xy+3y^2=2$ and $6x^2+8xy+4y^2=3$. (calculus is not allowed). I tried everything I could but whenever I got for example $or$ $x^2+y^2=f(y)$ or $f(x)$ the function $f$ would always be a concave up parabola, so I could not find a maximum for either variable. However, I also don't see how you could solve it if you leave both variables on one side. And by the way I know that you can solve for $x$ and $y$ using the quadratic formula and get $4$ different solutions but I am looking for a much more efficient way than that. 
This question came from a math competition from the Math Honor Society, Mu Alpha Theta.
 [/INST] Since my other entry is already rather long, I thought I'd separately add a couple of other methods that work (at least in principle), but which involve too much effort to be practical "contest math" answers.
The approach I discussed already can also be carried out in polar coordinates, with the equations
$$ 2 \cos^2 \theta \ + \ 5 \sin \theta \cos \theta \ + \ 3 \sin^2 \theta \ = \ \frac{2}{r^2} \ \ , \ \ 6 \cos^2 \theta \ + \ 8 \sin \theta \cos \theta \ + \ 4 \sin^2 \theta \ = \ \frac{3}{r^2}   $$
$$ \Rightarrow  \ \ 6 \cos^2 \theta \ + \ 15 \sin \theta \cos \theta \ + \ 9 \sin^2 \theta \ =  \ 12 \cos^2 \theta \ + \ 16 \sin \theta \cos \theta \ + \ 8 \sin^2 \theta $$
$$ \Rightarrow  \ \ 6 \cos^2 \theta \ + \  \sin \theta \cos \theta \ - \  \sin^2 \theta \ = \ 0 \ \ \Rightarrow \ \ 7 \cos^2 \theta \ + \  \sin \theta \cos \theta \  = \ 1  $$
$$ \Rightarrow \ \ 7 \ (  \frac{1}{2} [1 + \cos 2\theta] \ ) \ + \  \frac{1}{2}\sin 2 \theta  \  = \ 1 \ \ \Rightarrow \ \ 7 \  \cos 2\theta \ + \  \sin 2 \theta  \  = \ -5 $$
$$  \Rightarrow \ \ \frac{7}{\sqrt{50}} \  \cos 2\theta \ +  \ \frac{1}{\sqrt{50}} \ \sin 2 \theta  \  = \ -\frac{5}{\sqrt{50}} \ \ . $$
The typical method of solving this by the use of an auxiliary angle and an "angle-addition formula" is not particularly enlightening, so we will simply use $ \ x = \cos 2\theta \ $ to write
$$   \frac{7}{\sqrt{50}} \  x \ + \ \frac{1}{\sqrt{50}} \cdot  \sqrt{1-x^2}  \  = \ -\frac{1}{\sqrt{2}} \ \ \Rightarrow \ \ x^2 \ + \ \frac{7}{5} x \ + \ \frac{12}{25} \ = \ 0 $$
$$ \Rightarrow \ \ \cos 2 \theta \ = \ - \frac{3}{5} \ \ \Rightarrow \ \ \sin 2 \theta \ = \ - \frac{4}{5} \ \ ,  $$
$$ \Rightarrow \ \ \cos 2 \theta \ = \ - \frac{4}{5} \ \ \Rightarrow \ \ \sin 2 \theta \ = \  \frac{3}{5} \ \ ,  $$
after removing "spurious" solutions.  The "tangent half-angle formula" then yields
$$ \tan \theta \ = \ \frac{1 - \cos 2 \theta}{\sin 2 \theta} $$
$$ \Rightarrow \ \ \cos 2 \theta \ = \ - \frac{3}{5} \ \ \rightarrow \ \ \tan \theta \ = \ -2 \ \ , \ \ \cos 2 \theta \ = \ - \frac{4}{5} \ \ \rightarrow \ \ \tan \theta \ = \ +3 \ \ . $$
We have thus confirmed the slopes of the two lines along which the intersections of the conic sections lie.  To answer the original question, however, we only need, in polar coordinates, to find the maximum value of $ \ r^2 \ . $  The trigonometric values we have obtained can now be used in either of the conic section equations to show that the extrema are
$$ \tan \theta \ = \ +3 \ \ \Rightarrow \ \ \sin \theta \ = \ \frac{3}{\sqrt{10}} \ , \ \cos \theta \ = \ \frac{1}{\sqrt{10}} $$
$$ \Rightarrow \ \ r^2 \ = \ \frac{2}{2 \ \left(\frac{1}{10} \right) \ + \ 5 \left(\frac{3}{\sqrt{10}} \right) \left(\frac{1}{\sqrt{10}} \right) \ + \ 3 \left(\frac{9}{10} \right) } \ = \ \frac{5}{11} \ \ , $$
$$ \tan \theta \ = \ -2 \ \ \Rightarrow \ \ \sin \theta \ = \ \frac{2}{\sqrt{5}} \ , \ \cos \theta \ = \ -\frac{1}{\sqrt{5}} $$
$$ \Rightarrow \ \ r^2 \ = \ \frac{2}{2 \ \left(\frac{1}{5} \right) \ + \ 5 \left(\frac{2}{\sqrt{5}} \right) \left(\frac{-1}{\sqrt{5}} \right) \ + \ 3 \left(\frac{4}{5} \right) } \ = \ \frac{5}{2} \ \ . $$
[Inserting the trigonometric values into the other conic section equation leads to the same results, as does using the other set of sine and cosine values associated with each tangent value.]
$$ \ \ $$
Because the problem is looking for extremal values of a function at specific points, looking for "critical points" through the use of calculus is not all that helpful, even if it were permitted.  Since we are seeking the maximum value of a function for points that lie on two curves simultaneously, this suggests the use of Lagrange multipliers with two constraints.  We then wish to extremize $ \ f(x,y) = x^2 + y^2 \ $ with the constraint functions $ \ g(x,y) = 2x^2 + 5xy + 3y^2 - 2 \ $ and $ \ h(x,y) = 6x^2 + 8xy + 4y^2 - 3 \ . $  The Lagrange equations that emerge, however, do not seem to encourage quick solution:
$$ \nabla f \ = \ \lambda \ \nabla g \ + \ \mu \ \nabla h $$
$$ \Rightarrow \ \ 2x \ = \ \lambda \ (4x + 5y) \ + \ \mu \ (12x + 8y) \ \ , \ \  2y \ = \ \lambda \ (5x + 6y) \ + \ \mu \ (8x + 8y) \ . $$
[By using the results obtained by other means, we find that the two solutions are
$$ ( \pm \frac{1}{\sqrt{2}} \ , \ \mp \sqrt{2} ) \ \ , \ \ \lambda \ = \ -\frac{8}{5} \ \ , \ \ \mu \ = \ \frac{19}{10} \ \ , $$
$$ ( \pm \frac{1}{\sqrt{22}} \ , \ \pm \frac{3}{\sqrt{22}} ) \ \ , \ \ \lambda \ = \ \frac{38}{55} \ \ , \ \ \mu \ = \ -\frac{17}{55} \ \ . $$  This may serve to provide some idea of how readily this answer could be obtained.]
So  the original question would seem to be one more easily answered by the application of less sophisticated techniques than with the tools of calculus.
</s>