<s>[INST] Function Optimization with Non-linear Constraint, Lagrange Multipliers Fails I am trying to maximize the function $A(x,y)=\frac{1}{2}(x(12-x)+y(13-y))$ subject to the constraint $x^2+(12-x)^2-y^2-(13-y)^2=0$.
My attempt:
$\begin{align*} \nabla A=\frac{1}{2}\langle 12-2x,\,13-2y\rangle &= \lambda\langle4x-24,\, -4y+26\rangle\\ \implies&\begin{cases} -x+6=\lambda(4x-24)\\-y+\frac{13}{2}=\lambda(-4y+26)\\x^2+(12-x)^2-y^2-(13-y)^2=0\end{cases}\end{align*}$
But clearly there is no solution due to the first two equations.
Using Wolfram Alpha, however, yields a maximum at $\displaystyle \left(\frac{17}{2},\,\frac{13}{2}\right)$ being $A=36$ and shows a nice little graph.
 [/INST] For a constrained problem 
\begin{align}
\max{} & f(x,y) \\
& g(x,y)\le 0
\end{align}
you must write the Lagrangian:
$$
L(x,y,\lambda)=f(x,y)+\lambda g(x,y)
$$
Then you must find stationary points of your Lagrangian (attention this is only necessary conditions, see wiki)
\begin{align}
\partial_x L &= 0 = \partial_x f+ \lambda \partial_x g \\
\partial_y L &= 0 = \partial_y f+ \lambda \partial_y g \\
\partial_\lambda L &= 0 = g   
\end{align}
With your example, this is essentially computations:
$$
L(x,y,\lambda)=\frac{1}{2} ((12-x) x+(13-y) y)-\lambda 
   \left(x^2+(12-x)^2-y^2-(13-y)^2\right)
$$
your three equations are (after simplification):
\begin{align}
 -(-6 + x) (1 + 4 \lambda) &= 0 \\
\frac{1}{2} (-13 + 2 y) (-1 + 4 \lambda) &= 0\\
  -x^2-(12-x)^2+y^2+(13-y)^2 &=0 &\\
\end{align}
The (real) solutions are:
$$
(x,y,\lambda)=(\frac{7}{2},\frac{13}{2},-\frac{1}{4})
$$
and
$$
(x,y,\lambda)=(\frac{17}{2},\frac{13}{2},-\frac{1}{4})
$$
You can check that for these two solutions 
$$
f(\frac{7}{2},\frac{13}{2})=f(\frac{17}{2},\frac{13}{2})=36
$$
Extra: to check that these points are maximizers you must check that the Hessian of $f(x,y)$ is symmetric definite negative, which is clearly the case as:
$$
\nabla^2f=\left(\begin{array}{cc}-1& 0 \\ 0 & -1\end{array}\right)
$$
</s>