<s>[INST] Maximum area of triangle inscribed in an ellipse If a triangle is inscribed in an ellipse $\frac{x^2}{25}+\frac{y^2}{16}=1$, Find the Maximum area of Triangle
My try:
Let $A(5\cos p, 4\sin p)$, $B(5\cos q, 4\sin q)$ and $C(5\cos r, 4\sin r)$ be vertices of the triangle
Its area is:
$$\Delta=0.5 \times\begin{vmatrix}
5\cos p &4 \sin p  &1 \\ 
 5 \cos q& 4 \sin q &1 \\ 
5 \cos r &4 \sin r  & 1
\end{vmatrix}=10\begin{vmatrix}
\cos p & \sin p &1 \\ 
\cos q & \sin q &1 \\ 
 \cos r&\sin r  & 1
\end{vmatrix}$$
$$\Delta=40\sin\left(\frac{p-q}{2}\right)\sin\left(\frac{q-r}{2}\right)\sin\left(\frac{r-p}{2}\right)$$
EDIT:
According to the mind blowing hint given by Mohammad Zuhair khan:
$p \gt q \gt r$ 
$$p=q+m$$
$$q=r+n$$
where $m,n \gt 0$
So we have
$$p-q=m$$
$$q-r=n$$
$$r-p=-(m+n)$$
Then
$$\Delta=40 \sin\left(\frac{m}{2}\right)\sin\left(\frac{n}{2}\right)
\sin\left(\frac{m+n}{2}\right)$$
Ignoring the negative sign, since its is Area
Now
$$\Delta(m,n)=10(\sin m+\sin n-\sin(m+n))$$
Using Partial differentiation for optimization we have:
$$\frac{\partial \Delta}{\partial m}=0$$
$$\cos m=\cos(m+n)$$
$$m=2\pi-m-n$$
$$2m+n=2\pi$$
Like wise by symmetry:
$$2n+m=2\pi$$
So
$$m=n=\frac{2\pi}{3}$$
Hence
$$\Delta_{max}=10(\frac{\sqrt{3}}{2}+\frac{\sqrt{3}}{2}
+\frac{\sqrt{3}}{2})=15\sqrt{3}$$
But how to know using partial differentiation Maximum occurs?
Can any one give me a link when does partial differentiation applicable for optimization problems?
 [/INST] Introduce new coordinates $(x',y') = (\frac{4}{5}x,y)$ so that the ellipse becomes a circle in these new coordinates. Any area $A$ in the original coordinate system will have the area $A'=\frac{4}{5}A$ in terms of the new coordinates. So the problem is reduced to finding the largest triangle area inscribed in a circle, and then converting this area back to the corresponding area in the original coordinate system.
The largest triangle inscribed in a circle is the only equilateral triangle that fits, which has the area $A'=3\frac{\sqrt{3}}{4}r^2$, where $r$ is the radius of the circle, and $r=4$ in this case. See the proof in CY Aries answer for how to find this area.
Then convert back to the old coordinates: $A=\frac{5}{4}A'$. The result is $A=15\sqrt{3}$.
</s>