<s>[INST] Find the maximum of $P$ Given $a, b, c \geq 0$ and $a + b + c = 6$. Find the maximum of:
$$P = (a-b)(b-c)(c-a)$$
My guess is the maximum of $P$ would be just $0$ but I don't know whether this guess is true
And if it's true, how do I prove it?
 [/INST] Suppose $c = \min\{a,b,c\},$ so
$$(a-c)^2 \leqslant a^2, \quad (b-c)^2 \leqslant b^2.$$
By the AM-GM inequality we have
$$\begin{aligned}P^2 = (a-b)^2(b-c)^2(c-a)^2 & \leqslant a^2b^2(a-b)^2 = \frac14 \cdot 2ab \cdot 2ab \cdot (a-b)^2 \\& \leqslant \frac14\left(\frac{2ab+2ab+(a-b)^2}{3}\right) ^3\\&=\frac{(a+b)^6}{108}\leqslant \frac{(a+b+c)^6}{108}\end{aligned}$$
Therefore
$$(a+b+c)^3 \geqslant 6 \sqrt 3 (a-b)(b-c)(c-a).$$
Equality occur when $a^2+b^2=4ab,\,c= 0$ and permutation.
Because $a+b+c=6,$ so
$$(a-b)(b-c)(c-a) \leqslant \frac{(a+b+c)^3}{6\sqrt 3} = 12\sqrt 3.$$
Equality occur when
$$a^2+b^2=4ab,\,c= 0,\,a+b+c=6. \quad (1)$$
Solve $(1)$ we get $a = 3+\sqrt3,\,b = 3-\sqrt3,\, c = 0$ or $a = 3-\sqrt3, \,b = 3+\sqrt3,\, c = 0.$
Thefore $P_{\max} =  12\sqrt 3.$
</s>