<s>[INST] Prove that $a^2b+b^2c+c^2a \leqslant 3$ for $a,b,c >0$ with $a^ab^bc^c=1$ 
Let $a,b,c >0$ and $a^ab^bc^c=1$. Prove that
$$a^2b+b^2c+c^2a \leqslant 3.$$

I don't even know what to do with the condition $a^ab^bc^c=1$. At first I think $x^x>1$, but I was wrong. This inequality is true, following by the verification from Mathematica
 [/INST] We shall prove the following inequalities:

If $x+y+z=3$ and $x,y,z>0$, then
  $$\tag{1}x^2y+y^2z+z^2x+xyz\le 4\label{1},$$
  $$\tag{2}3x^xy^yz^z+xyz\ge 4\label{2},$$
  i.e.,
  $$\tag{*}x^2y+y^2z+z^2x\le 3x^xy^yz^z\label{*}.$$

Accept $\eqref{*}$ for a moment.  Homogenizing the inequality by substitution
$$x=\frac{3a}{a+b+c},~y=\frac{3b}{a+b+c},~z=\frac{3c}{a+b+c},$$
we have $\forall a,b,c>0$,
$$\tag{**}a^2b+b^2c+c^2a\le 3(a^ab^bc^c)^{3/(a+b+c)}.\label{**}$$
If $a^ab^bc^c=1$, then we get the original inequality as desired.

Proof of $\eqref{1}$:
Note the LHS is cyclic, we cannot assume a specific order like $x\le
y\le z$, but we can assume WLOG that $y$ is in the middle (neither the minimum nor the maximum), then
$$z(y-x)(y-z)\le 0\\
\Rightarrow x^2y+y^2z+z^2x+xyz\le y(x^2+2xz+z^2)=\frac{1}{2}(2y)(x+z)^2\le\frac{1}{2}\left(\frac{2(x+y+z)}{3}\right)^3=4,$$
by AM-GM.  Equality holds when $(x,y,z)=(1,1,1),(2,1,0)$ along with the cyclic permutations.

Proof of $\eqref{2}$:
WLOG we assume $x=\min(x,y,z)$ and consider the following two cases:


*

*$x\ge 1/3$, then $y,z\ge 1/3$.  We note that the function


$$f(t)=\left(t+\frac{1}{3}\right)\ln t$$
is convex for $t\ge 1/3$, as
$$f''(t)=\frac{3t-1}{3t^2}\ge 0.$$
By Jensen,
$$
\begin{align}
&~~~~~~~x^{x+1/3}y^{y+1/3}z^{z+1/3}=\exp\left\{f(x)+f(y)+f(z)\right\}
\ge\exp\left\{3f\left(\frac{x+y+z}{3}\right)\right\}=\exp\left\{3f(1)\right\}=1,\\
&\Rightarrow x^xy^yz^z\ge (xyz)^{-1/3},\\
&\Rightarrow 3x^xy^yz^z+xyz\ge 3(xyz)^{-1/3}+xyz\ge 4\sqrt[4]{(xyz)^{-1}(xyz)}=4,
\end{align}
$$
where we have applied AM-GM for the last line.  Equality holds when $x=y=z=1$.


*

*$0<x\le1/3$.  We note that the function


$$g(t)=t\ln t$$
is convex for $t>0$, as
$$g''(t)=\frac{1}{t}> 0.$$
Again by Jensen, we have
$$
\begin{align}
&~~~~~~~y^yz^z=\exp\left\{g(y)+g(z)\right\}\ge\exp\left\{2g\left(\frac{y+z}{2}\right)\right\}
=\exp\left\{2g\left(\frac{3-x}{2}\right)\right\},\\
&\Rightarrow x^xy^yz^z\ge\exp\left\{g(x)+2g\left(\frac{3-x}{2}\right)\right\}
\ge\exp\left\{g\left(\frac{1}{3}\right)+2g\left(\frac{4}{3}\right)\right\}
=\frac{32}{27}\sqrt[3]{2}>\frac{4}{3},\\
&\Rightarrow 3x^xy^yz^z+xyz>4+xyz>4.
\end{align}
$$
The second line follows from that $h(t):=g(t)+2g((3-t)/2)$ is monotonically
decreasing for $t\in(0,1)$, so $h(x)\ge h(1/3)$ as $x\le 1/3$.  Proof of monotonicity:
$$h'(t)=\ln\left(\frac{2t}{3-t}\right)<0\iff 0<t<1.$$

Post-mortem:


*

*Despite the flow of the proof, the starting point is the homogeneous version \eqref{**};

*\eqref{1} is due to Vasile, which allows us to strengthen the inequality into a
symmetric one as \eqref{2} (I am not able to find the originial post on artofproblemsolving, but see e.g.,
here).  It seems that converting \eqref{*} into other symmetric form such as $x^3+y^3+z^3$ is too strong for it to hold;

*For \eqref{2}, the case of $x\le 1/3$ is unfortunately necessary, as
$x^xy^yz^z\ge(xyz)^{-1/3}$ does NOT always hold, e.g., check $x\to0,y-z\to 0$.  Is there any better estimate of $x^xy^yz^z$ so we can avoid the derivatives?  Another follow-up question is to determine the smallest $k$ such that $kx^xy^yz^z+xyz\ge k+1$ holds given the constraints.  It definitely fails for $k\le 1/2$.
</s>