<s>[INST] Prove that $a\sqrt{b^3+1}+b\sqrt{c^3+1}+c\sqrt{a^3+1} \leq 5$ Let $a,b,c$ be nonnegative real numbers such that $a+b+c=3$. Prove that
$$a\sqrt{b^3+1}+b\sqrt{c^3+1}+c\sqrt{a^3+1} \leq 5$$
I found a point at which the equality is attended, say $a=0,b=1,c=2$. But I have no idea how to prove it. I tried to use the AM-GM inequality but then I obtained the more difficult one. Please help me. Thank you very much.
 [/INST] Start out as before:
$$
a\sqrt{b^{3}+1}+b\sqrt{c^{3}+1}+c\sqrt{a^{3}+1}\leq5
$$
$$
a\cdot\sqrt{\left(b+1\right)\left(b^{2}-b+1\right)}\leq a\cdot\frac{b^{2}+2}{2}
$$
$$
\frac{a\left(b^{2}+1\right)+b\left(c^{2}+1\right)+c\left(a^{2}+1\right)}{2}=3+\frac{ab^{2}+bc^{2}+ca^{2}}{2}
$$
Continue by lagrange multipliers:
$$
\Lambda=ab^{2}+bc^{2}+ca^{2}+\lambda\left(a+b+c-3\right)
$$
$$
\begin{cases}
\partial_{a}\Lambda=b^{2}+2ac-\lambda & \Rightarrow\lambda=b^{2}+2ac\\
\partial_{b}\Lambda=c^{2}+2ab-\lambda & \Rightarrow\lambda=c^{2}+2ab\\
\partial_{c}\Lambda=a^{2}+2bc-\lambda & \Rightarrow\lambda=a^{2}+2bc
\end{cases}
$$
$$
H=\left(\begin{array}{cccc}
2c & 2b & 2a & 1\\
2b & 2a & 2c & 1\\
2a & 2c & 2b & 1\\
1 & 1 & 1 & 0
\end{array}\right)
$$
$$
b^{2}+2ac=c^{2}+2ab\Rightarrow(b-c)\left(b+c-2a\right)=0
$$
$$
(b-c)\left(b+c-2a\right)=(a-c)\left(a+c-2b\right)=(a-b)\left(a+b-2c\right)=0
$$
If we have no two equal, then we arrive at a contradiction:
$$
b+c-2a=a+c-2b=a+b-2c=0\Rightarrow a=b=c
$$
Hence assume $a-b=0$, then either $a-c=0$ and getting $a=b=c=1$,
or: $a+c-2a=0\Rightarrow a=c$ yielding the same results. Hence the
only candidate where $abc\neq0$ is:
$$
a=b=c=1
$$
Here, we get:
$$
ab^{2}+bc^{2}+ca^{2}=3
$$
Assume $b=0$ if we are on the boundary, then by AM-GM:
$$
ab^{2}+bc^{2}+ca^{2}=ca^{2}=\frac{1}{2}\left(2\left(3-a\right)a^{2}\right)\leq\frac{1}{2}\left(\frac{2\left(3-a\right)+a+a}{3}\right)^{3}=4
$$
Hence we have 
$$
\frac{ab^{2}+bc^{2}+ca^{2}}{2}\leq2
$$
As needed (maybe not too detailed why the $a=b=c=1$ is a saddle point). THe surface is shown below:

</s>