<s>[INST] Min $P=\frac{x^5y}{x^2+1} +\frac{y^5z}{y^2+1} +\frac{z^5x}{z^2+1}$ 
Given $x,y,z$ are  positive numbers such that $$x^2y+y^2z+z^2x=3$$
Find the minium of value: $$P=\frac{x^5y}{x^2+1} +\frac{y^5z}{y^2+1} +\frac{z^5x}{z^2+1} $$

My Attempt:
$$P= x^3y+y^3z+z^3x - \left( \frac{x^3y}{x^2+1}+ \frac{y^3z}{z^2+1}+\frac{z^3x}{z^2+1}  \right)$$
$$x^2+1 \geq 2x \Rightarrow \left( \frac{x^3y}{x^2+1}+ \frac{y^3z}{z^2+1}+\frac{z^3x}{z^2+1}\right)\leq \frac{1}{2}\left(x^2y+y^2z+z^2x\right)=\frac{3}{2}$$
$$P \geq x^3y+y^3z+z^3x - \frac{3}{2}$$
I was stuck when I was finding the minium of $x^3y+y^3z+z^3x$

Help me, thank you so much
 [/INST] Too long for a comment.
Considering the minimization problem
$$
\max_{x,y,z} -f(x,y,z)\ \ \text{s. t.}\ \ g(x,y,z) = 0
$$
with
$$
\cases{
f(x,y,z) = \frac{x^5 y}{x^2+1}+\frac{x z^5}{z^2+1}+\frac{y^5 z}{y^2+1}\\
g(x,y,z) = x^2 y + y^2 z + z^2 x - 3
}
$$
and according to the notes, the point $x=y=z=1,\lambda=\frac 56$ is a stationary point on the lagrangian
$$
L=-f+\lambda g
$$
This point can be qualified using the bordered hessian which is
$$
H = \left(
\begin{array}{cccc}
 0 & 2 x y+z^2 & x^2+2 y z & 2 x z+y^2 \\
 2 x y+z^2 & -\frac{2 x^3 \left(3 x^4+9 x^2+10\right) y}{\left(x^2+1\right)^3} & -\frac{x^4 \left(3 x^2+5\right)}{\left(x^2+1\right)^2} & -\frac{z^4 \left(3
   z^2+5\right)}{\left(z^2+1\right)^2} \\
 x^2+2 y z & -\frac{x^4 \left(3 x^2+5\right)}{\left(x^2+1\right)^2} & -\frac{2 y^3 \left(3 y^4+9 y^2+10\right) z}{\left(y^2+1\right)^3} & -\frac{y^4 \left(3
   y^2+5\right)}{\left(y^2+1\right)^2} \\
 2 x z+y^2 & -\frac{z^4 \left(3 z^2+5\right)}{\left(z^2+1\right)^2} & -\frac{y^4 \left(3 y^2+5\right)}{\left(y^2+1\right)^2} & -\frac{2 x z^3 \left(3 z^4+9
   z^2+10\right)}{\left(z^2+1\right)^3} \\
\end{array}
\right)
$$
and at the point of interest we have
$$
H(1,1,1) = \left(
\begin{array}{cccc}
 0 & 3 & 3 & 3 \\
 3 & -\frac{11}{2} & -2 & -2 \\
 3 & -2 & -\frac{11}{2} & -2 \\
 3 & -2 & -2 & -\frac{11}{2} \\
\end{array}
\right)
$$
now according to the notes, $n = 3, m = 1$ so we need to consider the two minors $H_3$ and $H_4$. Following we have $\det(H_3) = 63 > 0$ and $\det(H_4) = -\frac{1323}{4} < 0$. The signs are $\text{sign}(\det(H_3)) = (-1)^{m+1} = 1$ and $\text{sign}(\det(H_4))$ should alternate as negative.
This characterizes the point $(x,y,z)=(1,1,1)$ as a local minimum for $f$ constrained.
NOTE
Taking from the restriction $x$ as
$$
x = \phi(y,z) = \frac{\sqrt{z^4-4 y^3 z+12 y}-z^2}{2 y}
$$
and substituting into $f(x,y,z)$ we obtain a surface
$$
u = f(\phi(y,z),y,z)
$$
which can be depicted from the following plot. In blue the relative minimum point with coordinates $u=\frac 32, y=1,z=1$

</s>