<s>[INST] Application of Poincaré-Bendixson theorem Consider the system
$$x' = 3xy^2-x^2y \\
y' = 5x^2y - xy^2$$
Show that the system has no periodic solutions.
This is a tricky example. Linearization leads nowhere and I'm having a hard time constructing a Lyapunov function that does the trick. $V = 1/2(x^2+y^2)$ gives
$$V'(x,y) = 3x^2y^2-x^3y +5x^2y^2 -xy^3 = 8x^2y^2 -xy(x^2+y^2))$$
But this doesn't tell us much nice things about the origin. If anything, it looks as though the origin is repelling since small perturbations gives us that the $8x^2y^2$ term dominates the minus term. Maybe it's possible to show that there are no elliptical orbits somehow, but that doesn't exclude other, more exotic, periodic trajectories.
How to proceed...?
 [/INST] I'll give the solution to your non-linear system $$\frac{dx}{dt}=3xy^2-x^2y$$
and $$\frac{dy}{dt}=5x^2y-xy^2$$
Divivding eq 2 by eq 1, we get $$\frac{dy}{dx}=\frac{5x^2y-xy^2}{3xy^2-x^2y}$$
Now, dividing by $x^3$ both the numenator and the denumenator we get that $$\frac{dy}{dx}=\frac{5\frac{y}{x}-(\frac{y}{x})^2}{3(\frac{y}{x})^2-\frac{y}{x}}$$
Let $y=x \cdot v$ then $\frac{dy}{dx}=x\frac{dv}{dx}+v$, thus the D.E transformed into a separable eq.
$$x\frac{dv}{dx}+v=\frac{5v-v^2}{3v^2-v}=\frac{5-v}{3v-1},$$ therefore
$$x\frac{dv}{dx}=\frac{5-v-3v^2+v}{3v-1}=\frac{5-3v^2}{3v-1}$$
$$ \Rightarrow \frac{3v-1}{5-3v^2}dv=\frac{dx}{x}$$ 
$$ \Rightarrow \int{\frac{3v-1}{5-3v^2}dv}=\int{\frac{dx}{x}}$$ 
Setting $
\sqrt {\frac{3}{5}} v= \sin \theta   \Rightarrow \sqrt {\frac{5}{3}} \cos \theta d\theta  = dv$, then 
$$\int{\frac{3\sqrt {\frac{5}{3}} \sin \theta 
-1}{5(1- \sin^2 \theta) }\sqrt {\frac{5}{3}} \cos \theta d\theta}=\int{\frac{dx}{x}}$$ 
so that $$\int{\frac{3\sqrt {\frac{5}{3}} \sin \theta 
-1}{5(1- \sin^2 \theta) }\sqrt {\frac{5}{3}} \cos \theta d\theta}=\int {\tan \theta d\theta }  - \sqrt {\frac{5}{3}} \frac{1}{5}\int {\sec \theta d\theta }  
\\
=  - \ln \left| {\cos \theta } \right| - \frac{1}{{\sqrt {15} }}\ln \left| {\sec \theta  + \tan \theta } \right|
$$
Hence the solution is, $$
 - \ln \left| {\cos \theta } \right| - \frac{1}{{\sqrt {15} }}\ln \left| {\sec \theta  + \tan \theta } \right|
=\ln \left| {x } \right|  +C$$ Finally, we may write the result as
$$\left| {\cos \theta } \right|\cdot
\left| {\sec \theta  + \tan \theta } \right|^{{\textstyle{1 \over {\sqrt {15} }}}} 
=\frac{k}{x}$$ where $k=\exp(-C)$.
</s>