<s>[INST] Show that the vector field does not have periodic orbit Show that the vector field $F(x,y)=(2x-x^5-xy^4,y-y^3-x^2y)$ defined in $R^2$ does not have periodic orbits; the Bendixson criterion is not useful.
 [/INST] @Robert Israel provides the best answer. This post is an elaboration of his remarks.
Dynamical System
Find the periodic solution for the dynamical system:
$$
%
\begin{align}
%
 \dot{x} &= -x \left(x^4+y-2\right) \\
%
 \dot{y} &= -y \left(x^2+y^2-1\right) \\
%
\end{align}
\tag{1}
%
$$
Poincaré–Bendixson Theorem
Use the theorem of Poincare and Bendixson to identify a trapping region, an area where the sign of the radial time derivative can change.
The trapping region must


*

*Be closed and bounded,

*Not contain any critical points.


Fixed points
Proceed by locating fixed points.
Nullclines
Find the nullclines.
$$
\begin{array}{ccccl}
  \dot{x} = 0 &\implies &y &= &\left\{ 2-x^4 \right\} \\
  \dot{y} = 0 &\implies &x &= &\left\{ 0, \pm\sqrt{1-x^2} \right\}
\end{array}
$$
Identify fixed points
The fixed points are:
$$
\left[ \begin{array}{c}
 \dot{x} \\ \dot{y}
\end{array} \right]_{(0,0)}
=
\left[ \begin{array}{c}
 \pm\sqrt[4]{2} \\ 0
\end{array} \right]
$$
At this juncture, invoke the arguments of @Robert Israel and you are done.
To reinforce the lesson, the answer continues with the canonical approach. The plot below shows the flow and the $\dot{x}$ nullcline in black and the $\dot{y}$ nullcline in red.

Compute $\dot{r}$
The polar coordinate transform
$$
%
\begin{align}
%
 x &= r \cos \theta \\
%
 y &= r \sin \theta \\
%
\tag{2}
\end{align}
%
$$
implies
$$
  r^{2} = x^{2} + y^{2}
\tag{3}
$$
Differentiate (3) with respect to time:
$$
2r\dot{r} = 2x \dot{x} + 2y \dot{y}
$$
Therefore
$$
\dot{r} = \frac{x \dot{x} + y \dot{y}} {r}
\tag{4}
$$
Transform $\dot{x}$ and $\dot{y}$ to $r$ and $\theta$ using (2):
$$
%
\begin{align}
%
 \dot{x} 
&= -x \left(x^4+y-2\right) 
= -r \cos \theta \left(r^4 \cos ^4\theta+r \sin \theta-2\right) \\
%
 \dot{y} 
&= -y \left(x^2+y^2-1\right)
= -r \left(r^2-1\right) \sin \theta \\
%
\end{align}
%
$$
Inserting these identities in $(4)$ produces the final differential equation:
$$
\dot{r} = -r \left(-\sin ^2\theta+r^4 \cos ^6\theta+r^2 \sin ^4\theta+\cos ^2\theta (r \sin \theta-1) (r \sin \theta+2)\right)
\tag{5}
$$
The fact that this differential equation is so difficult is a sign that you have either made a mistake, or missed the reasoning on stationary points. 
To close the discussion, examine the plot below which shows $\dot{r}\left(r, \theta\right)$. The purple line is the nullcline $\dot{r}=0$. Below the line $\dot{r}>0$, above the line $\dot{r}<0$.

</s>