<s>[INST] Dynamical System is fixed point at origin hyperbolic or asymptotically stable and is the system Hamiltonian I am given a system
$\dot x = f(x,y) = (x^2 + y^2)(x^3 + y^2x -2y - x) \\
\dot y = g(x,y) = (x^2 + y^2)(y^3 + x^2y +2x - y) $
and I am asked if the fixed point at $(0,0)$ is hyperbolic or asymptotically stable. I am also asked whether the system is a Hamiltonian.
So I know that a hyperbolic fixed point is a fixed point that does not have a centre manifold, so this makes me think that it is a hyperbolic fixed point but I'm not really sure.
Then onto asymptotically stable, my understanding of this is that we require the system to be both Liapounov stable and quasi-asymptotically stable, but I really dont know how to go about this.
Finally about it being a Hamiltonian system. I know in order to be a Hamiltonian the system can be written in the form
$H(q_i,p_i)$ such that 
$\dot q_i= \frac{\partial H}{\partial p_i} \\
\dot p_i= -\frac{\partial H}{\partial q_i}$
But I really dont know how to do this for this DS.
Also i it helps I converted the DS into polar coords, giving me
$\dot r = r^5 - r^3 \\
\dot \theta = 2r^2$
 [/INST] This is about the stability part:
Asymptotic stability of the equilibrium means that it is stable and attractive.
The origin $(0,0)$ is an equilibrium.
Consider the Lyapunov function candidate $V(x,y) = \frac{1}{2}x^2 + \frac{1}{2}y^2$. The derivative along the trajectories of $\dot x = f(x,y)$ and $\dot y = g(x,y)$ yields
\begin{align}
\dot V  = x\,\dot x + y\,\dot y &= x(x^2+y^2)(x^3+y^2x-2y-x) + y(x^2+y^2)(y^3+x^2y+2x-y) \\
& = (x^2+y^2)(x^4+y^2x^2-2yx-x^2) + (x^2+y^2)(y^4+x^2y^2+2xy-y^2) \\
& = (x^2+y^2)(x^4+y^4) + (x^2+y^2)(2y^2x^2) - (x^2+y^2)^2 \\
& = (x^2+y^2)\left((x^4+y^4) +(2y^2x^2) \right) - (x^2+y^2)^2 \\
& = (x^2+y^2)(x^2+y^2)^2  - (x^2+y^2)^2 
\end{align}
Insert $2V = x^2+y^2$:
\begin{align}
\dot V  &  = 8\,V^3  - 4 V^2  = -8\,V^2\left(\frac{1}{2}-V\right)
\end{align}
(Note: this is qualitatively similar to that what could be derived from the representation in polar coordinates)
Accordingly, $\dot V$ is only negative in the set $\{(x,y)\in\mathbb R^2: x^2+y^2<1 \}$ which contains the origin. That is, the origin is (locally) asymptotically stable. The unit disc defines also the region of attraction.
</s>