<s>[INST] Lyapunov's theory for differential equations Use Lyapunov's theory to discuss stability of the origin for
$$x' = -x^3 - xy^2 + xy^5$$
$$y' = -y^3 + x^2y - 3x^2y^4$$
$$L(x,y) = ax^2 + by^2$$
In class, we define $L(x,y)$ as being positive definite. Then we determine if the origin is a critical point. The we take the derivative of $L$ and determine if it's a negative definite or negative semidefinite. 
I can get to the point where we take the derivative. Simplifying and determining the negative definite or negative semidefinite is the problem. 
 [/INST] Given the system
$\dot x = -x^3 - xy^2 + xy^5, \tag 1$
$\dot y = -y^3 + x^2y - 3x^2y^4, \tag 2$
it is easily seen that $(0, 0)$ is a critical point; we seek a Lyapunov function of the form
$L(x,y) = ax^2 + by^2; \tag 3$
we have
$\dot L = 2ax \dot x+ 2by \dot y$
$= 2ax(-x^3 - xy^2 + xy^5) + 2by(-y^3 + x^2y - 3x^2y^4)$
$= -2ax^4 -2ax^2y^2 + 2ax^2y^5 -2by^4 +2bx^2y^2 -6bx^2y^5$
$= -2ax^4 - 2by^4 + 2(b - a)x^2y^2 + (2a - 6b)x^2y^5; \tag 4$
we may exploit our freedom in the selection of $a$ and $b$ to ensure that
$\dot L(x, y) < 0 \tag 5$
along the trajectories of (1)-(2) in a neighborhood of the origin:  we choose
$0 < a = 3b; \tag 6$
then
$2a - 6b = 0, \tag 7$
$b - a = -2b; \tag 8$
such a choice of $a$ and $b$ eliminates the complicating term $(2a - 6b)x^2y^5$ from (4); we are left with an expression containing only even powers of $x$ and $y$, viz.
$\dot L = -6bx^4 - 2by^4 - 2bx^2y^2 < 0 \tag 9$
whenever at least one of $x$, $y$ does not vanish; thus
$L = 3bx^2 + by^2 \tag{10}$
is a suitable Lyapunov function for (1)-(2) in the vicinity of $(0, 0)$, and in conclude that the origin is an asymptotically stable critical point of (1)-(2).
</s>