<s>[INST] Determine all the critical points of the given system and discuss their type and stability. 
Determine all the critical points of the given system and discuss their type and stability.

$$\frac{\mathrm{d} x}{\mathrm{d} t}=x(1-x-y)$$$$\frac{\mathrm{d} y}{\mathrm{d} t}=y({\frac{3}{4}-y-\frac{1}{2}x})$$$$(1,0),(\frac{1}{2},\frac{1}{2})$$$$A=\begin{pmatrix}
1-2x-y, &-x \\-\frac{y}{2}, 
 &\frac{3}{4}-2y-\frac{x}{2} 
\end{pmatrix}$$$$\begin{pmatrix}
\frac{1}{2} &-\frac{1}{2} \\-\frac{1}{4} 
 &-\frac{1}{2} 
\end{pmatrix}$$
$$(\lambda-\frac{1}{2})(\lambda+\frac{1}{2})=\frac{1}{8}$$
$$V_y=\frac{3}{4}-2y-\frac{x}{2}$$$$V_{xx}=1-2x-y$$$$V_{yy}=-\frac{5}{2}$$$$V_{xy}=-x$$$$R(x,y)=V_{xx}V_{yy}-2V_{xy}^2$$$$=-\frac{5}{2}+5x+\frac{5}{2}y-2x^2$$$$R(1,0)=\frac{1}{2}$$

But where to from here?

 [/INST] There is an issue with your critical points. We want to simultaneously solve
$$\tag 1 x(1-x-y) = 0 \\ y\left(\dfrac{3}{4}-y-\dfrac{1}{2}x\right) = 0 $$
We see that for $x = 0$ in the first equation, we solve the second and find $y = 0, y = \dfrac{3}{4}$.
We see that for $y = 0$ in the second equation, we solve the first and find $x = 0, x = 1$.
Next, we assume that $x \ne 0, y \ne 0$, and have to solve the system 
$$1-x-y = 0 \\ \dfrac{3}{4}-y-\dfrac{1}{2}x = 0 $$
Negating the first equation and adding gives $x = \dfrac{1}{2}, y = \dfrac{1}{2}$.
Summing up, we have four critical points as
$$(x, y) = (0, 0), \left(0, \dfrac{3}{4}\right), (1, 0),  \left(\dfrac{1}{2}, \dfrac{1}{2} \right)$$
Next we find the Jacobian matrix of $(1)$
$$J(x, y) = \begin{pmatrix}
\dfrac{\partial u}{\partial x}& \dfrac{\partial u}{\partial y}\\
\dfrac{\partial v}{\partial x}& \dfrac{\partial v}{\partial y}
\end{pmatrix}
=\begin{pmatrix}
1-2x-y &-x \\-\dfrac{y}{2} &\dfrac{3}{4}-2y-\dfrac{x}{2}
\end{pmatrix}$$
Hints on remaining parts


*

*Find the eigenvalues of the Jacobian at each critical point.

*Classify the stability using the following
If the eigenvalues are real


*

*Eigenvalues both positive = An Unstable Node: All trajectories in the neighborhood of the fixed point will be directed outwards and away from the fixed point.

*Eigenvalues both negative = A Stable Node: All trajectories in the neighborhood of the fixed point will be directed towards the fixed point.

*Eigenvalues opposite sign = An Unstable Saddle Node: Trajectories in the general direction of the negative eigenvalue's eigenvector will initially approach the fixed point but will diverge as they approach a region dominated by the positive (unstable) eigenvalue.
If the eigenvalues are complex conjugates - their real parts are equal and their imaginary parts have equal magnitudes but opposite sign.


*

*Real parts positive = An Unstable Spiral: All trajectories in the neighborhood of the fixed point spiral away from the fixed point with ever increasing radius.

*Real parts negative = An Stable Spiral: All trajectories in the neighborhood of the fixed point spiral into the fixed point with ever decreasing radius.
You might also like to review this and this and this.
Look at a phase portrait to verify the results

</s>