<s>[INST] Differential equation\higher order I need help solving this task, if anyone had a similar problem it would help me.
The task is: Find the general solution of the differential equation
$$ y''-y' - 2y=e^{2x} \cos^2 x$$
For homogeneous I get:
$$y_H=c_1e^{2x}+c_2e^{-x}$$
I have a problem with the particular, I tried:
$$y=(A\sin^2x+B\cos^2x)e^{2x}\\y'=(2A\sin x \cos x + 2B \sin x \cos x) e^{2x} +(A\sin^2 x + B \cos ^2 x) 2e^{2x}\\y''=2e^{2x}(2A \sin x  \cos x - 2B \sin x \cos x + A\sin^2 x +B \cos^2 x)$$
I have no idea if this is correct?
I don't know what's next from here ..
Thanks in advance !
 [/INST] Your homogeneous solution is correct.
Note that $\cos^2(x)=\frac{1}{2}(1+\cos(2x))$, so the particular solution should be of the form $$y_{p}(x)=Axe^{2x}+Be^{2x}\cos(2x)+Ce^{2x}\sin(2x).$$
Substituting we find that $A=\frac{1}{6}$, $B=-\frac{1}{26}$ and $C=\frac{3}{52}$.

$$y'_p=2C\mathrm{e}^{2x}\sin\left(2x\right)-2B\mathrm{e}^{2x}\sin\left(2x\right)+2C\mathrm{e}^{2x}\cos\left(2x\right)+2B\mathrm{e}^{2x}\cos\left(2x\right)+2Ax\mathrm{e}^{2x}+A\mathrm{e}^{2x}$$
and $$y''_p=\mathrm{e}^{2x}\left(-2\left(2C+2B\right)\sin\left(2x\right)+2\left(2C-2B\right)\cos\left(2x\right)+2A\right)$$
$$+2\mathrm{e}^{2x}\left(\left(2C-2B\right)\sin\left(2x\right)+\left(2C+2B\right)\cos\left(2x\right)+2Ax+A\right)$$
So we have $$y''_p-y'_p-2y_p$$
$$=e^{2x}\big[\cos(2x)(4C-4B+4C+4B-2C-2B-2B)+\sin(2x)(-4C-4B+4C-4B-2C+2B-2C)+2A+4Ax+2A-2Ax-A-2Ax\big]$$
$$=e^{2x}\big[\cos(2x)(6C-4B)+\sin(2x)(-4C-6B)+3A\big]$$
$$=\frac{e^{2x}}{2}(1+\cos(2x)).$$
So we have $3A=\frac{1}{2}\implies A=\frac{1}{6}$, $6C-4B=\frac{1}{2}$ and $-4C-6B=0$ which give $C=\frac{3}{52}$ and $B=-\frac{1}{26}$.
</s>