<s>[INST] Pretty solution to the trigonometric equation Problem
Consider the trigonometric equation:
$$
a\sin x+b\cos x-\cos x\sin x=0\qquad(0\le x<2\pi)\tag{*}
$$
try to analyze the number of solutions to equation (*) with parameters $a,b$, i.e, let $A=a^{2/3}+b^{2/3}-1$, we have:


*

*$A<0$, there are four distinct solutions.

*$A>0$, there are two distinct solutions.


Endeavors
Let $f(x)=a\sin x+b\cos x-\cos x\sin x$, we have $f^\prime(x)=a\cos x-b\sin x-\cos2x$. It seems no advance to calculate the derivative, because $f^\prime$ is as hard as $f$.
Let $u=\cos x$ and $v=\sin x$, we have $u^2+v^2=1$ and $av+bu=uv$. We can work on these equations, but I prefer the trigonometric way, i.e, analyze the properties of $f(x)$.
I want to illustrate some details about $f(x)$, which might be useful. Let $a=r\cos\phi$ and $b=r\sin\phi$, where $r=\sqrt{a^2+b^2}$, we have
$f(x)=r\sin(x+\phi)-\frac12\sin2x$. It's a linear combination of $\sin(x+\phi)$ and $\sin2x$. I don't know whether there's a systematical way to deal with it.
Any idea? Thanks!
 [/INST] Let us consider only four cases in our analysis. The other missing cases can be analyzed by a similar approach.
Let $A$ a real number such that $A=a^\frac{2}{3}+b^\frac{2}{3}-1$ and $g(x)= a\sin x + b\cos x-\cos x\sin x$ a function with domain $[0, 2\pi[$ , let's analyze the number of zeros of $g(x)$ in function of $A$.
(1) Case ($a=0$ and $b \neq 0$)
$$g(x)=0 \Leftrightarrow b\cos x-\cos x\sin x = 0$$
$$ \Leftrightarrow \cos x(b-\sin x)=0$$
For $b^2>1$ we have $A>0$ and two distinct solutions.
For $b^2<1$ we have $A<0$ and four distinct solutions.
For $b^2=1$ we have $A=0$ and a double root and a simple one.
(2) Case ($a \neq 0$ and $b = 0$)
$$g(x)=0 \Leftrightarrow a\sin x-\cos x\sin x=0$$
$$\Leftrightarrow \sin x(a-\cos x)=0$$
For $a^2>1$ we have $A>0$ and two distinct solutions.
For $a^2<1$ we have $A<0$ and four distinct solutions.
For $a^2=1$ we have $A=0$ and a double root and a simple one.
(3) Case ($a=0$ and $b = 0$)
$$g(x)=0 \Leftrightarrow -\cos x\sin x=0$$
We have $A<0$ and the equation has four distinct solutions.
(4) Case ($a>0$ and $b>0$)
$$g(x)=0 \Leftrightarrow a\sin x + b\cos x-\cos x \sin x=0$$
For the equation above $0$, $\frac{\pi}{2}$, $\pi$, and $\frac{3\pi}{2}$ are never solutions (Verify by yourself).
Let's make the following factorising:
$$g(x)=(a\tan x + b - \sin x)\cos x$$
And let's define $f(x)=a\tan x + b - \sin x$.
Every root of $f(x)$ is a root of $g(x)$.
So let's search for roots of $f(x)$.
$$f(x)=0 \Leftrightarrow a\tan x + b= \sin x$$
If we change the values of $a$ and $b$ ($a>0$ and $b>0$) the graph of $a\tan x+b$ will always cross the graph of $\sin x$ once in the intervals $[\frac{\pi}{2}, \frac{3\pi}{2}]$ and $[\frac{3\pi}{2}, 2\pi[$. So $f(x)$ has at least two distinct roots in $[0, 2\pi[$.
See the graph below:

For $x \in [0, \frac{\pi}{2}]$ we must analyze the values of $a$ and $b$.
The local minimum of $f(x)$ in $[0, \frac{\pi}{2}]$ occurs at $\cos x_0 = a^\frac{1}{3}$ i.e. $x_0= \arccos a^\frac{1}{3}$.
If we substitute $x_0$ in $f(x)$ we get:
$$f(x_0)= \frac{a \sin (\arccos a^\frac{1}{3})}{a^\frac{1}{3}} + b - \sin (\arccos a^\frac{1}{3})$$
After some algebra we get:
$$f(x_0) = -(1-a^\frac{2}{3})^\frac{3}{2} + b$$
Now let's analyze the possibilities.
Note that
$$ f(x_0) >0 \Leftrightarrow -(1-a^\frac{2}{3})^\frac{3}{2} + b>0 \Leftrightarrow  b>(1-a^\frac{2}{3})^\frac{3}{2}$$
$$\Leftrightarrow a^\frac{2}{3}+ b^\frac{2}{3} -1 >0 \Leftrightarrow A >0.$$
So
If $f(x_0) > 0$ ($A>0$), then $a\tan x +b$ and $\sin x$ have no intersection ($f(x_0) >0$), $f(x)$ has only two distinct roots in $[0, 2\pi[$ and so do $g(x)$.
If $f(x_0) =0$ ($A = 0$), then $a\tan x +b$ and $\sin x$ are tangent at $x_0$, $f(x)$ has a double root  in $[0,\frac{\pi}{2}]$ (a double root and two distinct ones in $[0, 2\pi[$) and so do $g(x)$.
If $f(x_0)<0$ ($ A < 0)$, then $a\tan x +b$ and $\sin x$ have two intersections, $f(x)$ has only two distinct roots in $[0,\frac{\pi}{2}]$ (four distinct roots in $[0, 2\pi[$) and so do $g(x)$.
</s>