<s>[INST] can not find root of the equation $(0,0)$ with semi-major axis a and semi-minor axis $b$">
I want find out for each point $(x,y)$ in the white region, the nearest point $(a \cos \theta, b \sin \theta )$ on the ellipse curve.I tried with the following approach.
$$
\text {distance} = \sqrt{(x-a \cos \theta)^2+(y-b \sin \theta)^2} 
$$
I find out the first derivative of the $$F(\theta)=(x-a \cos \theta)^2+(y-b \sin \theta)^2.$$ The equation after first derivative is: 
$$
ax \sec \theta- by \csc \theta=a^2-b^2.
$$
How to calculate theta for this equation?
 [/INST] You can get it by solving a constrained optimization problem. You want the point $(u,v)$ that minimizes $f(u,v)=(u-x)^2+(v-y)^2$ subject to the constraint $\frac{u^2}{a^2} + \frac{v^2}{b^2} = 1$. The point $(x,y)$ doesn't need to be inside the ellipse.
The Lagrangian is given by 
$$
L(u,v,\lambda) = (u-x)^2+(v-y)^2-\lambda\left(\frac{u^2}{a^2} + \frac{v^2}{b^2}-1\right)
$$
Now you just need to compute the critical points of $L$ and choose the ones that yield the smallest distance to $(x,y)$.
You can solve this by hand but, out of laziness, I give you Wolfram's solution... Two critical points
WARNING: Wolfram is not giving the full set of solutions. After computing $u,v$ in terms of $\lambda$ from the first two equations of the system $\nabla L = 0$ and substituting them in the last equation we get a fourth degree polynomial equation in $\lambda$. The solution must be conmputed in an alternative way if this polynomial has four real roots.
$$
\left( -\frac{a \sqrt{b^2-y^2}}{b}, \frac{-a^2 b^4 y+a^2 b^2 y^3+a b^3 x y \sqrt{b^2-y^2}+b^6 y-b^4
   y^3}{a^4 b^2-a^4 y^2-2 a^2 b^4-a^2 b^2 x^2+2 a^2 b^2 y^2+b^6-b^4
   y^2}\right)
$$
and 
$$
\left( \frac{a \sqrt{b^2-y^2}}{b}, \frac{-a^2 b^4 y+a^2 b^2 y^3-a b^3 x y \sqrt{b^2-y^2}+b^6 y-b^4
   y^3}{a^4 b^2-a^4 y^2-2 a^2 b^4-a^2 b^2 x^2+2 a^2 b^2 y^2+b^6-b^4
   y^2}\right)
$$
One will correspond to the minimum distance and the other will correspond to the maximum distance.
This is not relevant to your question, but the Lagrange multipliers are
$$
\frac{a^2 b^2-a^2 y^2+a b x \sqrt{b^2-y^2}}{b^2-y^2}
$$
and
$$
\frac{a^2 b^2-a^2 y^2-a b x \sqrt{b^2-y^2}}{b^2-y^2},
$$
respectively.
</s>