<s>[INST] shortest distance between points avoiding eclipse What is the shortest path between two points (-3,0) and (3,0) that avoids the interior of the ellipse $\frac{x^2}{2}+\frac{y^2}{1}=1$
I should use the maximum principle for state constraints, but I have a problem with the part where the path touches ellipse on some time interval? I know it should be a straight line away from the obstacle.
 [/INST] The shortest path from the point $\ (-3,0)\ $ to the point $\ (3,0)\ $  within the set $\ \left\{(x,y)\,\left|\,x^2+2y^2\ge2\right.\right\}\ $ comprises a straight line from the point $\ (-3,0)\ $ to the ellipse, and tangent to it at the point where they meet, a segment of the ellipse from that point to its reflection in the $\ y$-axis, and the straight line from the latter point to the point $\ (3,0)\ $.  There are two such paths, illustrated in red and green in the diagram below. While this is intuitively obvious, and I'm sure it's true, I'm not aware of any succinct way of proving it.  If you're willing to accept it as established, however, here's a way of finding the coordinates of the points where the lines from $\ (-3,0)\ $ are tangent to the ellipse.
Let $\ (a,b)\ $ be the coordinates of the point where the red line from $\ (-3,0)\ $ meets the ellipse.  Then $\ b>0\ $, and the coordinates where the green line from $\ (-3,0)\ $ meets the ellipse will be $\ (a,-b)\ $, by symmetry.  For the red line to be tangent to the ellipse at $\ (a,b)\ $, its slope must be the same as that of the ellipse at that point. The slope of the line is
$$
\frac{b}{a+3}\ ,
$$
and the slope of the ellipse is its derivative $\ \frac{dy}{dx}\ $ at the point in question.  Differentiating the equation of the ellipse gives
$$
0=x+2y\frac{dy}{dx}=a+2b\frac{dy}{dx}\ ,\ \text{  or}\\
\frac{dy}{dx}=-\frac{a}{2b}
$$
at the point $\ (a,b)\ $.  Equating this to the slope of the line gives
\begin{align}
\frac{b}{a+3}&=-\frac{a}{2b}\ ,\ \text{  or}\\
2b^2&=-a^2-3a\\
&=2-a^2\ ,
\end{align}
because the point $\ (a,b)\ $ lies on the ellipse.  It follows from this last equation that $\ a=-\frac{2}{3}\ $, and then $\ b=\frac{\sqrt{7}}{3}\ $.
The equation of the line segment from the point $\ (-3,0)\ $ to the point $\ \left(-\frac{2}{3}, \frac{\sqrt{7}}{3}\right)\ $ is
$$
y=\frac{x+3}{\sqrt{7}}\ .
$$
The reflection of the point $\ \left(-\frac{2}{3}, \frac{\sqrt{7}}{3}\right)\ $ in the $\ y$-axis is $\ \left(\frac{2}{3}, \frac{\sqrt{7}}{3}\right)\ $, and the equation of the line segment from that point to the point $\ (3,0)\ $ is
$$
y=\frac{3-x}{\sqrt{7}}\ .
$$

As TonyK has pointed out in a comment, the length of the elliptical part of these paths cannot be expressed in terms of elementary functions of the coordinates of its end points and the parameters of the ellipse. Its length is given by
\begin{align}
\int_{-\frac{2}{3}}^\frac{2}{3}\sqrt{1+\left(\frac{dy}{dx}\right)^2}\,dx&=2\int_0^\frac{2}{3}\sqrt{1+\frac{x^2}{4y^2}}\,dx\\
&=2\int_0^\frac{2}{3}\sqrt{\frac{1-\frac{x^2}{4}}{1-\frac{x^2}{2}}}\,dx\\
&=2\int_0^\frac{2}{3}\sqrt{\frac{1-\frac{1}{2}\left(\frac{x}{\sqrt{2}}\right)^2}{1-\left(\frac{x}{\sqrt{2}}\right)^2}}\,dx\\
&=2\sqrt{2}\int_0^\frac{\sqrt{2}}{3}\sqrt{\frac{1-\frac{t^2}{2}}{1-t^2}}\,dt\\
&=2\sqrt{2}E\left(\arcsin\left(\frac{\sqrt{2}}{3}\right)\,\left|\,\frac{1}{\sqrt{2}}\right.\right)\ ,
\end{align}
where $\ E(\,\cdot\,|\,\cdot\,)\ $ is the incomplete elliptic integral of the second kind.  The length of the straight line from $\ (-3,0)\ $ to $\ \left(-\frac{2}{3}, \frac{\sqrt{7}}{3}\right)\ $ is
$$
\sqrt{\left(3-\frac{2}{3}\right)^2+\frac{7}{9}}=\frac{2\sqrt{14}}{3}\ ,
$$
so the shortest distance is
$$
\frac{4\sqrt{14}}{3}+2\sqrt{2}E\left(\arcsin\left(\frac{\sqrt{2}}{3}\right)\,\left|\,\frac{1}{\sqrt{2}}\right.\right)\ ,
$$
for which WolframAlfa gives the approximation
$$
6.33877741851728513700827575564\ .
$$
</s>