<s>[INST] Given ellipse of axes $a$ and $b$, find axes of tangential and concentric ellipse at angle $t$ Let’s say I have an ellipse with horizontal axis $a$ and vertical axis $b$, centered at $(0,0)$.
I want to compute $a’$ and $b’$ of a smaller ellipse centered at $(0,0)$, with the axes rotated by some angle $t$, tangent to the bigger ellipse and $\frac{a’}{b’}=\frac{a}{b}$.

 [/INST] Let the oblique ellipse be
$$
\frac{x^2}{a^2}+
\frac{y^2}{b^2}-1=k
\left(
  \frac{x\sin \theta}{a}-
  \frac{y\cos \theta}{b}
\right)^2$$
which touches the standard ellipse at $(a\cos \theta,b\sin \theta)$.
This can be easily verified by plugging the point $(x',y')=(a\cos \theta,b\sin \theta)$ in the both ellipses and also have the same tangent line by referring to the useful fact in the very last of the post.
By finding the eigenvalues $(\lambda_1 <\lambda_2)$ of the matrix $\mathbb{M}$, namely
\begin{align}
  \mathbb{M} &=
  \begin{pmatrix}
    A & \frac{B}{2} \\ \frac{B}{2} & C
  \end{pmatrix} \\
  &=
  \begin{pmatrix}
    \dfrac{1-k\sin^2 \theta}{a^2} & \dfrac{k\sin \theta \cos \theta}{ab} \\
    \dfrac{k\sin \theta \cos \theta}{ab} & \dfrac{1-k\cos^2 \theta}{b^2}
  \end{pmatrix}
\end{align}
Using Mathematica, we can get
$$\frac{\lambda_2}{\lambda_1}=\frac{a^2}{b^2} \implies
k=\frac{(a^4-b^4)\cos 2\theta}{(a^2\cos^2 \theta+b^2\sin^2 \theta)^2}$$
which gives smaller ellipse if $k<0 \implies \cos 2\theta<0$ providing $a>b$.
Due to similarity, the new semi-axes are scaled by the same factor which can be determined by the determinant of the matrix $\mathbb{M}$.
Now,
\begin{align}
  \lambda_1 \lambda_2 &= \det \mathbb{M} \\
  \frac{1}{a'^2 b'^2} &= AC-\frac{B^2}{4} \\
  &= \frac{1-k}{a^2 b^2} \\
  \frac{ab}{a'b'} &= \sqrt{1-k} \\
  &=
  \frac{a^2\sin^2 \theta+b^2\cos^2 \theta}
       {a^2\cos^2 \theta+b^2\sin^2 \theta} \\
  \frac{a'}{a} &= \frac{b'}{b} \\
  &=
  \sqrt{
    \frac{a^2\cos^2 \theta+b^2\sin^2 \theta}
         {a^2\sin^2 \theta+b^2\cos^2 \theta}}
\end{align}
Using the result of another answer of mine, the angle of rotation is given by
\begin{align*}
  t &= \tan^{-1}
  \left(
    \frac{C-A}{B} \pm \frac{\sqrt{(A-C)^{2}+B^{2}}}{B} \:
  \right) \\
  t_1 &= \tan^{-1}
  \left(
    -\frac{2ab}{a^2+b^2} \cot 2\theta
  \right) \tag{semi-axis $a$} \\
  t_2 &= \tan^{-1}
  \left(
    \frac{a^2+b^2}{2ab} \tan 2\theta
  \right) \tag{semi-axis $b$}
\end{align*}

In principle, $\theta$ can be solved in terms of $t$ explicitly.  The work is tedious and verified experimentally by using Mathematica.
\begin{align}
  C & \equiv 2ab\cos t \\
  S & \equiv (a^2+b^2)\sin t+\sqrt{(a^2+b^2)^2\sin^2 t+C^2} \\
  \sqrt{1-k} &= \frac{a^2 S^2+b^2 C^2}{a^2 C^2+b^2 S^2}
\end{align}
Finally by rotational transformation, the oblique ellipse can be expressed as
$$
\frac{(x\cos t+y\sin t)^2}{a^2}+
\frac{(y\cos t-x\sin t)^2}{b^2}=
\frac{a^2 C^2+b^2 S^2}{a^2 S^2+b^2 C^2}$$
Further application
Illustration of spiral galaxy


Useful fact:
Equation of tangent for conics $ax^2+2hxy+by^2+2gx+2fy+c=0$ at the point $(x',y')$ is given by
$$ax'x+h(y'x+x'y)+by'y+g(x+x')+f(y+y')+c=0$$

</s>