<s>[INST] Show that transformation returns a hyperbola Given graph of $f(x)=1/x$, i.e. $G=\{(x,y)^t:xy=1\}$, I want to show that rotation of this grph by 45° clockwise gives us a hyperbolic graph.
This means I need to come to something of form $\{(x,y)^t: {x^2 \over a^2}-{y^2\over b^2}=1\}$. How do I calculate $a$ and $b$?
I wanted to roteate the original graph for 315° counter-clockwise (positive direction) using rotation matrix $A=\frac{1}{\sqrt{2}}\pmatrix{1 & 1 \\ -1 & 1}$, but don't know how to proceed.
 [/INST] You're on the right track. Notice that you can parametrize the the original curve as $\left(x, 1/x\right)$. Now just apply the transformation to get
$$
\frac{1}{\sqrt{2}}
\begin{pmatrix}
1 & 1 \\
-1 & 1
\end{pmatrix}
\begin{pmatrix}
x \\
1/x
\end{pmatrix}\\
=
\frac{1}{\sqrt{2}}
\begin{pmatrix}
\frac{1 + x^2}{x}\\
\frac{1 - x^2}{x}
\end{pmatrix}.
$$
This is the parametric equation of the new hyperbola. Put this into the hyperbola equation to get
$$
\frac{1}{2a^2}\left(\frac{x^4 + 2x^2 + 1}{x^2}\right) - \frac{1}{2b^2}\left(\frac{x^4 - 2x^2 + 1}{x^2}\right) = 1 \\
\implies b^2(x^4 + 2x^2 + 1) - a^2(x^4 - 2x^2 + 1) = 2a^2b^2x^2.
$$
Compare the terms of the two polynomials on both sides to find $a$ and $b$.
</s>