<s>[INST] Evaluating double integral on different domains $D$ $$\iint\limits_{D} \left(x^2+y^2\right)\mathrm{d}x \mathrm{d}y$$
where $D$ is given each time by
$D=x^2-y^2=1,\hspace{0.5cm} x^2-y^2=9,\hspace{0.5cm} xy=2,\hspace{0.5cm} xy=4$
I try to use Polar coordinate transformation
\begin{cases}
   x &= \rho \cos \theta  \\
   y &= \rho \sin \theta
\end{cases}
but I don’t know how to find the range of $\rho$ and $\theta$
 [/INST] $D$ is made of two disconnected but symmetric regions. $xy$ is positive, so either both $x,y$ are positive or they are both negative. Let $D_+$ be the part of $D$ in the first quadrant. The other part is mirrored in the fourth quadrant. The integrand $x^2+y^2$ is symmetric about the origin, so its integral over $D$ is exactly twice the integral over $D_+$.
Using polar coordinates:
First rewrite the boundaries of $D_+$.
$$x^2 - y^2 = a \implies r^2 \left(\cos^2(\theta) - \sin^2(\theta)\right) = a \implies r = \sqrt a \sqrt{\sec(2\theta)} \\
xy = b \implies r^2 \cos(\theta) \sin(\theta) = b \implies r = \sqrt{2b} \sqrt{\csc(2\theta)}$$
Split up $D_+$ into three regions:
$$D_+ = D_1 \cup D_2 \cup D_3 \\
D_1 = \left\{(r,\theta) \mid 2 \sqrt{\csc(2\theta)} \le r \le 3\sqrt{\sec(2\theta)} \text{ and } \theta_1 \le \theta \le \theta_2\right\} \\
D_2 = \left\{(r,\theta) \mid 2 \sqrt{\csc(2\theta)} \le r \le 2\sqrt2 \sqrt{\csc(2\theta)} \text{ and } \theta_2 \le \theta \le \theta_3\right\} \\
D_3 = \left\{(r,\theta) \mid \sqrt{\sec(2\theta)} \le r \le 2\sqrt2 \sqrt{\csc(2\theta)} \text{ and } \theta_3 \le \theta \le \theta_4\right\}$$
where $\theta_1,\theta_2,\theta_3,\theta_4$ are the values of $\theta$ corresponding to the vertices of $D_+$, represented by the rays in the plot in counter-clockwise order:

The specific coordinates are given in the table below.
$$\begin{array}{c|cr}
(x,y) & (r,\theta) \\
\hline
\left(\sqrt{\frac{9+\sqrt{97}}2}, \frac{\sqrt{97}-9}4 \sqrt{\frac{9+\sqrt{97}}2}\right) & \left(\sqrt[4]{97}, \tan^{-1}\left(\frac{\sqrt{97}-9}4\right)\right) & (\theta_1) \\
\left(\sqrt{\frac{9+\sqrt{145}}2}, \frac{\sqrt{145}-9}8 \sqrt{\frac{9+\sqrt{145}}2}\right) & \left(\sqrt[4]{145}, \tan^{-1}\left(\frac{\sqrt{145}-9}4\right)\right) & (\theta_2) \\
\left(\sqrt{\frac{1+\sqrt{17}}2}, \frac{\sqrt{17}-1}8 \sqrt{\frac{1+\sqrt{17}}2}\right) & \left(\sqrt[4]{17}, \tan^{-1}\left(\frac{\sqrt{17}-1}4\right)\right) & (\theta_3) \\
\left(\sqrt{\frac{1+\sqrt{65}}2}, \frac{\sqrt{65}-1}8 \sqrt{\frac{1+\sqrt{65}}2}\right) & \left(\sqrt[4]{65}, \tan^{-1}\left(\frac{\sqrt{65}-1}8\right)\right) & (\theta_4)
\end{array}$$
Then the integral is
$$\begin{align*}
\iint_D (x^2+y^2) \,dx\,dy &= 2 \iint_{D_+} r^3 \, dr \, d\theta \\[1ex]
&= 2 \left\{\int_{\theta=\theta_1}^{\theta_2} \int_{r=2\sqrt{\csc{2\theta}}}^{3\sqrt{\sec(2\theta)}} + \int_{\theta=\theta_2}^{\theta_3} \int_{r=2\sqrt{\csc{2\theta}}}^{2\sqrt2\sqrt{\csc(2\theta)}} + \int_{\theta=\theta_3}^{\theta_4} \int_{r=\sqrt{\sec{2\theta}}}^{2\sqrt2\sqrt{\csc(2\theta)}}\right\} r^3 \, dr \, d\theta
\end{align*}$$
Doable, yes, but you're much better off using Cartesian coordinates or making the substitution suggested in comments and the other answer.
</s>