<s>[INST] Evaluate $P[X^2> Y^2]$ 
Let $X$ and $Y$ have the joint probability density function 
  $$f(x,y)= \begin{cases}cxye^{-(x^2+2y^2)}, & x>0,y>0 \\ 0, &\text{otherwise} \end{cases}$$ 
  Evaluate $c$ and $P[X^2> Y^2]$.

Trial: We know that 
\begin{align} 
c\int_{0}^{\infty}\int_{0}^{\infty}xye^{-(x^2+2y^2)}\,dx\,dy&=1\\ \implies c(\int_{0}^{\infty}xe^{-x^2}\,dx)(\int_{0}^{\infty}ye^{-2y^2})\,dy&=1 \\ \implies c &=8
\end{align}
Now $$P[X^2>Y^2]= \iint_{x^2>y^2} 8xye^{-(x^2+2y^2)}\,dx\,dy$$
 Then I try to solve it by polar transformation. But I can not find limits of $r$ and $\theta$.Mainly I want to evaluate $P[X^2>Y^2]$. Please help.
 [/INST] The transformation you want is $x=r \cos{\theta}$, $y= \frac{r}{\sqrt{2}} \sin{\theta}$, and $dx \, dy = \frac{r}{\sqrt{2}} dr \, d \theta$.  As far as the integration limits go, $x^2 > y^2$ implies $0 < \tan{\theta} < \sqrt{2}$, while  $r \in [0,\infty)$.
The integral then becomes
$$\frac{8}{2} \underbrace{\int_0^{\infty} dr \: r^3 e^{-r^2}}_{1/2} \underbrace{\int_0^{\arctan{\sqrt{2}}} d \theta \: \sin{\theta} \cos{\theta}}_{(1/2) \sin^2{\arctan{\sqrt{2}}}=(1/2)(2/3) = 1/3} = \frac{2}{3} $$
</s>