<s>[INST] Evaluate $\int_{y=-\infty}^1\int_{x=0}^\infty 4xy\sqrt{x^2+y^2}\mathrm dx\mathrm dy$ Evaluate $$\int_{y=-\infty}^1\int_{x=0}^\infty 4xy\sqrt{x^2+y^2}\mathrm dx\mathrm dy$$
I tried to solve this for hours without any success. I tried substitution method of $u = x^2+y^2$ but doesn't seem to work.
Here is my attempt:

$$\int_{-\infty}^1\int_0^\infty (16x^2y^2)^{\frac{1}{2}}(x^2 + y^2)^\frac{1}{2} \; \mathrm{d}x \mathrm{d}{y} = \int_{-\infty}^1 \int_0^\infty (16x^3y^2 + 16x^2 y^3)^{\frac{1}{2}} \; \mathrm{d}x\mathrm{d}{y}.$$
Let
\begin{align}
u &= 16x^3 y^2 + 16x^2 y^3 \\
\mathrm{d}u &= (48x^2y^2 + 32xy^3) \; \mathrm{d}{x} \\
\mathrm{d}u &= 16xy^2(3x + 2y) \; \mathrm{d}{x} \\
\mathrm{d}u &= 4xy(12xy + 8y^2) \; \mathrm{d}{x} \\
4xy \; \mathrm{d}{x} &= \frac{\mathrm{d}{u}}{12xy + 8y^2}.
\end{align}
$$\int_{-\infty}^1 \int_0^\infty 4xy \sqrt{x^2 + y^2} \; \mathrm{d}x \mathrm{d}y = \int_{-\infty}^1 \int_0^\infty \frac{\mathrm{d}u}{(12xy + 8y^2)}.$$

Attaching image for reference


EDITS
Original Post asked integrate: 4xy sqrt(x^2+y^2) where 0<x, y<1, and added this image. .
Later edits misread 0<x,y<1 as if 0<x<1and 0<y<1
 [/INST] You first integrate with respect to $x$ while treating $y$ as a constant. Then, you integrate the result of that with respect to $y$:
$$
\int_{0}^{1}\int_{0}^{1} 4xy\sqrt{x^2+y^2}\,dx\,dy=
\int_{0}^{1}\left(\frac{4}{2}y\int_{0}^{1}\sqrt{x^2+y^2}\frac{d}{dx}(x^2+y^2)\,dx\right)\,dy=
\int_{0}^{1}\left(2y\int_{0}^{1}\sqrt{x^2+y^2}\,d(x^2+y^2)\right)\,dy=\\
\int_{0}^{1}\left(2y\frac{2\sqrt{(x^2+y^2)^3}}{3}\bigg|_{0}^{1}\right)\,dy=
\int_{0}^{1}\left[2y\left(\frac{2\sqrt{(1^2+y^2)^3}}{3}-\frac{2\sqrt{(0^2+y^2)^3}}{3}\right)\right]\,dy=\\
\frac{4}{3}\int_{0}^{1}\left(y\sqrt{(1+y^2)^3}-y^4\right)\,dy=\\
\frac{4}{3}\int_{0}^{1}y\sqrt{(1+y^2)^3}\,dy-\frac{4}{3}\int_{0}^{1}y^4\,dy=\\
\frac{4/3}{2}\int_{0}^{1}\sqrt{(1+y^2)^3}\frac{d}{dy}(1+y^2)\,dy-\frac{4}{3}\frac{y^5}{5}\bigg|_{0}^{1}=\\
\frac{2}{3}\int_{0}^{1}(1+y^2)^{3/2}\,d(1+y^2)-\frac{4}{3}\left(\frac{1^5}{5}-\frac{0^5}{5}\right)=\\
\frac{2}{3}\frac{2(1+y^2)^{5/2}}{5}\bigg|_{0}^{1}-\frac{4}{15}=\\
\frac{2}{3}\left(\frac{2(1+1^2)^{5/2}}{5}-\frac{2(1+0^2)^{5/2}}{5}\right)-\frac{4}{15}=\\
\frac{2}{3}\left(\frac{2\sqrt{2^5}}{5}-\frac{2}{5}\right)-\frac{4}{15}=\\
\frac{4(4\sqrt{2}-1)}{15}-\frac{4}{15}=\frac{16\sqrt{2}-4-4}{15}=\frac{16(\sqrt{2}-8)}{15}=\frac{8(2\sqrt{2}-1)}{15}.
$$
Wolfram Alpha check.
</s>