<s>[INST] Closed form for $\int_0^\alpha \int_0^{\alpha-x} \frac{1}{[(x+y)(\beta-x)(1-\beta-y)]^{3/2}} \,dy \,dx$ Let $0\leq \alpha \leq \frac{1}{2},$ and let $\beta$ be such that $\alpha \leq \beta \leq 1-\alpha$.
Define $\varphi(\alpha,\beta)$ to be the integral
$$\varphi(\alpha,\beta) = \int_0^\alpha \int_0^{\alpha-x} \frac 1 {[(x+y)(\beta-x)(1-\beta-y)]^{3/2}} \,dy\, dx.$$
It'd be nice to have some closed expression for this integral, but so far I've been unable to come up with one.
Mathematica starts to spit out some really large expressions involving lots of complex numbers, and I've been unable to convince it that the answer should be real.
Does anyone know of a substitution or other method that would yield a nicer expression?
 [/INST] Let $z = x+y$, we have
$$\varphi(\alpha,\beta) = \int_0^\alpha \frac{dz}{z^{3/2}}\left\{\int_0^z \frac{dy}{((\beta-z+y)(1-\beta-y))^{3/2}}\right\}$$
For fixed $z$, introduce variables $u, v$ such that $u = \beta -z + y$, $v = 1 - \beta-y$. It is easy to check
$$d\left(\frac{u-v}{\sqrt{uv}}\right)
= \left(\frac{2}{\sqrt{uv}} - \frac{(u-v)(v-u)}{2\sqrt{uv}^3}\right) dy
= \frac{(u+v)^2}{2\sqrt{uv}^3} dy = \frac{(1-z)^2}{2\sqrt{uv}^3} dy
$$
and hence
$$\begin{align}
\varphi(\alpha,\beta) 
=& 2 \int_0^\alpha \frac{dz}{z^{3/2}(1-z)^2} \left\{
\left[\frac{u-v}{\sqrt{uv}}\right]_{y=0}^{y=z}
\right\}\\
=& 2 \int_0^\alpha \frac{dz}{z^{3/2}(1-z)^2}\left\{
\frac{2\beta-1 +z}{\sqrt{\beta(1-\beta-z)}} - \frac{2\beta-1-z}{\sqrt{(1-\beta)(\beta-z)}}
\right\}\\
=& -4\frac{\partial}{\partial\beta} \int_0^\alpha \frac{dz}{z^{3/2}(1-z)^2}\left\{
\sqrt{\beta(1-\beta-z)} - \sqrt{(1-\beta)(\beta-z)}
\right\}\\
=& -4\frac{\partial}{\partial\beta}\left[
\sqrt{\beta(1-\beta)}
\left( F\left(\frac{1}{1-\beta},\alpha\right) - F\left(\frac{1}{\beta},\alpha\right) \right)
\right]
\end{align}
$$
where
$$F(\mu,z) = \int^z \frac{\sqrt{1-\mu t}}{t^{3/2}(1-t)^2} dt + \text{ constant }.$$
To evaluate $F(\mu,z)$, let us introduce new variables
$$s = \frac{t}{1-\mu t} \;\;\leftrightarrow\;\; t = \frac{s}{1+\mu s}
\quad\implies\quad ds = \frac{dt}{(1-\mu t)^2}
$$
and let $\nu = \mu - 1$. Notice $0 < \beta < 1$ implies both
$\frac{1}{1-\beta}$ and $\frac{1}{\beta}$ are bigger than $1$. This means in this problem,
$\nu$ is always positive. In terms of these variables, we can transform $F(\mu,z)$ as
$$
F(\mu,z) 
= \int^z \frac{dt}{(1-\mu t)^2} \sqrt{\frac{1-\mu t}{t}}\left(\frac{1-\mu t}{t}\right)^2 \frac{t}{(1-t)^2}
= \int \frac{ds}{\sqrt{s}} \frac{1+\mu s}{s(1+\nu s)^2}
$$
Notice 
$$\begin{align} \frac{1+\mu s}{s(1+\nu s)^2}
=& \frac{(1+\nu s) + s}{s(1+\nu s)^2} 
= \frac{1}{s(1+\nu s)} + \frac{1}{(1+\nu s)^2}
= \frac{1}{s} - \frac{\nu}{1+\nu s} + \frac{1}{(1+\nu s)^2}\\
=& \frac{1}{s} + \left(\frac{\partial}{\partial \nu} - 1 \right) \frac{\nu}{1+\nu s}
= \frac{1}{s} + \left(\nu \frac{\partial}{\partial \nu} + 1 - \nu\right)\frac{1}{1 + \nu s}
\end{align}$$
We get
$$\begin{align}
F(\mu, z) 
= & -\frac{2}{\sqrt{s}} + \left(\nu \frac{\partial}{\partial \nu} + 1 - \nu\right)
\int \frac{ds}{\sqrt{s}}\frac{1}{1+\nu s}\\
= & -\frac{2}{\sqrt{s}} + \left(\nu \frac{\partial}{\partial \nu} + 1 - \nu\right)
\left[\frac{2}{\sqrt{\nu}}\tan^{-1}\sqrt{\nu s}\right]\\
= & \left(\frac{s}{1+\nu s} - 2\right)\frac{1}{\sqrt{s}} + \frac{1-2\nu}{\sqrt{\nu}}\tan^{-1}\sqrt{\nu s}\\
= & \frac{3z-2}{\sqrt{z}(1-z)}\sqrt{1-\mu z} + \frac{3-2\mu}{\sqrt{\mu - 1}} 
\tan^{-1}\sqrt{\frac{(\mu - 1)z}{1-\mu z}}
\end{align}
$$
This leads to
$$\begin{align}
& \sqrt{\beta(1-\beta)}\left(
F(\frac{1}{1-\beta},\alpha) - F(\frac{1}{\beta},\alpha)\right)\\
= & \frac{3\alpha-2}{\sqrt{\alpha}(1-\alpha)}
\left(\sqrt{\beta(1-\beta-\alpha)} - \sqrt{(1-\beta)(\beta-\alpha)}\right)\\
&+(1-3\beta)\tan^{-1}\sqrt{\frac{\beta \alpha}{1-\beta-\alpha}}
-(3\beta-2)\tan^{-1}\sqrt{\frac{(1-\beta)\alpha}{\beta-\alpha}}
\end{align}$$
and as a result,
$$\begin{align}
\varphi(\alpha,\beta) 
= &
2\left\{
\frac{3\alpha-2}{\sqrt{\alpha}(1-\alpha)}\left(
 \frac{2\beta-1+\alpha}{\sqrt{\beta(1-\beta-\alpha)}}
-\frac{2\beta-1-\alpha}{\sqrt{(1-\beta)(\beta-\alpha)}}
\right)
+6\left(
\tan^{-1}\sqrt{\frac{\beta \alpha}{1-\beta -\alpha}}
+\tan^{-1}\sqrt{\frac{(1-\beta)\alpha}{\beta-\alpha}}
\right)
-\sqrt{\alpha}
\left(
  \frac{1-3\beta}{(1-\beta)\sqrt{\beta(1-\beta-\alpha)}}
+ \frac{3\beta-2}{\beta\sqrt{(1-\beta)(\beta-\alpha)}}
\right)
\right\}\\
\end{align}$$
</s>