<s>[INST] How to evaluate this integral $\int_0^1 \frac{x\log ^2(\sqrt{x^2+1}+1)}{\sqrt{1-x^2}} \, dx$ How to evaluate $$A=\int_0^1 \frac{x \log ^2\left(\sqrt{x^2+1}+1\right)}{\sqrt{1-x^2}} \, dx$$See the details here from a similar question: Evaluating $\int_0^1 \frac{z \log ^2\left(\sqrt{z^2+1}-1\right)}{\sqrt{1-z^2}} \, dz$.
 By applying the same process, I find:
 $$A=\int_0^1\ln^2\left(\sqrt{2-y^2}+1\right)dy=\int_1^{\sqrt2}\frac {x\log^2 (x+1)}{\sqrt{2-x^2}}dx=\sqrt2 \int_{0}^{\pi/4}\cos\varphi\ln^2\left(\sqrt2 \cos\varphi+1\right)d\varphi.$$
$$A=\ln^22+32(1+\sqrt2)\int_{0}^{\sqrt2-1}\frac{t^2}{\left(1+t^2\right)^2(t+1+\sqrt2)(-t+1+\sqrt2)}\ln\left(\sqrt2\, \frac{1-t^2}{1+t^2}+1\right)dt.$$
 Now,put $$A=\int_0^1 \frac{x \log ^2\left(\sqrt{x^2+1}+1\right)}{\sqrt{1-x^2}} \, dx,  B=\int_0^1 \frac{x \log ^2\left(\sqrt{x^2+1}-1\right)}{\sqrt{1-x^2}} \, dx.$$
We have:
$$A-B=4\int_0^{1}\frac {x\log x\log(\sqrt{1+x^2}+1)}{\sqrt{1-x^2}}dx-4\int_0^{1}\frac {x\log^2 x}{\sqrt{1-x^2}}dx$$
$$A+B=-2\int_0^{1}\frac {x\log (\sqrt{1+x^2}-1)\log(\sqrt{1+x^2}+1)}{\sqrt{1-x^2}}dx+4\int_0^{1}\frac {x\log^2 x}{\sqrt{1-x^2}}dx$$
 $$\int_0^{1}\frac {x\log^2 x}{\sqrt{1-x^2}}dx=\ln^22-\frac{\pi^2}{12}-2\ln2+2.$$
But how to deduce these integrals.(using Mathematica?)
 [/INST] I can at least get you going on the last one. Let
$$I(p)=\int_0^1 \frac{x^p}{\sqrt{1-x^2}}dx$$
Then we have
$$\begin{align}
I(p)
&=\int_0^1 \frac{x^p}{\sqrt{1-x^2}}dx\\
&=\frac{1}{2}\int_0^1 \frac{x^{\frac{p-1}{2}}}{\sqrt{1-x}}dx\\
&=\frac{1}{2}\mathrm{B}\bigg(\frac{p+1}{2},\frac{1}{2}\bigg)\\
&=\frac{\sqrt{\pi}}{2}\frac{\Gamma(\frac{p+1}{2})}{\Gamma(\frac{p+2}{2})}\\
\end{align}$$
where $\mathrm{B}$ is the Beta function and $\Gamma$ is the Gamma function. This gives us
$$I''(1)=\int_0^1 \frac{x\ln^2(x)}{\sqrt{1-x^2}}dx=\frac{\sqrt{\pi}}{2} \frac{d^2}{dp^2}\frac{\Gamma(\frac{p+1}{2})}{\Gamma(\frac{p+2}{2})}$$
You can easily finish off this double derivative by brute force (or Mathematica, as you proposed) using a special value of the Digamma function.
</s>