<s>[INST] How to evaluate this nonelementary integral? Let $x>0$. I have to prove that
$$
\int_{0}^{\infty}\frac{\cos x}{x^p}dx=\frac{\pi}{2\Gamma(p)\cos(p\frac{\pi}{2})}\tag{1}
$$
by converting the integral on the left side to a double integral using the expression below:
$$
\frac{1}{x^p}=\frac{1}{\Gamma(p)}\int_{0}^{\infty}e^{-xt}t^{p-1}dt\tag{2}
$$
By plugging $(2)$ into $(1)$ I get the following double integral:
$$
\frac{1}{\Gamma(p)}\int_{0}^{\infty}\int_{0}^{\infty}e^{-xt}t^{p-1}\cos xdtdx\tag{3}
$$
However, I unable to proceed any further as I am unclear as to what method should I use in order to compute this integral. I thought that an appropriate change of variables could transform it into a product of two gamma functions but I cannot see how that would work. Any help would be greatly appreciated.
 [/INST] So let us follow your initial line of thought and convert the integral to a double integral. As you correctly observe, as
$$\frac{1}{x^p} = \frac{1}{\Gamma (p)} \int_0^\infty e^{-xt} t^{p - 1} \, dt,$$
which, by the way, is just the Laplace transform for the function $x^{p -1}$, as a double integral your integral can be rewritten as
$$\int_0^\infty \frac{\cos x}{x^p} \, dx = \frac{1}{\Gamma (p)} \int_0^\infty \int_0^\infty e^{-xt} \cos x t^{p - 1} \, dt \, dx,$$
or
$$\int_0^\infty \frac{\cos x}{x^p} \, dx = \frac{1}{\Gamma (p)} \int_0^\infty t^{p - 1} \int_0^\infty e^{-xt} \cos x \, dx \, dt,$$
after changing the order of integration.
The inner $x$-integral can be readily found. Either using integration by parts twice, or recognising the integral as the Laplace transform for the function $\cos x$, as
$$\int_0^\infty e^{-xt} \cos x \, dx = \frac{t}{1 + t^2},$$
we have
$$\int_0^\infty \frac{\cos x}{x^p} \, dx = \frac{1}{\Gamma (p)} \int_0^\infty \frac{t^p}{1 + t^2} \, dt.$$
Enforcing a substitution of $t \mapsto \sqrt{t}$ leads to
\begin{align}
\int_0^\infty \frac{\cos x}{x^p} \, dx &= \frac{1}{2 \Gamma (p)} \int_0^\infty \frac{t^{\frac{p}{2} - \frac{1}{2}}}{1 + t} \, dt = \frac{1}{2 \Gamma (p)} \int_0^\infty \frac{t^{\frac{p + 1}{2} - 1}}{(1 + t)^{\frac{p + 1}{2} + \frac{1 - p}{2}}}.
\end{align}
As this is exactly of the form of the Beta function (see the second of the integral representations in the link) we have
\begin{align}
\int_0^\infty \frac{\cos x}{x^p} \, dx &= \frac{1}{2 \Gamma (p)} \text{B} \left (\frac{p + 1}{2}, \frac{1 - p}{2} \right )\\
&= \frac{1}{2 \Gamma (p)} \Gamma \left (\frac{p}{2} + \frac{1}{2} \right ) \Gamma \left (\frac{1}{2} - \frac{p}{2} \right ) \\
&= \frac{1}{2 \Gamma (p)} \Gamma \left (\frac{p}{2} + \frac{1}{2} \right ) \Gamma \left [1 - \left (\frac{p}{2} + \frac{1}{2} \right ) \right ] \\
&= \frac{1}{2 \Gamma (p)} \frac{\pi}{\sin (p + 1)\pi/2} \qquad (*)\\
&= \frac{\pi}{2 \Gamma (p) \cos \left (\frac{\pi p}{2} \right )},
\end{align}
as required. Note Euler's reflection formula was used in ($*$).  
</s>