<s>[INST] A Sine integral: problem I Is it possible to demonstrate a solution for the integral
\begin{align}
\int_{0}^{\infty} x^{n} \, \sin\left( a x^{2} + \frac{b}{x^{2}} \right) \, dx 
\end{align}
 [/INST] Part 1
Lets assume that $n$ is even ($n=2m$), and the parameters are chosen that the integral converges. Call our integral of interest $J(a,b)$.
Now observe that 
\begin{align}x^{2m}\sin(ax^2+b/x^2)&= (-1)^{(m+3)/2}\frac{d^m}{da^m}\cos(ax^2+b/x^2),\qquad 
\text{if} \; m=1,3,5,\ldots
\\
x^{2m}\sin(ax^2+b/x^2)&= (-1)^{m/2}\frac{d^m}{da^m}\sin(ax^2+b/x^2),\qquad\;\,\quad
\text{if}\;m=2,4,6...
\end{align}
So essentially we only have to calculate 
$\displaystyle\int_0^{\infty}\cos(ax^2+b/x^2)\mathrm{d}x$ and $\displaystyle\int_0^{\infty}\sin(ax^2+b/x^2)\mathrm{d}x$.
This can be done as follows (we only do the sine integral the other cosine one goes along the same lines):


*

*First complete the square in the Integrands
\begin{align}
a/x^2+b/x^2= (\sqrt{a}x-\sqrt{b}/x)^2+2\sqrt{ab},
\end{align}

*Use the trigonometric identity 
\begin{align}
\sin(x+y)=\sin x\cos y+\cos x\sin y,
\end{align}
to obtain
\begin{align} 
_{e}I(a,b)&=\int_0^{\infty}\sin(ax^2+b/x^2)\mathrm{d}x \\
&= \cos(2\sqrt{ab})\int_0^{\infty}\sin(\sqrt{a}x-\sqrt{b}/x)^2\mathrm{d}x+\sin(2\sqrt{ab})\int_0^{\infty}\cos(\sqrt{a}x-\sqrt{b}/x)^2\mathrm{d}x
\end{align}


*Now we apply the inversion $\displaystyle x \rightarrow \frac{\sqrt{b}}{\sqrt{a}x}$ to both integrals and add them together with their non-inverted versions. We end up with 


\begin{align} 
2 _{e}I(a,b) &=
\cos(2\sqrt{ab})\int_0^{\infty}\sin(\sqrt{a}x-\sqrt{b}/x)^2 \left(1+\frac{\sqrt{b}}{\sqrt{a}x^2}\right)\mathrm{d}x \\
& \hspace{10mm} +\sin(2\sqrt{ab})\int_0^{\infty}\cos(\sqrt{a}x-\sqrt{b}/x)^2\left(1+\frac{\sqrt{b}}{\sqrt{a}x^2}\right)\mathrm{d}x.
\end{align}


*Substitute $\sqrt{a}x-\sqrt{b}/x=y$ and $\displaystyle\frac{dy}{\sqrt{a}}=\left(1+\frac{\sqrt{b}}{\sqrt{a} x^2}\right)\mathrm{d}x$ to get


\begin{align}
2 _{e}I(a,b) &=\frac{\cos(2\sqrt{ab})}{\sqrt{a}} \int_{-\infty}^{\infty} \sin y^2\mathrm{d}y + \frac{\sin(2\sqrt{ab})}{\sqrt{a}} \int_{-\infty}^{\infty}\cos y^2\mathrm{d}y \\
&= \frac{2 \, \cos(2\sqrt{ab})}{\sqrt{a}} \int_{0}^{\infty} \sin y^2\mathrm{d}y + \frac{2\, \sin(2\sqrt{ab})}{\sqrt{a}} \int_{0}^{\infty}\cos y^2 \mathrm{d}y.
\end{align}


*Use the well known values $\displaystyle\int_0^{\infty}\sin y^2\mathrm{d}y=\int_0^{\infty}\cos y^2\mathrm{d}y=\sqrt{\frac{\pi}{8}}$.


So in the end we have
\begin{align}
_{e}I(a,b) = \sqrt{\frac{\pi}{8 a}}\left(\cos(2\sqrt{ab})+\sin(2\sqrt{ab})\right)
= \frac{1}{2} \sqrt{\frac{\pi}{a}} \, \sin\left( 2 \sqrt{ab} + \frac{\pi}{4} \right).
\end{align}
The final result is now
\begin{align}
J(a,b)=\int_0^{\infty}x^{2m}\sin(ax^2+b/x^2)\mathrm{d}x=(-1)^{(m+3)/2}\frac{d^m}{da^m}_{e}I(a,b),
\end{align}
if $m=1,3,5\ldots$
Edit:
For $m=2,4,6\ldots$
we get
\begin{align}
_{e}K(a,b) &= \int_0^{\infty}\cos(ax^2+b/x^2)\mathrm{d}x = \sqrt{\frac{\pi}{8 a}}\left(\cos(2\sqrt{ab})-\sin(2\sqrt{ab})\right) \\
&= \frac{1}{2} \sqrt{ \frac{\pi}{a} } \, \cos\left( 2 \sqrt{ab} + \frac{\pi}{4} \right),
\end{align}
where we used
$\cos(x+y)=\cos x\cos y-\sin x \sin y$ and 
\begin{align}
J(a,b)=\int_0^{\infty}x^{2m}\cos(ax^2+b/x^2)\mathrm{d}x=(-1)^{m/2}\frac{d^m}{da^m}{_{e}K(a,b)}.
\end{align}
Part 2
I finally managed managed to get an answer for $n=2m+1$
We again observe the differentiation trick works again:
\begin{align}x^{2m+1}\sin(ax^2+b/x^2)&= (-1)^{(m+3)/2}\frac{d^m}{da^m}x\cos(ax^2+b/x^2),
\qquad \text{if}\; m=1,3,5\ldots
\\
x^{2m+1}\sin(ax^2+b/x^2)&= (-1)^{m/2}\frac{d^m}{da^m}x
\sin(ax^2+b/x^2),\qquad\quad\;\,
\text{if} \; m=2,4,6\ldots
\end{align}
So we only need $\displaystyle\int_0^{\infty}x\cos(ax^2+b/x^2)\mathrm{d}x$ and $\displaystyle\int_0^{\infty}x\sin(ax^2+b/x^2)\mathrm{d}x$. 
We do this as follows (only the sine case in detail...)


*

*Substitute $x\rightarrow\sqrt{y}$,$dx=dy/\sqrt{y}$ and luckily the intergal simplifies a lot: 
\begin{align}
_oI(a,b)=\int_0^{\infty}\sin(ay+b/y)\mathrm{d}y.
\end{align}

*Observe that our Integral can after rescaling $y=\sqrt\frac{b}{a}y$ be written as
\begin{align}
_oI(a,b)=-\sqrt{\frac{b}{a}}\Im\int_0^{\infty}e^{-\sqrt{ab}i(y+1/y)}\mathrm{d}y
\end{align}

*Referring to the the beautiful answers in this link and analytically (which is as far as I can see legitimate) continuing them to purely imaginary parameters ($a-\rightarrow ia$,$b\rightarrow ib$)
we can write
\begin{align}
_oI(a,b)=-2\sqrt{\frac{b}{a}}\Im K_1(2i\sqrt{ab})
\end{align}
where $K_1(z)$ is a modified Bessel function

*Using furthermore the Relations $K_1(z)=\frac{i \pi}{2}e^{i \pi/2}H_1(iz)$ and $H_1(z)=J_1(z)+iN_1(z)$, with $H_1(z)$ an Hankel function and $J_1(z)$/$N_1(z)$ Bessel functions of first/second kind, see here
 for example, we finally get
\begin{align}
_oI(a,b)=-\pi\sqrt{\frac{b}{a}}N_1(2\sqrt{ab})
\end{align}


*Repeating this procedure (with $\Re$ instead of $\Im$) for the Cosine case  we get 


\begin{align}
_oK(a,b)=-\pi\sqrt{\frac{b}{a}}J_1(2\sqrt{ab}).
\end{align}
The whole Integral is then given by 
\begin{align}
J(a,b)=(-1)^{(m+3)/2}\frac{d^m}{da^m}{_{o}I(a,b)},
\end{align}
if $m=1,3,5\ldots$ or
\begin{align}
J(a,b)=(-1)^{m/2}\frac{d^m}{da^m}{_{o}K(a,b)},
\end{align}
if $m=2,4,6\ldots$
</s>