<s>[INST] Find the maximum of the value $\frac{\gcd(n,[n\sqrt{5}])}{\sqrt{n}}$ For any postive integer $n$ find the maximum of the value
$$\dfrac{\gcd(n,\left \lfloor{n\sqrt{5}}\right \rfloor)}{\sqrt{n}}$$
Let $m=[n\sqrt{5}]$,then we have $n\sqrt{5}-1<m\le n\sqrt{5}$,it is said consider pell equation?
 [/INST] Your inequality can be written as $$\sqrt5-\frac1n < \frac{m}n < \sqrt5 \implies \left| \sqrt5 - \frac{m}n \right| < \frac1n$$
so need good rational approximations to $\sqrt5$.  i.e, we need solutions to $5n^2-m^2=k$ for "small" positive $k$.  Now, what is "small"?  Note $\lfloor n\sqrt5 \rfloor = m \implies m^2 + k < (m+1)^2 \implies k < 2m+1$.  
Further note that if $g = \gcd(m, n)$, we may write $m = gx, n = gy, k= g^2z$ for some positive $z$.  As we seek maximum of $g^2/n = g/y \approx \sqrt5 g/x$, we need to keep $x, y$ smallest while having largest possible $g$.  So write the equation as
$$x^2- 5y^2 = - z$$
where $0< z < 2x/g +1/g^2 \implies gz \le 2x \implies g/x \le 2/z$.  Clearly largest value then for $g/x$ is when $z=1$, so we try
$x^2-5y^2=-1$ first, which has the smallest solution $(2, 1)$.  Now with $g = 4$, and we get the maximum when $m = gx = 8, n = gy = 4$ to give max $\dfrac{\gcd(4, \lfloor 4\sqrt5\rfloor)}{\sqrt4} = 2$.
</s>