<s>[INST] What is the 4-dimensional matrix representation of rotation operator? The rotation operator is $$\exp\left(-i\frac{\theta}{2}\boldsymbol{J}\cdot\hat{\boldsymbol{n}}\right).$$ 


*

*If $\boldsymbol{\sigma}$ is the Pauli matrix, the operator can be written as a matrix form $$\boldsymbol{1}\cos(\phi/2)-i\boldsymbol{\sigma}\cdot\hat{\boldsymbol{n}}\sin(\phi/2).$$ 

*But when $J$ is the spin-3/2 operator, $J$ is 4-dimensional. Is there a matrix representation of operator $\exp\left(-i\frac{\theta}{2}\boldsymbol{J}\cdot\hat{\boldsymbol{n}}\right)$? I find that when $\{J_x,J_y\}\neq0$ for spin-3/2, not like Pauli matrices. 

*What is the case when $J$ is spin-1 operator?
 [/INST] There are several ways to proceed.  
First, be aware that Mathematica has a built-in function called WignerD
and this function will give you the matrix element of a rotation matrix.  I have found that this function does not seem to use the same parametrization as Varshalovich, Dmitriĭ Aleksandrovich, Anatolij Nikolaevič Moskalev, and Valerii Kel'manovich Khersonskii. Quantum theory of angular momentum. 1988., which in my opinion remains the bible.  It seems you need to use the negative of all the angles in WignerD  to obtain the formulas of Varshalovich.
There are various closed form expressions for 
\begin{align}
d^j_{mm'}(\beta)&:= \langle jm\vert R_y(\beta)\vert jm'\rangle \, 
\end{align}
such as
\begin{align}
d^j_{mm'}(\beta)&=(-1)^{j-m'} 
\sqrt{(j+m)!(j-m)!(j+m')!(j-m')!}\\
&\times \sum_k (-1)^k
\frac{\left(\cos\frac{1}{2}\beta\right)^{m+m'+2k}
\left(\sin\frac{1}{2}\beta\right)^{2j-m-m'-2k}}
{k!(j-m-k)!(j-m'-k)!(m+m'+k)!}\, .
\end{align}
There is also an expression in terms of Jacobi polynomials:
\begin{align}
d^j_{mm'}(\beta)&=\xi_{mm'} 
\sqrt{\frac{s!(s+\mu+\nu)!}{(s+\mu)!(s+\nu)!}}
\left(\sin\textstyle\frac{1}{2}\beta\right)^\mu
\left(\cos\textstyle\frac{1}{2}\beta\right)^\nu
P_s^{\mu,\nu}(\cos\beta)
\end{align}
with 
$$
\mu=\vert m-m'\vert\, ,\quad
\nu=\vert m+m'\vert\, ,\quad
s=j-\frac{1}{2}(\mu+\nu)
$$
and the phase 
$$
\xi_{mm'}=\left\{\begin{array}{cc}
1&\quad \hbox{if } m'\ge m\, \\
(-1)^{m'-m}&\quad \hbox{if } m'<m\, .\end{array}\right.
$$
Finally, there is a method based on recursion relations.  This is described in details in Wolters, G. F. "Simple method for the explicit calculation of d-functions." Nuclear Physics B 18.2 (1970): 625-653. Starting with 
$$
\langle jm\vert R_y(\beta) L_x\vert jm'\rangle
$$
one can obtain a recursion relation 
\begin{align}
&\sqrt{(j-m')(j+m'+1)}d^j_{m,m'+1}(\beta)
+\sqrt{(j+m')(j-m'+1)}d^j_{m,m'-1}(\beta)\\
&\qquad = 2\hbox{cosec}(\beta)(m'\cos\beta-m)d^j_{mm'}(\beta)
\end{align}
The function
\begin{align}
d^j_{mj}(\beta)&={2j \choose j+m}^{1/2}
\left(\cos\textstyle\frac{1}{2}\beta\right)^{j+m}
\left(\sin\textstyle\frac{1}{2}\beta\right)^{j-m}\, ,\\
\end{align}
can be used as a seed for the recursion.
As a matrix with elements $d^{3/2}_{mm'}(\beta)$, the explicit results for $j=3/2$ is
\begin{align}
&R_y(\beta)=\\
&{\scriptsize\left(
\begin{array}{cccc}
 \cos ^3\left(\frac{\beta }{2}\right) & -\sqrt{3} \cos ^2\left(\frac{\beta }{2}\right) \sin \left(\frac{\beta }{2}\right) & \sqrt{3} \cos \left(\frac{\beta
   }{2}\right) \sin ^2\left(\frac{\beta }{2}\right) & -\sin ^3\left(\frac{\beta }{2}\right) \\
 \sqrt{3} \cos ^2\left(\frac{\beta }{2}\right) \sin \left(\frac{\beta }{2}\right) & \frac{1}{2} \cos \left(\frac{\beta }{2}\right) (3 \cos (\beta )-1) &
   -\frac{1}{2} (3 \cos (\beta )+1) \sin \left(\frac{\beta }{2}\right) & \sqrt{3} \cos \left(\frac{\beta }{2}\right) \sin ^2\left(\frac{\beta }{2}\right) \\
 \sqrt{3} \cos \left(\frac{\beta }{2}\right) \sin ^2\left(\frac{\beta }{2}\right) & \frac{1}{2} (3 \cos (\beta )+1) \sin \left(\frac{\beta }{2}\right) &
   \frac{1}{2} \cos \left(\frac{\beta }{2}\right) (3 \cos (\beta )-1) & -\sqrt{3} \cos ^2\left(\frac{\beta }{2}\right) \sin \left(\frac{\beta }{2}\right) \\
 \sin ^3\left(\frac{\beta }{2}\right) & \sqrt{3} \cos \left(\frac{\beta }{2}\right) \sin ^2\left(\frac{\beta }{2}\right) & \sqrt{3} \cos ^2\left(\frac{\beta
   }{2}\right) \sin \left(\frac{\beta }{2}\right) & \cos ^3\left(\frac{\beta }{2}\right) \\
\end{array}
\right)}
\end{align}
with the columns and rows ordered as $3/2,1/2,-1/2,-3/2$.
</s>