<s>[INST] Determining axis of rotation from the rotation matrix without using eigenvalues and eigenvectors 
Consider the following rotation matrix:
$$R=\begin{pmatrix} -\frac{1}{\sqrt{2}}&0&\frac{1}{\sqrt{2}} \\
 \sqrt{\frac{3}{8}} &\frac{1}{2}& \sqrt{\frac{3}{8}} \\
 -\frac{1}{\sqrt{8}}&\sqrt{\frac{3}{4}}&-\frac{1}{\sqrt{8}}\end{pmatrix}$$
Determine the angle and the axis of rotation by using the following
   equations:
  $$T=R+R^T-[\text{Tr}R-1]I\\\cos{\varphi}=\frac{1}{2}(\text{Tr}R-1)$$
Where $\text{Tr}R$ is the trace of $R$.

This is what I have done so far:
$$R+R^T-(\text{Tr}R-1)I=\begin{pmatrix} -\frac{1}{\sqrt{2}}&0&\frac{1}{\sqrt{2}} \\
 \sqrt{\frac{3}{8}} &\frac{1}{2}& \sqrt{\frac{3}{8}} \\
 -\frac{1}{\sqrt{8}}&\sqrt{\frac{3}{4}}&-\frac{1}{\sqrt{8}}\end{pmatrix}+\begin{pmatrix}-\frac{1}{\sqrt{2}}&\sqrt{\frac{3}{8}}&-\frac{1}{\sqrt{8}} \\ 0&\frac{1}{2}&\sqrt{\frac{3}{4}} \\ \frac{1}{\sqrt{2}} & \sqrt{\frac{3}{8}}&-\frac{1}{\sqrt{8}}\end{pmatrix}-\left (-\frac{1}{\sqrt{2}}+\frac{1}{2}-\frac{1}{\sqrt{8}}-1\right )I \\ $$
$$=\begin{pmatrix} -\frac{2}{\sqrt{2}}&\sqrt{\frac{3}{8}}& \frac{1}{2\sqrt{2}} \\ \sqrt{\frac{3}{8}}&1& \frac{\sqrt{2}\sqrt{3}+\sqrt{3}}{2\sqrt{2}} \\\frac{1}{2\sqrt{2}}&\frac{\sqrt{2}\sqrt{3}+\sqrt{3}}{2\sqrt{2}}&-\frac{2}{\sqrt{8}}\end{pmatrix}=...?$$
How do I determine the axis of rotation from this matrix?
 [/INST] Consider Rodrigues' formula for a rotation matrix:
$$
R=I+\sin\varphi K+(1-\cos\varphi)K^2
$$
with
$$
K=\pmatrix{
0&-k_3&k_2\\
k_3&0&-k_1\\
-k_2&k_1&0
}\;,
$$
where $k$ is the (unit) rotation axis.
To my mind it seems easier to just take the antisymmetric part of $R$, which is $\sin\varphi K$, and normalise to get rid of the $\sin\varphi$, but your lecture notes apparently take the other option and extract $K$ from the symmetric part:
$$
R+R^\top=2\left(I+(1-\cos\varphi)K^2\right)
$$
and thus
\begin{align}
R+R^\top-(\operatorname{Tr}R-1)I
&=2\left(I+(1-\cos\varphi)K^2\right)-(\operatorname{Tr}R-1)I\\
&=2\left(I+(1-\cos\varphi)K^2\right)-2\cos\varphi I\\
&=2(1-\cos\varphi)(K^2+I)\;.
\end{align}
$K^2$ has $-k_2^2-k_3^2$, $-k_3^2-k_1^2$ and $-k_1^2-k_2^2$ on the diagonal, so $K^2+I$ has $k_1^2$, $k_2^2$ and $k_3^2$ on the diagonal.
It follows that you must have made a mistake, as the diagonal elements should all have the same sign. Indeed it seems you forgot to subtract the multiple of the identity in the last line.
This doesn't give you the signs of the $k_i$; you're going to have to extract them from the other components of $K^2$, or use the antisymmetric part after all.
</s>