<s>[INST] Rotation matrix from given axis Let $T\colon\mathbb{R^3}\to\mathbb{R^3}$ denotes linear transformation which rotates by $\frac{\pi}{3}$ counter-clockwise along the vector $u=(1,1,1)$.
If $T(0,1,0)=(a,b,c)$, Find $3a^2+b^2+c^2$.

My Attempt
Consider a plane which contains a point $(0,1,0)$ and uses $u$ as normal vector.
Then I got : $x+y+z=1$.
Since $T$ is rotation, $T(0,1,0)=(a,b,c)$ must be on the same plane $x+y+z=1$.
This means $a+b+c=1$.
Again, $T$ preserves norm of vector : $1=|(0,1,0)|=|T(a,b,c)|=a^2+b^2+c^2$.
So, I got two equations :
$$a+b+c=1 \\ a^2+b^2+c^2=1$$
I want to Find $a, b, c$ without finding exact form of $T$ but I need one more equation about $a,b,c$ to solve this system.
Is there any property of rotation which can give me one more equation about $a,b,c$?
 [/INST] The additional equation comes from the angle of rotation.
Set $A=(0,1,0)$ and $B=(a,b,c)$, then consider the point where the rotation axis intersects the given plane
$$
P_0 = \left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)
$$
and consider the two vectors
\begin{align}
u_A &= A-P_0 = \left(-\frac{1}{3},\frac{2}{3},-\frac{1}{3}\right) \\
u_B &= B-P_0 = \left(a-\frac{1}{3},b-\frac{1}{3},c-\frac{1}{3}\right) \\
\end{align}
Then
$$
\frac{u_A \cdot u_B}{\|u_A\|\|u_B\|} = \cos(\pi/3) = \frac{1}{2}
$$
so
$$
-\frac{a-2 b+c}{\sqrt{6 \left(a^2+b^2+c^2\right)-4 (a+b+c)+2}}=-\frac{1}{2}(a-2b+c) = \frac{1}{2}
$$
The system becomes
\begin{align}
& a+b+c=1 \\
& a^2+b^2+c^2 =1 \\
& a-2b+c= -1
\end{align}
and has two solutions. The correct one is determined by the counter-clockwise information, that could be formalized by the condition
$$
u_A \times u_B \cdot u > 0
$$
</s>