<s>[INST] Geometric transformation (symmetric point to line) I'm trying my own exercise: 

Let $(ε):y=3x-4$ be a line in $\Bbb R^2$ given in cartesian coordinates. Find the geometric transformation that sends every point in $\Bbb R^2$ to it's symmetrical as to the line $(ε)$.

My attempt:
Let $$\begin{array}\\f_1: \Bbb R^2 \rightarrow \Bbb R^2 \\ P \rightarrow f_1(P)=P'(x',y'): \\ x'=x\\y'=y-4 \end{array}$$(creates a new coordinate system $4$ units down the $y$-axis).
Thus, $(ε)$ in $O'x'y'$ will be $y'=3x'$
Let $$\begin{array}\\ f_2: \Bbb R^2 \rightarrow \Bbb R^2 \\ P' \rightarrow f_2(P')=P''(x'',y'') \end{array}$$ (rotation of the coordinate system by angle $θ$ clockwise)
$$\begin{pmatrix}x''\\ y''\end{pmatrix}=\begin{pmatrix}cos(\frac{\pi}{2}-arctan(3)) & sin(\frac{\pi}{2}-arctan(3)) \\ -sin(\frac{\pi}{2}-arctan(3)) & cos(\frac{\pi}{2}-arctan(3))\end{pmatrix}\begin{pmatrix}x'\\ y'\end{pmatrix}$$
So $$\begin{array}\\ f: \Bbb R^2 \rightarrow \Bbb R^2 \\ P \rightarrow f(P)=P''(x'',y''): \\ x''=xcosθ+(y-4)sinθ \\ y''=-xsin+(y-4)cosθ \end{array}$$ (the final transformation, where $θ=\frac{\pi}{2}-arctan(3)$).

So I've found the symmetric of the point $(1,4)$ using GeoGebra, which is $(4,3)$.
And I used this to test my answer but it is giving me a wrong result. I can't seem to find where the mistake is. Thanks in advance!
 [/INST] Let us look for a solution of the form:
$$\tag{1}\begin{pmatrix}x'\\ y'\end{pmatrix}=\begin{pmatrix}a & \ \ b \\b & -a\end{pmatrix}\begin{pmatrix}x\\ y\end{pmatrix}+\begin{pmatrix}e\\ f\end{pmatrix}$$
(see Edit 2 below)
(a symmetry matrix is orthogonal with determinant -1)
It suffices to write that points of intersection of the line with axes, i.e., 
$$\begin{pmatrix}x\\ y\end{pmatrix}=\begin{pmatrix}0\\ -4\end{pmatrix} \ \ \text{and} \ \ \begin{pmatrix}x\\ y\end{pmatrix}=\begin{pmatrix}-\tfrac{4}{3}\\ \ \ 0\end{pmatrix}$$
are invariant through this transformation, which gives, by plugging these expressions into (1), four equations with the 4 unknowns $a,b,c,d$:
$$\begin{cases}-4b+c&=&0\\ \ \  4a+d&=&-4\\ \ \ \tfrac{4}{3}a+c&=&\tfrac{4}{3}\\ \ \  \tfrac{4}{3}b+d&=&0\end{cases}$$
out of which we obtain: 

$$a=-4/5, b=3/5, c=12/5, d=-4/5.$$


Edit 1: Another solution (closer to what you have done).
We could have considered the succession of operations 
$$\tag{3}S = T^{-1} \circ R^{-1} \circ S' \circ R \circ T \ \ \ \text{where}$$


*

*T is the translation with vector $\binom{4}{0}$ bringing line with equation $y=3x-4$ onto line (D) with equation $y=3x$. 

*R is the rotation with angle -atan$(3)$ mapping (D) onto $x$ axis.
(we will denote by c and s resp. the cosine and sine of atan$(3)$):

*S' is the (very simple) symmetry with respect to the $x$ axis.
Let us now give (3) under a matricial form:
$$\binom{x'}{y'}=\binom{0}{-4}+\underbrace{\begin{pmatrix}c&-s\\s&c\end{pmatrix}\begin{pmatrix}1& \ \ 0\\0&-1\end{pmatrix}\begin{pmatrix}c&s\\-s&c\end{pmatrix}}_{\begin{pmatrix}(c^2-s^2)&2cs\\2cs&-(c^2-s^2)\end{pmatrix}}(\binom{x}{y}+\binom{0}{4})$$
(the form we find here has exactly the same structure as the solution of Emilio Novati ; recall formulas $\cos^2a-\sin^2a=\cos(2a)$ and $2 \cos a \sin a = \sin(2a)$).
It is essential to understand that, in (3), you have to "follow the same way back" in the reverse order, of course.

Edit 2: There are 2 kinds of plane (vectorial) isometries : rotations, and symmetries with resp. expressions:
$$\tag{2}\begin{pmatrix}a & \ \ b \\-b & \ \ a\end{pmatrix} \ \ \ \text{and} \ \ \ \ \begin{pmatrix}a & \ \ b \\ \ \ b & -a\end{pmatrix} \ \ \ \text{with} \ \ \ a^2+b^2=1.$$
Why that ? Because an isometry has an orthogonal matrix, and an orthogonal matrix must be such that its column constitute an orthonormal basis.
Clearly, the first column is arbitrary. Let us denote by $\binom{a'}{b'}$ the second column of the matrix of an isometry. It must be orthogonal to the first one, i.e., dot product  $\binom{a'}{b'} . \binom{a}{b}=0 \ \iff aa'+bb'=0$ with $a'^2+b'^2=1$. There are two solutions: $\binom{a'}{b'} = \binom{-b}{a}$ (direct orientation basis) or $\binom{a'}{b'} = \binom{b}{-a}$ (indirect orientation), whence formulas (2).
</s>