<s>[INST] reduction of $6xy + 8 y^2 -12x-26y + 11 = 0$ to canonical one of a second-order curve I have this polynomial 
$$
6xy + 8 y^2 -12x-26y + 11 = 0
$$
and I need to reduce it to a canonical equation of a second-order curve. The correct answer from the textbook is that it is a hyperbola 
$$
\frac{X^2}{1} - \frac{Y^2}{9} = 1
$$
and the coordinate system must be rotated to an angle equal to 
$$
\arctan{3}
$$
and the origin of the new coordinate system must be moved to the point $O'(-1,2)$. I checked this in Mathematica® program, so I think it is safe to assume that there are no errors in the textbook.

And now about the problem itself. I calculated the angle of rotation according to the formula in my textbook. It is (the expression above is the example of a polynom to explain indexes like $a,b,c$ etc.)
$$
ax^2 + 2bxy + cy^2 + 2dx + 2ey +g = 0 \\
\cot{2\alpha} = \frac{a-c}{2b}
$$
I solved this equation and got the correct answer: $\alpha = \arctan{3}$
Then I substituted $x$ and $y$ variables with their values according to the formulae:
$$
x = \cos{\alpha}x' - \sin{\alpha}y'\\
y = \sin{\alpha}x' + \cos{\alpha}y'
$$
Let me show this incrementally for convenience
$$
6xy + 8 y^2 =\\
6(\cos{\alpha}x' - \sin{\alpha}y')(\sin{\alpha}x' + \cos{\alpha}y') + 8(\sin{\alpha}x' + \cos{\alpha}y')^2 =\\
6(\sin{\alpha}\cos{\alpha} * x'^2 +\cos^2{\alpha} * x'y' -\sin^2{\alpha} * x'y' - \sin{\alpha}\cos{\alpha} * y'^2) + 8(\sin^2{\alpha} * x'^2 + 2\sin{\alpha}\cos{\alpha} * x'y' + \cos^2{\alpha} * y'^2) =\\
6\sin{\alpha}\cos{\alpha} * x'^2 + 8\sin^2{\alpha} * x'^2 - 6\sin{\alpha}\cos{\alpha} * y'^2 + 8\cos{\alpha}^2 * y'^2 + 6\cos{\alpha}^2 * x'y' -6\sin^2{\alpha} * x'y' + 16\sin{\alpha}\cos{\alpha} * x'y' = \\
(6\sin{\alpha}\cos{\alpha} + 8\sin^2{\alpha})x'^2 + (- 6\sin{\alpha}\cos{\alpha} + 8\cos^2{\alpha})y'^2 + (6\cos{\alpha}^2 -6\sin^2{\alpha} + 16\sin{\alpha}\cos{\alpha})x'y'
$$
the factor at $x'y'$ evaluates to 0 given $\alpha = \arctan{3}$, I checked this in Mathematica, so from the part above only this remains:
$$
(6\sin{\alpha}\cos{\alpha} + 8\sin^2{\alpha})x'^2 + (- 6\sin{\alpha}\cos{\alpha} + 8\cos^2{\alpha})y'^2
$$
And the remaining part is:
$$
-12x-26y + 11 = \\
-12(\cos{\alpha}*x' - \sin{\alpha}*y') - 26(\sin{\alpha}*x' + \cos{\alpha}*y') + 11 = \\
-12\cos{\alpha}*x' + 12\sin{\alpha}*y' -26\sin{\alpha}*x' - 26\cos{\alpha}*y' + 11 = \\
(-12\cos{\alpha} - 26\sin{\alpha})*x' + (12\sin{\alpha} - 26\cos{\alpha})*y' + 11 = \\
2((-6\cos{\alpha} - 13\sin{\alpha}))*x' + 2(6\sin{\alpha} - 13\cos{\alpha})*y' + 11
$$
So now this curve has the next equation in the rotated coordinate system (let me use $x$ and $y$ instead of $x'$ and $y'$):
$$
\alpha = \arctan{3}\\
a = 6\sin{\alpha}\cos{\alpha} + 8\sin^2{\alpha} \\
c = - 6\sin{\alpha}\cos{\alpha} + 8\cos^2{\alpha} \\
d = -6\cos{\alpha} - 13\sin{\alpha} \\
e = 6\sin{\alpha} - 13\cos{\alpha}\\
g = 11\\
ax^2 + cy^2 + 2dx  + 2ey + g = 0
$$
Then it is stated in my textbook that one can further reduce this equation by moving the origin of a coordinate system to the point $O(-\frac{d}{a}, -\frac{e}{c})$
And this is the place where I can't get the correct answer (which is $O(-1,2)$), cause this $-\frac{d}{a}$ gives me 1.58114 instead of -1 and this $-\frac{e}{c}$ gives 1.58114 instead of 2. 
I've checked this in Mathematica.


Could anyone explain me what where I'm wrong, please?
Thank you in advance.
 [/INST] I always try to complete the squares:
$$6xy+8y^2 −12x−26y+11=0$$
$$\left[ 8y^2+2.2 \sqrt{2}y\left(\frac{3x}{2\sqrt{2}}-\frac{13}{2\sqrt{2}}\right)\right]-12x+11=0 $$
$$\left[ 8y^2+2.2 \sqrt{2}y\left(\frac{3x}{2\sqrt{2}}-\frac{13}{2\sqrt{2}}\right)+\left(\frac{3x}{2\sqrt{2}}-\frac{13}{2\sqrt{2}}\right)^2\right]-\left(\frac{3x}{2\sqrt{2}}-\frac{13}{2\sqrt{2}}\right)^2-12x+11=0 $$
$$\left[2\sqrt{2}y + \frac{3x}{2\sqrt{2}}-\frac{13}{2\sqrt{2}} \right]^2-\frac{1}{8}(9x^2+18x+81)=0 $$
$$\left[2\sqrt{2}y + \frac{3x}{2\sqrt{2}}-\frac{13}{2\sqrt{2}} \right]^2-\frac{1}{8}(3x+3)^2=9 $$
$$\frac{1}{8}(8y+3x-13)^2-\frac{1}{8}(3x+3)^2=9 $$
$$(8y+3x-13)^2-(3x+3)^2=72$$
$$(8y+3x-13)^2-\frac{(x+1)^2}{9}=72$$
$$X^2-\frac{Y^2}{9}=72$$
</s>