<s>[INST] GCD of cubic polynomials I would appreciate some help finding $GCD(a^3-3ab^2, b^3-3ba^2)$; $a,b \in \mathbb{Z}$. So far I've got here: if $GCD(a,b)=d$ then $\exists  \alpha, \beta$ so that $GCD(\alpha, \beta)=1$ and $\alpha d=a$, $\beta d=b$.
Therefore we know that $GCD(a^3-3ab^2, b^3-3ba^2)=d^3 GCD(\alpha^3-3\alpha \beta^2, \beta^3-3\beta \alpha^2)$. However I don't know to figure out $GCD(\alpha^3-3\alpha \beta^2, \beta^3-3\beta \alpha^2)$ given that $GCD(\alpha,\beta)=1$.
 [/INST] As you've already shown, factoring out the cube of the GCD of $a$ and $b$ gives a new equation
$$e = \gcd\left(\alpha^3-3\alpha \beta^2, \beta^3-3\beta \alpha^2 \right) \tag{1}\label{eq1}$$
where
$$\gcd\left(\alpha,\beta \right) = 1 \tag{2}\label{eq2}$$
Update: Here is a simpler solution than what I originally wrote. First, note that no factor of $e$ may divide $\alpha$ or $\beta$. If any do, let's say $\alpha$, then it must divide $\beta^3 - 3\beta\alpha^2$ and, thus, must divide $\beta^3$, which is not possible due to \eqref{eq2}. Thus, from the first term of \eqref{eq1}, since $\alpha^3-3\alpha \beta^2 = \alpha\left(\alpha^2 - 3\beta^2\right)$, this means that $e \mid \alpha^2 - 3\beta^2$. Similarly, for the second term, $\beta^3-3\beta \alpha^2 = \beta\left(\beta^2 - 3\alpha^2\right)$ gives that $e \mid \beta^2 - 3\alpha^2$. Also, $e$ must divide any linear combination of these values, including $e | \alpha^2 - 3\beta^2 + 3\left(\beta^2 - 3\alpha^2\right) = -8\alpha^2$. Thus, $e$ can only be a power of $2$. To finish this off, go to the second last paragraph. Otherwise, for the rest of the original, longer solution, continue reading.
$ $
Next, note that if $f = \gcd(g,h)$, then $f$ divides $g$ and $h$ and, thus, will also divide any linear combination of $g$ and $h$, including their sum & difference. From \eqref{eq1}, first check the sum of the $2$ inside values:
\begin{align}
\alpha^3-3\alpha \beta^2 + \beta^3-3\beta \alpha^2 & = \alpha^3 + \beta^3 -3\left(\alpha \beta\right)\beta - 3\left(\alpha \beta\right)\alpha \\
& = \left(\alpha + \beta\right)\left(\alpha^2 - \alpha\beta + \beta^2\right) - 3\alpha\beta\left(\alpha + \beta\right) \\
& = \left(\alpha + \beta\right)\left(\alpha^2 - 4\alpha\beta + \beta^2\right) \tag{3}\label{eq3}
\end{align}
Suppose there's a factor $m \gt 1$ which divides $e$ and $\alpha + \beta$. Then, $\alpha \equiv -\beta \pmod m$, so $\alpha^3-3\alpha \beta^2 \equiv 2\beta^3 \pmod m$. From \eqref{eq2}, this means that $m = 2$, and that any other factors of $e$ must divide $\alpha^2 - 4\alpha\beta + \beta^2$.
From \eqref{eq1}, next check the difference of the $2$ inside values:
\begin{align}
\alpha^3-3\alpha \beta^2 - \beta^3 + 3\beta \alpha^2 & = \alpha^3 - \beta^3 -3\left(\alpha \beta\right)\beta + 3\left(\alpha \beta\right)\alpha \\
& = \left(\alpha - \beta\right)\left(\alpha^2 + \alpha\beta + \beta^2\right) + 3\alpha\beta\left(\alpha - \beta\right) \\
& = \left(\alpha - \beta\right)\left(\alpha^2 + 4\alpha\beta + \beta^2\right) \tag{4}\label{eq4}
\end{align}
Suppose there's a factor $n \gt 1$ which divides $e$ and $\alpha - \beta$. Then, $\alpha \equiv \beta \pmod n$, so $\alpha^3-3\alpha \beta^2 \equiv -2\beta^3 \pmod m$. From \eqref{eq2}, this means that $n = 2$, and that any other factors of $e$ must divide $\alpha^2 + 4\alpha\beta + \beta^2$.
This shows that any factor, other than $2$, which divides $e$ must divide both $\alpha^2 - 4\alpha\beta + \beta^2$ and $\alpha^2 + 4\alpha\beta + \beta^2$. Thus, it must also divide their difference, which is $8\alpha\beta$. This can only be true for $2$, $4$ or $8$.
At this shows overall, only powers of $2$ may possibly divide $e$. Since $e$ is relatively prime to $\alpha$ & $\beta$, this means they must both be odd. From $\alpha^3 - 3\alpha\beta^2 = \alpha\left(\alpha^2 - 3\beta^2\right)$, note that $\alpha^2 \equiv \beta^2 \equiv 1 \pmod 4$, so $\alpha^2 - 3\beta^2 \equiv -2 \pmod 4$. In other words, there will only be $1$ factor of $2$.
In summary, with your original equation of $d = \gcd\left(a,b\right)$, we get that $\gcd\left(a^3-3ab^2, b^3-3ba^2\right)$ is $2d^3$ if both $\frac{a}{d}$ and $\frac{b}{d}$ are odd, else it's $d^3$.
</s>