<s>[INST] Find the power series of rational function How to find the power series of rational functions of type, e.g.
$$\frac{1}{1-6x+12x^2},\frac{6x}{1-6x+12x^2}$$
where denominator can't be factorized over the real domain.
Is there a way to use the method of undetermined coefficients (by completing the square), or is it necessary to use complex domain and trigonometry?
 [/INST] Here is an approach based upon the geometric series expansion (see the comment from @YvesDaoust).

We obtain
  \begin{align*}
\frac{1}{1-6x+12x^2}&=\frac{1}{1-6x(1-2x)}\\
&=\sum_{k=0}^\infty (6x)^k(1-2x)^k\\
&=\sum_{k=0}^\infty (6x)^k\sum_{j=0}^k\binom{k}{j}(-2x)^j\\
&=1+6x+24x^2+72x^3+144x^4\\
&\qquad+\color{grey}{0}x^5-1728x^6-10368x^7-41472x^8-\cdots
\end{align*}

It is convenient to use the coefficient of operator $[x^n]$ to extract the coefficient of $x^n$.

We get
  \begin{align*}
[x^n]\frac{1}{1-6x+12x^2}&=[x^n]\sum_{k=0}^\infty (6x)^k\sum_{j=0}^k\binom{k}{j}(-2x)^j\\
&=\sum_{k=0}^n6^k[x^{n-k}]\sum_{j=0}^k\binom{k}{j}(-2x)^j\tag{1}\\
&=\sum_{k=0}^n6^k\binom{k}{n-k}(-2)^{n-k}\\
&=(-2)^n\sum_{k=0}^n\binom{k}{n-k}(-3)^k\tag{2}
\end{align*}
  and find this way a summation formula for the coefficient of the power series.

Comment:


*

*In (1) we use the linearity of the coefficient of operator and the rule $[x^p]x^qA(x)=[x^{p-q}]A(x)$.

*In (2) we select the coefficient of $x^{n-k}$.

Note: Since the zeros of $f(x)=1-6x+12x^2$ are non-real, we don't expect to get a closed formula of the binomial sum formula in real numbers only. In fact the following is valid (see e.g. formula 1.70 in this paper by R. Sprugnoli).
  \begin{align*}
\sum_{k=0}^n\binom{n-k}{k}z^k
=\frac{1}{\sqrt{1+4z}}
\left(\left(\frac{1+\sqrt{1+4z}}{2}\right)^{n+1}-\left(\frac{1-\sqrt{1+4z}}{2}\right)^{n+1}\right)
\end{align*}
  Replacing the index $k$ with $n-k$ gives
  \begin{align*}
\sum_{k=0}^n\binom{k}{n-k}z^{n-k}
=\frac{1}{\sqrt{1+4z}}
\left(\left(\frac{1+\sqrt{1+4z}}{2}\right)^{n+1}-\left(\frac{1-\sqrt{1+4z}}{2}\right)^{n+1}\right)
\end{align*}
This expression evaluated at $z=-\frac{1}{3}$ and multiplied with $(-2)^n$ gives the coefficient of the power series and we obtain
  \begin{align*}
(-2)^n\sum_{k=0}^n\binom{k}{n-k}(-3)^k
=\frac{i\sqrt{3}}{2}
\left(\left(1-\frac{i}{\sqrt{3}}\right)^{n+1}-\left(1+\frac{i}{\sqrt{3}}\right)^{n+1}\right)3^n
\end{align*}

</s>