<s>[INST] About the formulas to determine the integral ideal with specific norm (1) For $\mathbb{Q}(\sqrt{10})$, the ring of integers is $\mathbb{Z}[\sqrt{10}]$, and to determine the integral ideal with norm $3$, we can use the formula below:
By $X^2 − 10\equiv (X + 1)(X − 1)\text{ mod 3}$, then we get the conjugate ideals with norm 3: $(3, \sqrt{10}+1), (3, \sqrt{10}-1)$.
(2) For $\mathbb{Q}(\sqrt{17})$  the ring of integers is $\mathbb{Z}[\alpha]$, where $\alpha=\frac{1+\sqrt{17}}{2}$ and to determine the integral ideal with norm $2$, we can use the formula below:
$0=(\alpha-1/2)^2-17/4=\alpha^2-\alpha-4$ $X^2 − X − 4 ≡ X(X + 1) \text{ mod 2}$ then we get the conjugate ideals with norm 2: $(2,\alpha),(2,\alpha+1)$.
I am confused with the reason of these two formulas, where are these formulas from. Actually, I can't find any reference to illustrate these formulas. Can anybody explain these two formulas? Thank you!
 [/INST] This answer, despite its length, is not a complete answer either, but I think it'll give you some insights. Your question feels like another one I've answered before, although I think the earlier question was more concerned with numbers than with ideals.
Regardless, I feel that the excessive early emphasis on defining $\alpha$ (sometimes $\theta$ or $\omega$, but I think the latter should be reserved for a specific ring) only serves to confuse students.
However, I am not certain the formulas you were given are correct, but I could be wrong, as I've approached this topic from a different, far less systematic path than yours.
Given a squarefree integer $d$, the norm function is defined for any number in $\mathbb Q(\sqrt d)$, whether or not that number is also in the ring of algebraic integers. Namely, if $a$ and $b$ are rational real numbers, then $N(a + b \sqrt d) = a^2 - db^2$. And if $a + b \sqrt d$ is an algebraic integer, then its norm is an ordinary integer.
For example, consider the number $$\frac{3 + \sqrt{10}}{2}.$$ Then $$N\left(\frac{3 + \sqrt{10}}{2}\right) = \left(\frac{3}{2}\right)^2 - 10\left(\frac{1}{2}\right)^2 = \frac{9}{4} - \frac{10}{4} = -\frac{1}{4},$$ which is not an integer.
In fact, for a number $a + b\sqrt{10}$ to be an algebraic integer, $a$ and $b$ must both be ordinary integers. From there it is easy to see that $a - 10b = \pm 3$ has no solutions, hence no number in this domain can have a norm of 3, and so the ideal with that norm can't be a principal ideal.
The smallest positive integer multiple of 3 that is a possible norm in this domain is 6, but 9 works, too, giving us the ideals $\langle 3, 1 - \sqrt{10} \rangle$ and $\langle 3, 1 + \sqrt{10} \rangle$.
Now let's go to $\mathbb Q(\sqrt{17})$, and consider the number $$\frac{3 + \sqrt{17}}{2}.$$ Then $$N\left(\frac{3 + \sqrt{17}}{2}\right) = \left(\frac{3}{2}\right)^2 - 17\left(\frac{1}{2}\right)^2 = \frac{9}{4} - \frac{17}{4} = -\frac{8}{4} = -2,$$ which is an integer.
Since the fundamental unit of this domain is of norm 1 rather than norm $-1$, a number with a norm of 2 is impossible. (Oops, silly mistake, there's $29 + 7\sqrt{17}$ divided by 2, and infinitely many others. I stand by everything else.)
But with ideals, we don't have to worry about signs as much, and so $$\langle 2 \rangle = \left\langle \frac{3 - \sqrt{17}}{2} \right\rangle \left\langle \frac{3 + \sqrt{17}}{2} \right\rangle,$$ a product of principal ideals.
Of course if you want to interpret those in terms of $$\alpha = \frac{1 + \sqrt{17}}{2},$$ you can, like so:
$$\frac{3}{2} - \frac{\sqrt{17}}{2} = \frac{4}{2} - \left(\frac{1}{2} + \frac{\sqrt{17}}{2}\right) = 2 - \alpha$$ and $$\frac{3}{2} + \frac{\sqrt{17}}{2} = \frac{2}{2} + \left(\frac{1}{2} + \frac{\sqrt{17}}{2}\right) = 1 + \alpha.$$
You should recognize the latter number as a co-generator of one of the ideals you found: $\langle 2, 1 + \alpha \rangle$. But since $(2 - \alpha)(1 + \alpha) = -2$, the ideal $\langle 2 \rangle$ is contained in the ideal $\langle 1 + \alpha \rangle$, and therefore 2 is a redundant generator in $\langle 2, 1 + \alpha \rangle$.
As for the ideal $\langle 2, \alpha \rangle$, that can also be boiled down to a principal ideal, namely $\langle \gcd(2, \alpha) \rangle$, but I don't think that's a prime ideal, since, for starters, $N(\alpha) = -4$.
</s>