<s>[INST] Simplifying $\sqrt[4]{161-72 \sqrt{5}}$ $$\sqrt[4]{161-72 \sqrt{5}}$$
I tried to solve this as follows:
the resultant will be in the form of $a+b\sqrt{5}$ since 5 is a prime and has no other factors other than 1 and itself. Taking this expression to the 4th power gives $a^4+4 \sqrt{5} a^3 b+30 a^2 b^2+20 \sqrt{5} a b^3+25 b^4$. The integer parts of this must be equal to $161$ and the coeffecients of the roots must add to $-72$. You thus get the simultaneous system:
$$a^4+30 a^2 b^2+25 b^4=161$$
$$4 a^3 b+20 a b^3=-72$$
In an attempt to solve this, I first tried to factor stuff and rewrite it as:
$$\left(a^2+5 b^2\right)^2+10 (a b)^2=161$$
$$4 a b \left(a^2+5 b^2\right)=-72$$
Then letting $p = a^2 + 5b^2$ and $q = ab$ you get
$$4 p q=-72$$
$$p^2+10 q^2=161$$
However, solving this yields messy roots. Am I going on the right path?
 [/INST] Denesting $\sqrt w = \sqrt{a+b\sqrt{n}}\,$ can be done by a simple formula that I discovered in my youth.
$ {\bf Simple\ Denesting\ Rule} \ \ \overbrace{\rm \color{#0a0}{subtract\ out}\ \sqrt{norm}^{\phantom .}}^{\textstyle\!\!\! w \to w - \sqrt{ww'} =:\, s\!\!\!\!\!\!}\!\!\!, \ {\rm then}\ \  \overbrace{\color{brown}{\rm divide\ out}\ \sqrt{{\rm trace}}^{\phantom .}}^{\textstyle s\,\to\, s/\sqrt{s+s'}\!\!\!\!\!\!\!}$ from that.
$\!\begin{align}{\rm Recall}\ \ w = a + b\sqrt{n}\rm \ \ has\ \ {\bf norm}\  &=\: w\:\cdot\: w' = (a + b\sqrt{n})\ \cdot\: (a - b\sqrt{n})\ =\: a^2\! - n\: b^2\\[4pt]
{\rm and,\ furthermore,\ }w\rm \ \ has\ \ {\bf trace}\ &=\: w+w' =  (a + b\sqrt{n}) + (a - b\sqrt{n})\: =\:  2\,a\end{align}$
In the  norm/trace sqrts either sign works e.g. $\sqrt 1 = \pm1,\,$ so we choose what proves simplest.

Here $\:161-72\sqrt 5\:$ has norm $= 1.\:$ $\rm\ \color{#0a0}{subtracting\ out}\ \sqrt{norm}\ = -1\ $ yields $\  162-72\sqrt 5\:$
which has $\ {\rm\ \sqrt{trace}}\: =\: \sqrt{324}\ =\ 18.\ \ \ \ \rm \color{brown}{Dividing\ it\ out}\ \,$ of the above  yields $\ \ \ 9-4\sqrt 5$
Checking: $\ (9 - 4\sqrt 5)^2 = 9^2\!+\! 4^2(5)- 2(9)4\sqrt 5 = 161-72\sqrt 5 \ \ \large \color{#c00}\checkmark$

Next $\:9-4\sqrt 5\:$ has norm $= 1.\:$ $\rm\ \color{#0a0}{subtracting\ out}\ \sqrt{norm}\ = 1\ $ yields $\  8-4\sqrt 5\:$
with $\ {\rm\ \sqrt{trace}}\: =\: \sqrt{16}\ =\ 4.\ \ \ \ \rm \color{brown}{Dividing\ it\ out}\,\ $ of the above  yields $\,\ \ \ 2-\sqrt 5$
Checking: $\ (2 - \sqrt 5)^2 = 2^2\!+\! 5 - 2\cdot 2\sqrt 5 = 9 - 4\sqrt 5 \ \ \large \color{#c00}\checkmark$
Negating $\,2-\sqrt 5\,$ to get the positive square-root yields the sought result. We chose the signs in  $\,\sqrt 1 = \pm 1$ so that  arithmetic is simplest. Any choice will work as the proof below shows (e.g. we do both here). For many worked examples see prior posts on denesting.  Below is a sketch of a proof.
Lemma $\ \  \sqrt w\, =\, \dfrac{s}t,\ \ \ \begin{align}s &\,=\, w \pm \sqrt{ww'}\\[.1em] t &\,=\: \pm\sqrt{s+s'}\end{align}\ $ when $\ \ \color{#90f}{\sqrt{ww'}\in\Bbb Q}$
Proof $\quad\ s^2 =\, w (w+w' \pm 2\sqrt{ww'})\, =\, w\, t^2$
Necessarily $\ \color{#90f}{\sqrt{ww'}\in \Bbb Q}\,$ if a denesting $\sqrt w = v = c + d\sqrt n\,$ exists, since
$$w = v^2\,\Rightarrow\, w' = v'^2\Rightarrow\, ww' = (vv')^2\in\Bbb Q^2$$
</s>