projector quantizes the companion matrix which is a form of a Kostant slice.

##  5\. Kostant-Whittaker reduction and quantization

In this section, we consider the Whittaker analog of the parabolic reduction
functor from Section 1.5 that we call (following [BF08]) the _Kostant-
Whittaker reduction_. This is a finite-dimensional analog of the Drinfeld-
Sokolov reduction, see [Ara17].

###  5.1. Mirabolic setting

Consider the category $\mathrm{HC}_{\hbar}(\mathrm{M}_{N})$ of Harish-Chandra
bimodules over the mirabolic subalgebra $\mathfrak{m}_{N}$ as in Section 1.2.
Denote by $\mathrm{Wh}_{\hbar}(\mathfrak{m}_{N})$ the category of right
Whittaker modules over $\mathfrak{m}_{N}$ from 4.1. It has a distinguished
object
$Q_{\mathfrak{m}_{N}}=\mathfrak{n}_{-}^{\psi}\mathrm{U}_{\hbar}(\mathfrak{m}_{N})\backslash\mathrm{U}_{\hbar}(\mathfrak{m}_{N})$.
Similarly to (1.11), we can define a functor:

$\mathrm{act}_{k}^{\psi}\colon\mathrm{Vect}\rightarrow\mathrm{Wh}_{\hbar}(\mathfrak{m}_{N}),\qquad
A\mapsto A\otimes Q_{\mathfrak{m}_{N}},$

Then we have an analog of _Skryabin’s equivalence_ [Pre02] (originally proved
in [Kos78] for principal $W$-algebras):

######  Theorem 5.1 (Mirabolic Skryabin’s equivalence).

Functor $\mathrm{act}_{k}^{\psi}$ is an equivalence.

###### Proof.

We proceed as in Proposition 1.11. The functor
$(-)^{\mathfrak{n}_{-}^{\psi}}\colon\mathrm{Wh}_{\hbar}(\mathfrak{m}_{N})\rightarrow\mathrm{Vect}$
is right adjoint to $\mathrm{act}_{k}^{\psi}$:

$\displaystyle\mathrm{Hom}_{\mathrm{Wh}_{\hbar}(\mathfrak{m}_{N})}(\mathrm{act}_{k}^{\psi}(A),M)$
$\displaystyle=\mathrm{Hom}_{\mathrm{Wh}_{\hbar}(\mathfrak{m}_{N})}(A\otimes
Q_{\mathfrak{m}_{N}},M)\cong$
$\displaystyle\cong\mathrm{Hom}_{\mathrm{U}_{\hbar}(\mathfrak{m}_{N})}(A\otimes
Q_{\mathfrak{m}_{N}},M)\cong$
$\displaystyle\cong\mathrm{Hom}(A,M^{\mathfrak{n}_{-}^{\psi}}).$

The unit of the adjunction is given by

$A\mapsto(A\otimes Q_{\mathfrak{m}_{N}})^{\mathfrak{n}_{-}^{\psi}}.$

The Kirillov projector gives an isomorphism with the quotient functor as in
(4.4):

$(-)_{\mathfrak{b}^{\vec{u}}}\colon\mathrm{Wh}_{\hbar}(\mathfrak{m}_{N})\rightarrow\mathrm{Vect},\qquad
M\mapsto M/\mathfrak{b}^{\vec{u}}.$

So, the functor $(-)^{\mathfrak{n}_{-}^{\psi}}$ is exact. We have

$(A\otimes Q_{\mathfrak{m}_{N}})^{\mathfrak{n}_{-}^{\psi}}\cong(A\otimes
Q_{\mathfrak{m}_{N}})/\mathfrak{b}^{\vec{u}}$

and the composition

$A\mapsto(A\otimes
Q_{\mathfrak{m}_{N}})^{\mathfrak{n}_{-}^{\psi}}\cong(A\otimes
Q_{\mathfrak{m}_{N}})/\mathfrak{b}^{\vec{u}}$

is an isomorphism by the PBW theorem. The rest follows by the same arguments
as in Proposition 1.11. ∎

Similarly to (1.12), one can define the action functor

$\mathrm{act}_{\mathfrak{m}_{N}}^{\psi}\colon\mathrm{HC}_{\hbar}(\mathrm{M}_{N})\rightarrow\mathrm{Wh}_{\hbar}(\mathfrak{m}_{N}),\qquad
X\mapsto Q\otimes_{\mathrm{U}_{\hbar}(\mathfrak{m}_{N})}X.$

The following definition is an analog of 1.12:

######  Definition 5.2.

The _mirabolic Kostant-Whittaker reduction_ functor
$\mathrm{res}_{\mathfrak{m}_{N}}^{\psi}\colon\mathrm{HC}_{\hbar}(\mathrm{M}_{N})\rightarrow\mathrm{Vect}$
is the composition

$\mathrm{HC}_{\hbar}(\mathrm{M}_{N})\xrightarrow{\mathrm{act}_{\mathfrak{m}_{N}}}\mathrm{Wh}_{\hbar}(\mathfrak{m}_{N})\xrightarrow{(-)^{\mathfrak{n}_{-}^{\psi}}}\mathrm{Vect}.$

Explicitly, it is given by a quantum Hamiltonian reduction
$X\mapsto(\mathfrak{n}_{-}^{\psi}\backslash X)^{\mathfrak{n}_{-}^{\psi}}$.
There is a natural lax monoidal structure on
$\mathrm{res}_{\mathfrak{m}_{N}}^{\psi}$:

$(\mathfrak{n}_{-}^{\psi}\backslash
X)^{\mathfrak{n}_{-}^{\psi}}\otimes(\mathfrak{n}_{-}^{\psi}\backslash
Y)^{\mathfrak{n}_{-}^{\psi}}\rightarrow(\mathfrak{n}_{-}^{\psi}\backslash
X\otimes_{\mathrm{U}_{\hbar}(\mathfrak{m}_{N})}Y)^{\mathfrak{n}_{-}^{\psi}}.$

As in Theorem 1.13, we have

######  Theorem 5.3.

The functor $\mathrm{res}_{\mathfrak{m}_{N}}^{\psi}$ is colimit-preserving and
monoidal.

###### Proof.

Follows from exactly the same arguments as [KS22, Corollary 4.18]. ∎

Recall the monoidal functor
$\mathrm{free}\colon\operatorname{Rep}(\mathrm{M}_{N})\rightarrow\mathrm{HC}_{\hbar}(\mathrm{M}_{N})$
from Section 1.2. On the one hand, we get a monoidal functor
$\operatorname{Rep}(\mathrm{M}_{N})\rightarrow\mathrm{Vect}$ by composing with
$\mathrm{res}_{\mathfrak{m}_{N}}^{\psi}$. On the other hand, there is a
forgetful functor
$\mathrm{forget}\colon\operatorname{Rep}(\mathrm{M}_{N})\rightarrow\mathrm{Vect}$.
So, we have a diagram

(5.1)
$\textstyle{\operatorname{Rep}(\mathrm{M}_{N})\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\mathrm{free}}$$\scriptstyle{\mathrm{forget}}$$\textstyle{\mathrm{HC}_{\hbar}(\mathrm{M}_{N})\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\mathrm{res}_{\mathfrak{m}_{N}}^{\psi}}$$\textstyle{\mathrm{Vect}}$

The following result is an analog of Theorem 1.14:

######  Theorem 5.4.

The Kirillov projector defines a natural isomorphism

$P_{\mathfrak{m}_{N}}(\vec{u})_{V}\colon
V\rightarrow(\mathfrak{n}_{-}^{\psi}\backslash\mathrm{U}_{\hbar}(\mathfrak{m}_{N})\otimes
V)^{\mathfrak{n}_{-}^{\psi}}.$

In other words, the diagram (5.1) is commutative.

###### Proof.

By (4.4), we have

$(\mathfrak{n}_{-}^{\psi}\backslash\mathrm{U}_{\hbar}(\mathfrak{m}_{N})\otimes
V)^{\mathfrak{n}_{-}^{\psi}}\cong\mathfrak{n}_{-}^{\psi}\backslash\mathrm{U}_{\hbar}(\mathfrak{m}_{N})\otimes
V/\mathfrak{b}^{\vec{u}}.$

By the PBW theorem, the composition

$V\rightarrow(\mathfrak{n}_{-}^{\psi}\backslash\mathrm{U}_{\hbar}(\mathfrak{m}_{N})\otimes
V)^{\mathfrak{n}_{-}^{\psi}}\cong\mathfrak{n}_{-}^{\psi}\backslash\mathrm{U}_{\hbar}(\mathfrak{m}_{N})\otimes
V/\mathfrak{b}^{\vec{u}}$

is an isomorphism. ∎

In particular, there is a monoidal structure on $\mathrm{forget}$ induced from
$\mathrm{res}_{\mathfrak{m}_{N}}^{\psi}$:

######  Theorem 5.5.

  1. (1)

There is a collection of maps $F_{VW}(\vec{u})$ natural in
$V,W\in\operatorname{Rep}(\mathrm{M}_{N})$, such that the diagram

$\textstyle{V\otimes
W\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{F_{VW}(\vec{u})}$$\scriptstyle{(P_{\mathfrak{m}_{N}}(\vec{u}))_{V}\otimes(P_{\mathfrak{m}_{N}}(\vec{u}))_{W}}$$\textstyle{V\otimes
W\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{(P_{\mathfrak{m}_{N}}(\vec{u}))_{V\otimes
W}}$$\textstyle{V\otimes
W\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{V\otimes
W}$

is commutative, where the lower right arrow is the natural tensor structure on
$\mathrm{res}_{\mathfrak{m}_{N}}^{\psi}$. In particular, the collection of
$F_{VW}(\vec{u})$ satisfies the twist equation (1.1), and
$R_{VW}(\vec{u}):=F_{WV}(\vec{u})^{-1}F_{VW}(\vec{u})$ is a solution to the
quantum Yang-Baxter equation.

  2. (2)

The map $F_{VW}(\vec{u})$ has the form

$F_{VW}(\vec{u})\in\mathrm{id}_{V\otimes
W}+\hbar\mathrm{U}_{\hbar}(\mathfrak{n}_{-})^{>0}\otimes\mathrm{U}_{\hbar}(\mathfrak{b})^{>0},$

where the upper subscript $>0$ means the augmentation ideal.

  3. (3)

The inverse $F_{VW}(\vec{u})^{-1}$ also satisfies

$F_{VW}(\vec{u})^{-1}\in\mathrm{id}_{V\otimes
W}+\hbar\mathrm{U}_{\hbar}(\mathfrak{n}_{-})^{>0}\otimes\mathrm{U}_{\hbar}(\mathfrak{b})^{>0},$

###### Proof.

We proceed exactly as in Theorem 1.15. For brevity, denote
$P=P_{\mathfrak{m}_{N}}(\vec{u})$. Every element in
$(\mathfrak{n}_{-}^{\psi}\backslash\mathrm{U}_{\hbar}(\mathfrak{m}_{N})\otimes
V)^{\mathfrak{n}_{-}^{\psi}}$ can be presented as $PvP$ for $v\in V$ by
Theorem 5.4. Therefore, we need to show that

(5.2) $PvPwP=PF(v\otimes w)P.$

First, let us choose the following PBW basis: $\\{E_{ij}^{\psi}\\}$ for
$\mathrm{U}_{\hbar}(\mathfrak{n}_{-})$ and

$(E_{kl},E_{kk}+u_{k}|1\leq k<l\leq N-1)$

for $\mathrm{U}_{\hbar}(\mathfrak{b})$ (exactly in this order for the latter).
It is clear from the construction (4.2) and Remark 4.12 that

(5.3) $P\in
1+\mathfrak{n}_{-}^{\psi}\mathrm{U}_{\hbar}(\mathfrak{m}_{N})\cap\mathrm{U}_{\hbar}(\mathfrak{m}_{N})\mathfrak{b}^{\vec{u}}$

(recall the notation $\mathfrak{b}^{\vec{u}}$ from Eq. 4.3). Let us write it
in this PBW basis:

$P=1+\hbar^{k_{i}}\sum\limits_{i}f^{\psi}_{i}e_{i}^{u},$

where
$f^{\psi}_{i}\in\mathrm{U}_{\hbar}(\mathfrak{n}_{-}),e_{i}^{u}\in\mathrm{U}_{\hbar}(\mathfrak{b})$,
and $k_{i}$ is some negative number. This is well-defined, since a part of $P$
that acts non-trivially is actually finite. It is clear from (5.3) that
$e^{u}_{i}=1$ if and only if $f^{\psi}_{i}=1$.

Now consider the middle $P$ in (5.2). As in Theorem 1.15, we push the
$f_{i}^{\psi}$-term to the left until it meets $P$ and becomes zero. Likewise,
we push the $e_{i}^{u}$-term to right until it meets $P$ and becomes zero as
well. Since every term

$\hbar^{-k}(-E_{ij}^{\psi})(-1)^{(i+j)k}\frac{\prod_{i=0}^{k-1}L_{j\ldots
i-1}^{j\ldots i-1}(u-\hbar i+\hbar)}{k!}$

in (4.1) acts with the power at least $\hbar^{k}$, the second part of the
theorem follows. The third part follows from the fact that $F_{VW}(\vec{u})$
is upper-triangular in the natural filtration on $V\otimes W$ induced from a
$\mathfrak{n}_{-}$-filtration on $V$ and $\mathfrak{b}$-filtration on $W$.

∎

######  Remark 5.6.

In general, let $X$ be any Whittaker module and $V$ any representation of
$\mathrm{M}_{N}$. Then for any $x\in X$ and $v\in V$,

$PxvP=\sum\limits_{i}Px_{i}Pv_{i}P$

for some $x_{i}\in X,v_{i}\in V$. Indeed: let us define an adjoint action on
$X$ by

$\operatorname{ad}_{\xi}(x):=x\xi^{\psi},\ \xi\in\mathfrak{n}_{-}.$

Since $\psi$ is a character, it is indeed an action, moreover, it is locally
nilpotent by the Whittaker assumption on $X$. Therefore, we can proceed
exactly as in the proof of Theorem 5.5 and use the fact that the analog of
$F_{VW}(\vec{u})$ in this case is invertible as well.

######  Example 5.7.

Let $N=2$. The Kirillov projector is given by

$\sum\limits_{k\geq 0}(E_{21}^{\psi})^{k}\hbar^{-k}\frac{L_{1}^{1}(u)\ldots
L_{1}^{1}(u-\hbar k+\hbar)}{k!}.$

Then the procedure described in the proof of Theorem 5.5 gives the following
formula for the twist:

(5.4) $F(u)=\sum\limits_{k\geq
0}\frac{(-\hbar)^{k}}{k!}E_{21}^{k}\otimes\prod_{i=0}^{k-1}(E_{11}-i).$

Observe that it does not depend on $u$. A version of it appeared in [GGS92],
see also [KST98]. An additional parameter in the formulas from _loc. cit._
($t$ in the former and $\lambda$ in the latter) corresponds to a choice of a
non-degenerate character $\psi\colon\mathfrak{n}_{-}\rightarrow k$ that we
assumed to be $\psi(E_{21})=1$ for simplicity.

######  Definition 5.8.

A _rational Cremmer-Gervais twist_ $F^{\mathrm{CG}}(\vec{u})$ is the
collection of isomorphisms $F_{VW}(\vec{u})$. A _rational Cremmer-Gervais
$R$-matrix_ is
$R^{\mathrm{CG}}(\vec{u}):=(F_{WV}^{\mathrm{CG}}(\vec{u}))^{-1}F_{VW}^{\mathrm{CG}}(\vec{u})$.

To justify the name, let us compute the semiclassical limit of
$F_{VW}^{\mathrm{CG}}(\vec{u})$:

######  Theorem 5.9.

The tensor structure of Theorem 5.5 quantizes a family of rational Cremmer-
Gervais $r$-matrices $r^{\mathrm{CG}}(\vec{u})$ from 2.3.

###### Proof.

We use notations and conventions from the proof of Theorem 5.5. Denote by

$f_{VW}=\sum\limits_{N\geq i>j\geq 1}E_{ij}\otimes\alpha_{ij},$

where $\alpha_{ij}\in\mathfrak{b}\subset\mathfrak{m}_{N}$. It will follow from
the proof that the semiclassical limit depends only on the zeroth
$\hbar$-power of $\vec{u}$, therefore, we can assume without loss of
generality that $\vec{u}$ does not depend on $\hbar$.

Recall that by Proposition 2.2, it is enough to prove that these elements
satisfy the relation

$\alpha_{i+k,i}=-E_{i,i+k-1}-(u_{i}-u_{i+k-1})\alpha_{i+k-1,i}+\delta_{i>1}\alpha_{i+k-1,i-1}.$

It will follow from the following

######  Lemma 5.10.

Let

$PvPwP=Pv+P\left(\sum\limits_{N\geq i>j\geq
1}\operatorname{ad}_{E_{ij}}(v)\beta_{ij}+O(\hbar^{2})\right)wP$

Then, up to higher order terms,

  1. (1)

$\beta_{ij}$ are sums of diagonal minor $L_{a\ldots b}^{a\ldots b}(u_{a})$
with some coefficients;

  2. (2)

$\beta_{ij}$ satisfy the relation

(5.5) $\displaystyle\begin{split}&\beta_{i+k,i}=\\\ =&(-1)^{k+1}(L_{i\ldots
i+k-1}^{i\ldots i+k-1}(u_{i})-(u_{i}-u_{i+k-1})L_{i\ldots i+k-2}^{i\ldots
i+k-2}(u_{i}))-(u_{i}-u_{i+k-1})\beta_{i+k-1,i}+\beta_{i+k-1,i-1}.\end{split}$

###### Proof.

Before proving the statement, let us show that

$PvL_{k\ldots l}^{k\ldots
l}(u_{k})xP\in\hbar\sum\limits_{i=k}^{l}(-1)^{i-k}\prod_{m=i+1}^{l}(u_{k}-u_{m})\cdot\operatorname{ad}_{E_{ki}}(x)P+O(\hbar^{2}).$

for any $x\in\mathrm{U}_{\hbar}(\mathfrak{m}_{N})\otimes W$ (in particular,
nonzero $\hbar$-degrees in $\vec{u}$ do not contribute to the semiclassical
limit). Indeed, by Proposition 3.1, we have

$L_{k\ldots l}^{k\ldots
l}(u_{k})=\sum\limits_{i=k}^{l}(-1)^{i-k}L_{k\ldots\hat{i}\ldots l}^{k+1\ldots
l}(u_{k}-\hbar)L_{ki}(u_{k}).$

Observe that for every $i$,

$L_{ki}(u_{k})wP=\hbar\operatorname{ad}_{E_{ki}}(w)P+wL_{ki}(u_{k})P=\hbar\operatorname{ad}_{E_{ki}}(w)$

by the left invariance of the Kirillov projector from Theorem 4.13. Also
observe that for every $\xi\in\mathfrak{n}_{-}$,

$Pv\xi^{\psi}=-\hbar P\operatorname{ad}_{\xi}(v)$

by the right invariance Theorem 4.5. Therefore, it is enough to consider the
terms $L_{k\ldots\hat{i}\ldots l}^{k+1\ldots l}(u_{k}-\hbar)$ in the quotient
$\mathfrak{n}_{-}^{\psi}\backslash\mathrm{U}_{\hbar}(\mathfrak{m}_{N})$.
Again, by Proposition 3.1,

$L_{k\ldots\hat{i}\ldots l}^{k+1\ldots
l}(u_{k}-\hbar)=\sum\limits_{j=k+1}^{l}(-1)^{j-k-1}L_{jk}(u-\hbar
l+k\hbar-\hbar)L_{k+1\ldots\hat{i}\ldots l}^{k+1\ldots\hat{j}\ldots
l}(u_{k}-\hbar).$

In particular, all terms except for $j=k+1$ will be zero in the quotient.
Therefore, the class of $L_{k\ldots\hat{i}\ldots l}^{k+1\ldots l}(u_{k})$ is
equal to that of $L_{k+1\ldots\hat{i}\ldots l}^{k+2\ldots l}(u_{k}-\hbar)$. By
obvious induction, we can conclude that its class is equal to

(5.6) $L_{k\ldots\hat{i}\ldots l}^{k+1\ldots l}(u_{k}-\hbar)\equiv
L_{i+1\ldots l}^{i+1\ldots l}(u_{k}-\hbar)\
\mod(\mathfrak{n}_{-}^{\psi}\mathrm{U}_{\hbar}(\mathfrak{m}_{N})).$

Moreover,

$L_{i+1\ldots l}^{i+1\ldots
l}(u_{k}-\hbar)=\prod\limits_{m=i+1}^{l}(u_{k}-u_{m})+O(\hbar).$

Therefore,

(5.7) $PvL_{k\ldots l}^{k\ldots
l}(u_{k})xP\in\hbar\sum\limits_{i=k}^{l}(-1)^{i-k}\prod_{m=i+1}^{l}(u_{k}-u_{m})\cdot\operatorname{ad}_{E_{ki}}(x)P+O(\hbar^{2}).$

We will prove the statement of the lemma by induction on $N$ in
$\mathfrak{m}_{N}$. The base $N=2$ follows from (5.4). Assume we proved it for
$N-1$, in particular, we can apply it to ${}_{1}\mathfrak{m}_{N}$. Let

$P^{\prime}:=P_{{}_{1}\mathfrak{m}_{N}}(u_{2},\ldots,u_{N-1}).$

Then we consider

$PvPwP=PvwP+\sum\limits_{N\geq i>j\geq
2}\operatorname{ad}_{E_{ij}}(v)({}_{1}\beta_{ij})P_{21}^{\psi}(u_{1})\ldots
P_{N1}^{\psi}(u_{1})wP,$

where $({}_{1}\beta_{ij})$ are the coefficients of the lemma for $P^{\prime}$.
Let us study the action of $P_{k1}^{\psi}(u_{1})$. It is clear that it is
enough to consider only degree one truncations

$\tilde{P}_{k1}^{\psi}(u_{1}):=1+(-1)^{k+1}(-E_{k1}^{\psi})L_{1\ldots
k-1}^{1\ldots k-1}(u_{1})$

of $P_{k1}^{\psi}(u_{1})$, since

$\prod_{i=0}^{k-1}L_{1\ldots k-1}^{1\ldots k-1}(u_{1}-i\hbar)wP\in
O(\hbar^{k}).$

By our assumption, each ${}_{1}\beta_{ij}$ is a sum of diagonal minors of the
form $L_{a\ldots b}^{a\ldots b}(u_{a})$. So, let $2\leq a<b$ be arbitrary. We
have

$\displaystyle P\operatorname{ad}_{E_{ij}}(v)L_{a\ldots b}^{a\ldots
b}(u_{a})\cdot(-1)^{k+1}(-E_{k1}^{\psi})L_{1\ldots k-1}^{1\ldots
k-1}(u_{1})wP=$ $\displaystyle=$
$\displaystyle(-1)^{k+1}P\operatorname{ad}_{E_{k1}}\operatorname{ad}_{E_{ij}}(v)L_{a\ldots
b}^{a\ldots b}(u_{a})L_{1\ldots k-1}^{1\ldots k-1}(u_{1})wP+$ $\displaystyle+$
$\displaystyle(-1)^{k+1}P\operatorname{ad}_{E_{ij}}(v)\operatorname{ad}_{E_{k1}}(L_{a\ldots
b}^{a\ldots b}(u_{a}))L_{1\ldots k-1}^{1\ldots k-1}(u_{1})wP.$

Observe that

$L_{a\ldots b}^{a\ldots b}(u_{a})L_{1\ldots k-1}^{1\ldots k-1}(u_{1})wP\in
O(\hbar^{2}).$

Since $L_{1\ldots k-1}^{1\ldots k-1}(u_{1})w\in O(\hbar)$, it is enough to
consider the image of $\operatorname{ad}_{E_{k1}}(L_{a\ldots b}^{a\ldots
b}(u_{a}))$ in the quotient

$\mathfrak{n}_{-}\backslash\mathrm{U}_{\hbar}(\mathfrak{m}_{N})\cong\mathrm{U}_{\hbar}(\mathfrak{b}).$

We have

$\operatorname{ad}_{E_{k1}}(L_{a\ldots b}^{a\ldots
b}(u_{a}))=(-1)^{k-a-1}\delta_{a\leq k\leq
b}L_{1,a,\ldots,\hat{k},\ldots,b}^{a,\ldots,b}(u_{a}).$

By Proposition 3.1,

$L_{1,a,\ldots,\hat{k},\ldots,b}^{a,\ldots,b}(u_{a})=\sum\limits_{i=a}^{b}(-1)^{i-a}E_{i1}L_{a\ldots\hat{k}\ldots
b}^{a\ldots\hat{i}\ldots b}(u_{a}-\hbar).$

Therefore, its image in the quotient is zero unless $i=a=2$, hence for
$a\not=2$,

(5.8) $P\operatorname{ad}_{E_{ij}}(v)L_{a\ldots b}^{a\ldots
b}(u_{a})\cdot(-1)^{k+1}(-E_{k1}^{\psi})L_{1\ldots k-1}^{1\ldots
k-1}(u_{1})wP\in O(\hbar^{2})$

So, we need to determine the image of $L_{2\ldots\hat{k}\ldots b}^{3\ldots
b}(u_{2}-\hbar)$ in the quotient, which is $L_{k+1\ldots b}^{k+1\ldots
b}(u_{2}-(k-1)\hbar)$ by (5.6). Therefore, up to higher order terms, we have
for $a=2$

$\displaystyle(-1)^{k+1}P\operatorname{ad}_{E_{ij}}(v)\operatorname{ad}_{E_{k1}}(L_{2\ldots
b}^{2\ldots b}(u_{2}))L_{1\ldots k-1}^{1\ldots k-1}(u_{1})wP=$
$\displaystyle=$ $\displaystyle P\operatorname{ad}_{E_{ij}}(v)L_{k+1\ldots
b}^{k+1\ldots b}(u_{2}-(k-1)\hbar)L_{1\ldots k-1}^{1\ldots k-1}(u_{1})wP.$

Next, let $b\geq 1$ be arbitrary and $k>b+1$. Let us study the term

$P\operatorname{ad}_{E_{ij}}(v)L_{1\ldots b}^{1\ldots
b}(u_{1})\cdot(-1)^{k+1}(-E_{k1}^{\psi})L_{1\ldots k-1}^{1\ldots
k-1}(u_{1})wP.$

By exactly the same argument, it is enough to look at the image of

$\operatorname{ad}_{E_{k1}}(L_{1\ldots b}^{1\ldots
b}(u_{1}))=L_{1,2,\ldots,b}^{k,2,\ldots,b}(u_{1})=\sum\limits_{i=1}^{b}(-1)^{i-1}E_{ki}L_{1\ldots\hat{i}\ldots
b}^{2\ldots b}(u_{1}).$

in the quotient (again, we use Proposition 3.1). However, by our assumption on
$k$ and $b$, they are just zero. Therefore, the terms of this kind do not
contribute to degree one coefficient, and the first part of the lemma follows.

As for the second part, let assume first that $i>2$. It follows from (5.8)
that the coefficient $\beta_{i+k,i}$ does not depend on $N$. Therefore, it is
enough to prove the statement for $\beta_{Ni}$.

By induction assumption, we have

$\displaystyle\begin{split}&{}_{1}\beta_{Ni}=\\\ =&(-1)^{N-i+1}(L_{i\ldots
N-1}^{i\ldots N-1}(u_{i})-(u_{i}-u_{N-1})L_{i\ldots N-1}^{i\ldots
N-1}(u_{i}))-(u_{i}-u_{N-1})\cdot{}_{1}\beta_{N-1,i}+{}_{1}\beta_{N-1,i-1}.\end{split}$

It follows from arguments before that

$\beta_{Ni}={}_{1}\beta_{Ni}+\sum\limits_{k=1}^{N}(-1)^{k+1}\operatorname{ad}_{E_{k1}}({}_{1}\beta_{Ni})L_{1\ldots
k-1}^{1\ldots k-1}(u_{1}).$

But by (5.8), $\operatorname{ad}_{E_{k1}}(L_{i\ldots N-1}^{i\ldots
N-1}(u_{i}))$ does not contribute to the first power. In particular,

$\operatorname{ad}_{E_{k1}}({}_{1}\beta_{Ni})=-(u_{i}-u_{N-1})\cdot\operatorname{ad}_{E_{k1}}({}_{1}\beta_{N-1,i})+\operatorname{ad}_{E_{k1}}({}_{1}\beta_{N-1,i-1})+O(\hbar^{2}),$

and the result follows.

For $i=2$, observe that

$\displaystyle{}_{1}\beta_{N2}$ $\displaystyle=(-1)^{N}L_{2\ldots
N-1}^{2\ldots N-1}(u_{2}),$ $\displaystyle{}_{1}\beta_{N-1,2}$
$\displaystyle=(-1)^{N-1}L_{2\ldots N-2}^{2\ldots N-2}(u_{2}).$

Therefore,

$\displaystyle\beta_{N2}$ $\displaystyle=(-1)^{N}L_{2\ldots N-1}^{2\ldots
N-1}(u_{2})+(-1)^{N}\sum\limits_{k=1}^{N-1}\prod_{l=k+1}^{N-1}(u_{2}-u_{l})L_{1\ldots
k-1}^{1\ldots k-1}(u_{1}),$ $\displaystyle\beta_{N-1,2}$
$\displaystyle=(-1)^{N-1}L_{2\ldots N-2}^{2\ldots
N-2}(u_{2})+(-1)^{N-1}\sum\limits_{k=1}^{N-2}\prod_{l=k+1}^{N-2}(u_{2}-u_{l})L_{1\ldots
k-1}^{1\ldots k-1}(u_{1}),$ $\displaystyle\beta_{N-1,1}$
$\displaystyle=(-1)^{N}L_{1\ldots N-2}^{1\ldots N-2}(u_{1}).$

The statement follows by direct calculation of (LABEL:eq::beta_relation).

Finally, for $i=1$, the statement follows from
$\beta_{k1}=(-1)^{k+1}L_{1\ldots k-1}^{1\ldots k-1}(u_{1})$. ∎

Now observe that due to (5.7), the semiclassical limit of
(LABEL:eq::beta_relation) is

$\beta_{i+k,i}=-E_{i,i+k-1}-(u_{i}-u_{i+k-1})\beta_{i+k-1,i}+\beta_{i+k-1,i-1}.$

But this is exactly (2.4). The theorem follows. ∎

###  5.2. General setting

In this subsection, we formulate and show some properties of the Kostant-
Whittaker reduction in the case of the full algebra $\mathfrak{gl}_{N}$. The
proof of almost all the statements can be directly translated from the
mirabolic setting using the Kirillov projector and will be mostly omitted.

Denote by
$\mathbf{A}_{\mathfrak{gl}_{N}}:=\mathbf{A}(\mathrm{U}_{\hbar}(\mathfrak{gl}_{N}))$
the center of the universal enveloping algebra of $\mathfrak{gl}_{N}$. In this
case, it has a very explicit presentation (for instance, see [MNO96]):

######  Theorem 5.11.

Let

$L_{1\ldots N}^{1\ldots
N}(v)=v^{N}+\sum\limits_{i=0}^{N-1}(-1)^{N-i}A_{i}v^{i}.$

Then $\mathbf{A}_{\mathfrak{gl}_{N}}$ is a commutative polynomial algebra
freely generated by $\\{A_{i}\\}$.

Denote by $\mathrm{Wh}_{\hbar}(\mathfrak{gl}_{N})$ the category of
$(\mathbf{A}_{\mathfrak{gl}_{N}},\mathrm{U}_{\hbar}(\mathfrak{gl}_{N}))$-bimodules
which are Whittaker with respect to the right
$\mathrm{U}_{\hbar}(\mathfrak{gl}_{N})$-action (the definition is the same as
in the mirabolic setting of 4.1). It has a distinguished object
$Q_{\mathfrak{gl}_{N}}=\mathfrak{n}_{-}^{\psi}\backslash\mathrm{U}_{\hbar}(\mathfrak{gl}_{N})$.
In particular, we have a functor

$\mathrm{act}_{\mathbf{A}_{\mathfrak{gl}_{N}}}^{\psi}\colon{}_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{BiMod}_{\mathbf{A}_{\mathfrak{gl}_{N}}}\rightarrow\mathrm{Wh}_{\hbar}(\mathfrak{gl}_{N}),\qquad
X\mapsto X\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}Q_{\mathfrak{gl}_{N}},$

where
${}_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{BiMod}_{\mathbf{A}_{\mathfrak{gl}_{N}}}$
is the category of $\mathbf{A}_{\mathfrak{gl}_{N}}$-bimodules. The functor of
Whittaker invariants is right adjoint:

$(-)^{\mathfrak{n}_{-}^{\psi}}\colon\mathrm{Wh}_{\hbar}(\mathfrak{gl}_{N})\rightarrow{}_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{BiMod}_{\mathbf{A}_{\mathfrak{gl}_{N}}}.$

######  Theorem 5.12 (Skryabin’s equivalence).

The functor $(-)^{\mathfrak{n}_{-}^{\psi}}$ is an equivalence.

###### Proof.

The main difference with the mirabolic setting is the following fact: we need
to show that for any
$X\in{}_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{BiMod}_{\mathbf{A}_{\mathfrak{gl}_{N}}}$,
the unit of adjunction

$X\mapsto(X\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}Q_{\mathfrak{gl}_{N}})^{\mathfrak{n}_{-}^{\psi}}$

is an isomorphism. As before, the Kirillov projector gives an isomorphism

$(X\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}Q_{\mathfrak{gl}_{N}})^{\mathfrak{n}_{-}^{\psi}}\cong(X\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}Q_{\mathfrak{gl}_{N}})_{\mathfrak{b}^{\vec{u}}}.$

By the PBW theorem, we have

$(X\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}Q_{\mathfrak{gl}_{N}})/\mathfrak{b}^{\vec{u}}\cong
X\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{U}_{\hbar}(\mathfrak{u}_{N})$

with a suitable $\mathbf{A}_{\mathfrak{gl}_{N}}$-action on
$\mathrm{U}_{\hbar}(\mathfrak{u}_{N})$, see Section 4.3. But it follows from
Proposition 4.19 that the map
$\mathrm{U}_{\hbar}(\mathfrak{u}_{N})\rightarrow\mathbf{A}_{\mathfrak{gl}_{N}}$
induced by the Kirillov projector is an isomorphism. Therefore, the unit of
adjunction

$X\mapsto(X\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}Q_{\mathfrak{gl}_{N}})^{\mathfrak{n}_{-}^{\psi}}\cong(X\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}Q_{\mathfrak{gl}_{N}})/\mathfrak{b}^{\vec{u}}$

is an isomorphism as well. The rest follows by the same arguments as Theorem
5.1. ∎

Recall the category $\mathrm{HC}_{\hbar}(\mathrm{GL}_{N})$ of Harish-Chandra
bimodules over $\mathfrak{gl}_{N}$. We have a functor

$\mathrm{act}_{\mathfrak{gl}_{N}}^{\psi}\colon\mathrm{HC}_{\hbar}(G)\rightarrow\mathrm{Wh}_{\hbar}(\mathfrak{gl}_{N}),\qquad
X\mapsto
Q_{\mathfrak{gl}_{N}}\otimes_{\mathrm{U}_{\hbar}(\mathfrak{gl}_{N})}X.$

The following functor was studied in [BF08]:

######  Definition 5.13.

The _Kostant-Whittaker reduction_ functor
$\mathrm{res}_{\mathfrak{gl}_{N}}^{\psi}\colon\mathrm{HC}_{\hbar}(G)\rightarrow\mathbf{A}_{\mathfrak{gl}_{N}}$
is the composition

$\mathrm{HC}_{\hbar}(G)\xrightarrow{\mathrm{act}_{\mathfrak{gl}_{N}}}\mathrm{Wh}_{\hbar}(\mathfrak{gl}_{N})\xrightarrow{(-)^{\mathfrak{n}_{-}^{\psi}}}{}_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{BiMod}_{\mathbf{A}_{\mathfrak{gl}_{N}}}.$

Explicitly, it is given by the quantum Hamiltonian reduction
$X\mapsto(\mathfrak{n}_{-}^{\psi}\backslash X)^{\mathfrak{n}_{-}^{\psi}}$.
There is a natural lax monoidal structure on
$\mathrm{res}_{\mathfrak{gl}_{N}}^{\psi}$:

$(\mathfrak{n}_{-}^{\psi}\backslash
X)^{\mathfrak{n}_{-}^{\psi}}\otimes(\mathfrak{n}_{-}^{\psi}\backslash
Y)^{\mathfrak{n}_{-}^{\psi}}\rightarrow(\mathfrak{n}_{-}^{\psi}\backslash
X\otimes_{\mathrm{U}_{\hbar}(\mathfrak{m}_{N})}Y)^{\mathfrak{n}_{-}^{\psi}}.$

######  Theorem 5.14.

The functor $\mathrm{res}_{\mathfrak{gl}_{N}}^{\psi}$ is colimit-preserving
and monoidal.

Recall the monoidal functor
$\mathrm{free}\colon\operatorname{Rep}(G)\rightarrow\mathrm{HC}_{\hbar}(G)$.

######  Theorem 5.15.

The Kirillov projector defines a natural isomorphism

$(P_{\mathfrak{m}_{N}}(\vec{u}))_{V}\colon
V\otimes\mathbf{A}_{\mathfrak{gl}_{N}}\rightarrow\mathrm{res}_{\mathfrak{gl}_{N}}^{\psi}(\mathrm{U}_{\hbar}(\mathfrak{gl}_{N})\otimes
V).$

of _right_ $\mathbf{A}_{\mathfrak{gl}_{N}}$-modules. Similarly, there is a
natural isomorphism of _left_ modules

$\mathbf{A}_{\mathfrak{gl}_{N}}\otimes
V\rightarrow\mathrm{res}_{\mathfrak{gl}_{N}}^{\psi}(\mathrm{U}_{\hbar}(\mathfrak{gl}_{N})\otimes
V)$

######  Remark 5.16.

For a description of the bimodule structure on
$\mathrm{res}_{\mathfrak{gl}_{N}}^{\psi}(\mathrm{U}_{\hbar}(\mathfrak{gl}_{N})\otimes
V)$, see [BF08]. We will only need the fact that it is free on both sides.

######  Theorem 5.17.

There is a collection of _constant_ maps $F_{VW}(\vec{u})$ natural in
$V,W\in\operatorname{Rep}(G)$ such that the diagram

$\textstyle{(V\otimes\mathbf{A}_{\mathfrak{gl}_{N}})\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}(W\otimes\mathbf{A}_{\mathfrak{gl}_{N}})\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{F_{VW}(\vec{u})}$$\textstyle{V\otimes
W\otimes\mathbf{A}_{\mathfrak{gl}_{N}}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{\mathrm{res}_{\mathfrak{gl}_{N}}^{\psi}(\mathrm{U}_{\hbar}(\mathfrak{gl}_{N})\otimes
V)\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{res}_{\mathfrak{gl}_{N}}^{\psi}(\mathrm{U}_{\hbar}(\mathfrak{gl}_{N})\otimes
W)\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{\mathrm{res}_{\mathfrak{gl}_{N}}^{\psi}(\mathrm{U}_{\hbar}(\mathfrak{gl}_{N})\otimes
V\otimes W)}$

is commutative. The maps $F_{VW}(\vec{u})$ are the same as in Theorem 5.5, in
particular, they quantize the family of rational Cremmer-Gervais solutions
(2.4).

##  6\. Application: vertex-IRF transformation

The main reference for this section is [Kal21]. For reader’s convenience, we
repeat the main steps of the construction and refer the reader to _loc. cit._
for details.

###  6.1. _A_ vertex-IRF transformation

Let $R_{VW}(\lambda)$ be a quantum dynamical $R$-matrix as in Section 1.3.

######  Definition 6.1.

Let $S_{V}(\lambda)\in\mathrm{End}(V)$ be a collection of invertible
$\mathrm{End}(V)$-valued functions on $\mathfrak{h}^{*}$ natural in
$\operatorname{Rep}(G)$. A _generalized gauge transformation_ is

$R_{VW}(\lambda)\mapsto R^{S}_{VW}(\lambda)=(S_{V}(\lambda-h_{W})\otimes
S_{W}(\lambda))R_{VW}(\lambda)(S_{V}(\lambda)^{-1}\otimes S_{W}(\lambda-
h_{U})^{-1}).$

A generalized gauge transformation is called a _gauge transformation_ if
$S_{V}(\lambda)$ respects the $\mathfrak{h}$-module structure on $V$. A
generalized gauge transformation is called a _vertex-IRF transformation_ if
$R_{VW}^{S}(\lambda)$ is constant.

One can similarly define a gauge transformation for dynamical twists, see
[ES01].

######  Proposition 6.2.

Gauge transformations preserve the set of quantum dynamical $R$-matrices.

In the setting of Section 1.3, it has the following categorical
interpretation:

######  Proposition 6.3.

[KS22, Proposition 3.8] The data of a gauge transformation between dynamical
twists $J_{1}(\lambda)$ and $J_{2}(\lambda)$ is equivalent to the data of a
natural monoidal isomorphism:

$\textstyle{\operatorname{Rep}(G)\ignorespaces\ignorespaces}$$\textstyle{\scriptstyle
J_{1}(\lambda)}$$\textstyle{\scriptstyle
J_{2}(\lambda)}$$\textstyle{\mathrm{HC}_{\hbar}(H).}$

It is also possible to interpret generalized gauge transformations in this
language. There is a forgetful functor
$\mathrm{HC}_{\hbar}(H)\rightarrow\mathrm{RMod}_{\mathrm{U}_{\hbar}(\mathfrak{h})}$.
While it is not monoidal, it can be upgrade to a _monoidal action_ of
$\mathrm{HC}_{\hbar}(H)$ on
$\mathrm{RMod}_{\mathrm{U}_{\hbar}(\mathfrak{h})}$:

######  Definition 6.4.

Let $\mathcal{C}$ be a monoidal category. A _$\mathcal{C}$ -module category_
$\mathcal{M}$ is a category $\mathcal{M}$ with a functor
$\mathcal{M}\otimes\mathcal{C}\rightarrow\mathcal{M}$ which we denote by
$(M,X)\mapsto X\otimes M$ with a natural isomorphisms

$\Psi_{M,X,Y}\colon(M\otimes X)\otimes Y\rightarrow M\otimes(X\otimes Y),$

satisfying a pentagon axiom (see [Eti+15]). A _functor_ between
$\mathcal{C}$-module categories
$F\colon\mathcal{M}_{1}\rightarrow\mathcal{M}_{2}$ is a functor of plain
categories with natural isomorphisms

$\alpha_{V,X}\colon F(V)\otimes X\rightarrow F(V\otimes X),$

satisfying the unit and the pentagon axioms.

The Harish-Chandra category $\mathrm{HC}_{\hbar}(H)$ naturally acts on the
category $\mathrm{RMod}_{\mathrm{U}_{\hbar}(\mathfrak{h})}$ of _right_
$\mathrm{U}_{\hbar}(\mathfrak{h})$-modules by

$(M,X)\mapsto M\otimes_{\mathrm{U}_{\hbar}(\mathfrak{h})}X.$

In particular, if we have a monoidal functor
$\operatorname{Rep}(G)\rightarrow\mathrm{HC}_{\hbar}(H)$, then there is an
action of $\operatorname{Rep}(G)$ on
$\mathrm{RMod}_{\mathrm{U}_{\hbar}(\mathfrak{h})}$. One can easily see that
the structure morphisms of 6.4 are given by the dynamical twist maps
$J_{VW}(\lambda)$ as in Section 1.3.

Let $A$ be an algebra with a map
$A\rightarrow\mathrm{U}_{\hbar}(\mathfrak{h})$. Assume that there is an action
of $\operatorname{Rep}(G)$ on $\mathrm{RMod}_{A}$ such that $(A,V)\mapsto
A\otimes V$ from 6.4 is a free right $A$-module. Moreover, assume that the
structure maps $\Psi_{A,V,W}\colon(A\otimes V)\otimes Y\rightarrow
A\otimes(V\otimes W)$ are given by $\mathrm{id}_{A}\otimes F_{VW}$ for some
_constant_ $F_{VW}$. One example is $A=k$; another is the setup of Theorem
5.17.

There is an extension of scalars functor
$-\otimes_{A}\mathrm{U}_{\hbar}(\mathfrak{h})\colon\mathrm{RMod}_{A}\rightarrow\mathrm{RMod}_{\mathrm{U}_{\hbar}(\mathfrak{h})}$.

######  Theorem 6.5.

[Kal21, Proposition 2.4.3] A generalized gauge transformation between
$J_{VW}(\lambda)$ and $F_{VW}$ is equivalent to the data of a
$\operatorname{Rep}(G)$-module structure on the functor
$-\otimes_{A}\mathrm{U}_{\hbar}(\mathfrak{h})$.

###  6.2. _The_ vertex-IRF transformation

Recall a monoidal functor
$\operatorname{Rep}(\mathrm{GL}_{N})\rightarrow\mathrm{HC}_{\hbar}(H)^{\mathrm{gen}}$
from Section 1.5 obtained from parabolic restriction whose monoidal structure
morphisms are given by the standard dynamical twist of 1.16. In particular,
there is an action of $\operatorname{Rep}(\mathrm{GL}_{N})$ on the category
$\mathrm{RMod}_{\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}}$ of right
$\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}$-modules, whose structure
morphisms are given by the standard dynamical twist. Likewise, there is an
action of $\operatorname{Rep}(\mathrm{GL}_{N})$ on the category
$\mathrm{RMod}_{\mathbf{A}_{\mathfrak{gl}_{N}}}$ of right modules over the
center $\mathbf{A}_{\mathfrak{gl}_{N}}$ whose structure morphisms are given by
the rational Cremmer-Gervais twist of 5.8. Also recall the Harish-Chandra
homomorphism
$\mathbf{A}_{\mathfrak{gl}_{n}}\rightarrow\mathrm{U}_{\hbar}(\mathfrak{h})$.
To show that there is a vertex-IRF transformation between them, it is enough
to prove by Theorem 6.5 (or, rather, generic version thereof) that the natural
functor
$-\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}\colon\mathrm{RMod}_{\mathbf{A}_{\mathfrak{gl}_{N}}}\rightarrow\mathrm{RMod}_{\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}}$
is a functor of module categories.

Recall that the parabolic restriction $\mathrm{res}_{\mathfrak{gl}_{N}}$ is a
functor
$\mathrm{HC}_{\hbar}(\mathrm{GL}_{N})\rightarrow\mathrm{HC}_{\hbar}(H)^{\mathrm{gen}}$.
The Harish-Chandra category $\mathrm{HC}_{\hbar}(H)^{\mathrm{gen}}$ is a
subcategory of
${}_{\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}}\mathrm{BiMod}_{\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}}$,
and there is restriction functor from the latter to
${}_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{BiMod}_{\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}}$.
In what follows, we consider $\mathrm{res}_{\mathfrak{gl}_{N}}$ to take values
in
${}_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{BiMod}_{\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}}$.

######  Proposition 6.6.

[Kal21, Corollary 2.7.7] There is a natural isomorphism

$\mathrm{res}_{\mathfrak{gl}_{N}}\rightarrow\mathrm{res}_{\mathfrak{gl}_{N}}^{\psi}\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}$

of functors
$\mathrm{HC}_{\hbar}(G)\rightarrow{}_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{BiMod}_{\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}}$.

###### Proof.

Let $X\in\mathrm{HC}_{\hbar}(\mathrm{GL}_{N})$. Consider a sequence of maps

(6.1) $(\mathfrak{n}_{-}^{\psi}\backslash
X)^{\mathfrak{n}_{-}^{\psi}}\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{U}_{\hbar}(\mathfrak{h})\xrightarrow{p_{X}\otimes\mathrm{id}_{\mathrm{U}_{\hbar}(\mathfrak{h})}}(\mathfrak{n}_{-}^{\psi}\backslash
X\mathfrak{n})\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{U}_{\hbar}(\mathfrak{h})\xrightarrow{\mathrm{act}}\mathfrak{n}_{-}^{\psi}\backslash
X/\mathfrak{n},$

where $p_{X}$ is the projection and $\mathrm{act}$ is the right action of
$\mathrm{U}_{\hbar}(\mathfrak{h})$ on $\mathfrak{n}_{-}^{\psi}\backslash
X/\mathfrak{n}$. By [GK22, Lemma 6.2.1], it is an isomorphism. On the other
hand, there is a map

(6.2)
$(X/\mathfrak{n})^{\mathfrak{n}}\rightarrow\mathfrak{n}_{-}^{\psi}\backslash
X/\mathfrak{n}.$

We claim that after extension of scalars to
$\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}$, it is an isomorphism.
Indeed: since $\mathrm{HC}_{\hbar}(\mathrm{GL}_{N})$ is generated by free
Harish-Chandra bimodules by Proposition 1.4 and both functors in question are
colimit-preserving by Theorem 1.13 and Theorem 5.14, it is enough to check the
statement for free Harish-Chandra bimodules
$X=\mathrm{U}_{\hbar}(\mathfrak{g})\otimes V$. There is an
$\mathfrak{n}$-stable filtration on $V$ with one-dimensional quotients; in
particular, we can choose a (weight) basis $\\{v_{\mu}\\}$ and a partial order
on it such that $\operatorname{ad}_{\mathfrak{n}}(v_{\mu})>v_{\mu}$.

By the PBW theorem, we have a right $\mathrm{U}_{\hbar}(\mathfrak{h})$-module
isomorphism

$\mathfrak{n}_{-}^{\psi}\backslash
X/\mathfrak{n}\cong\mathrm{U}_{\hbar}(\mathfrak{h})\otimes V,$

given by the generators $\\{1\otimes v_{\mu}\\}$. Likewise, the extremal
projector gives an isomorphism

$P\colon\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}\otimes
V\rightarrow(X/\mathfrak{n})^{\mathfrak{n}}$

as in Theorem 1.14. It follows from definition of $P$ that

$P(1\otimes v_{\mu})=1\otimes v_{\mu}+\sum_{\lambda>\mu}f_{\lambda}\otimes
v_{\lambda}\in(X/\mathfrak{n})\otimes_{\mathrm{U}_{\hbar}(\mathfrak{h})}\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}\cong\mathrm{U}_{\hbar}(\mathfrak{b})^{\mathrm{gen}}\otimes
V,$

where $f_{\lambda}\in\mathrm{U}_{\hbar}(\mathfrak{b})$ are some elements. In
particular, its class in the left quotient by $\mathfrak{n}_{-}^{\psi}$ is
given by some strictly upper-triangular transformation in the basis
$\\{1\otimes v_{\mu}\\}$ with coefficients in
$\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}$, in particular, it is
invertible.

Therefore, composing (6.1) with the inverse of (6.2), we conclude. ∎

######  Remark 6.7.

In particular, (6.1) shows why it was natural to consider right
$\mathrm{U}_{\hbar}(\mathfrak{gl}_{N})$-modules in the Whittaker setting
instead of left ones.

######  Proposition 6.8.

The functor
$\mathrm{RMod}_{\mathbf{A}_{\mathfrak{gl}_{N}}}\rightarrow\mathrm{RMod}_{\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}},\
M\mapsto
M\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{U}_{\hbar}(\mathfrak{h})^{\mathrm{gen}}$
is a functor of $\mathrm{HC}_{\hbar}(\mathrm{GL}_{N})$-module categories, in
particular of $\operatorname{Rep}(\mathrm{GL}_{N})$-module categories.

###### Proof.

Since the category $\mathrm{RMod}_{\mathbf{A}_{\mathfrak{gl}_{N}}}$ is
generated by free modules, it is enough to check compatibility of actions on
$\mathbf{A}_{\mathfrak{gl}_{N}}$. By Proposition 6.6, we have an isomorphism

$\mathrm{res}^{\psi}_{\mathfrak{gl}_{N}}(X)\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{U}_{\hbar}(\mathfrak{h})\rightarrow\mathrm{res}_{\mathfrak{gl}_{N}}(X)$

(for brevity, we omit the index “gen”). Therefore, we only need to check the
pentagon axiom:

$\textstyle{\mathrm{res}^{\psi}_{\mathfrak{gl}_{N}}(X)\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{res}^{\psi}_{\mathfrak{gl}_{N}}(Y)\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{U}_{\hbar}(\mathfrak{h})\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{\mathrm{res}^{\psi}_{\mathfrak{gl}_{N}}(X\otimes_{\mathrm{U}_{\hbar}(\mathfrak{g})}Y)\otimes_{\mathbf{A}_{\mathfrak{gl}_{N}}}\mathrm{U}_{\hbar}(\mathfrak{h})\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{(X/\mathfrak{n})^{\mathfrak{n}}\otimes_{\mathrm{U}_{\hbar}\mathfrak{h}}(Y/\mathfrak{n})^{\mathfrak{n}}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{(X\otimes_{\mathrm{U}_{\hbar}\mathfrak{g}}Y/\mathfrak{n})^{\mathfrak{n}}}$

In terms of elements, let
${}_{\mathfrak{n}^{\psi}_{-}}[y]\in(\mathfrak{n}_{-}^{\psi}\backslash
Y)^{\mathfrak{n}_{-}^{\psi}}$ and
${}_{\mathfrak{n}^{\psi}_{-}}[x]\in(\mathfrak{n}_{-}^{\psi}\backslash
X)^{\mathfrak{n}_{-}^{\psi}}$. We denote their image in the double quotient
$\mathfrak{n}_{-}^{\psi}\backslash Y/\mathfrak{n}$ by
${}_{\mathfrak{n}^{\psi}_{-}}[y]_{\mathfrak{n}}$ (same for $X$). Let $F$ be
the inverse of (6.2). Then we need to show that

$F({}_{\mathfrak{n}^{\psi}_{-}}[x]_{\mathfrak{n}})\otimes_{\mathrm{U}_{\hbar}\mathfrak{g}}F({}_{\mathfrak{n}^{\psi}_{-}}[y]_{\mathfrak{n}})=F({}_{\mathfrak{n}^{\psi}_{-}}[x\otimes_{\mathrm{U}_{\hbar}(\mathfrak{g})}y]_{\mathfrak{n}}).$

We show that the projections of both sides to
$\mathfrak{n}_{-}^{\psi}\backslash(X\otimes_{\mathrm{U}_{\hbar}(\mathfrak{g})}Y)/\mathfrak{n}$
coincide. Indeed: since $F$ is an isomorphism, we have

$F({}_{\mathfrak{n}^{\psi}_{-}}[y]_{\mathfrak{n}})\in[y]_{\mathfrak{n}}+\mathfrak{n}_{-}^{\psi}Y/\mathfrak{n}.$

The element ${}_{\mathfrak{n}_{-}^{\psi}}[x]$ is
$\mathfrak{n}_{-}^{\psi}$-invariant, i.e.
$x\cdot\mathfrak{n}_{-}^{\psi}\in\mathfrak{n}_{-}^{\psi}X$, therefore, the
left-hand side belongs to

$[x\otimes_{\mathrm{U}_{\hbar}(\mathfrak{g})}y]_{\mathfrak{n}}+\mathfrak{n}_{-}^{\psi}(X\otimes_{\mathrm{U}_{\hbar}\mathfrak{g}}Y)/\mathfrak{n},$

and so the projections to both sides coincide. ∎

In particular, it implies

######  Theorem 6.9.

There is a vertex-IRF transformation between the standard quantum dynamical
$R$-matrix $R^{\mathrm{dyn}}(\lambda)$ from 1.16 and the constant quantum
rational Cremmer-Gervais $R$-matrix $R^{\mathrm{CG}}(\vec{u})$ from 5.8.

## References

  * [AR94] Jiří Adámek and Jiří Rosický  “Locally presentable and accessible categories” 189, London Mathematical Society Lecture Note Series  Cambridge University Press, Cambridge, 1994, pp. xiv+316  DOI: 10.1017/CBO9780511600579
  * [Ara17] Tomoyuki Arakawa  “Introduction to W-algebras and their representation theory”  In _Perspectives in Lie theory_ 19, Springer INdAM Ser.  Springer, Cham, 2017, pp. 179–250
  * [AST71] R.. Asherova, Ju.. Smirnov and V.. Tolstoy  “Projection operators for simple Lie groups”  In _Teoret. Mat. Fiz._ 8.2, 1971, pp. 255–271
  * [Bax89] Rodney J. Baxter  “Exactly solved models in statistical mechanics” Reprint of the 1982 original  Academic Press, Inc. [Harcourt Brace Jovanovich, Publishers], London, 1989, pp. xii+486
  * [BCJ15] Martin Brandenburg, Alexandru Chirvasitu and Theo Johnson-Freyd  “Reflexivity and dualizability in categorified linear algebra”  In _Theory Appl. Categ._ 30, 2015, pp. Paper No. 23808–835  arXiv:1409.5934 [math.CT]
  * [BDF90] J. Balog, L. Dąbrowski and L. Fehér  “Classical $r$-matrix and exchange algebra in WZNW and Toda theories”  In _Phys. Lett. B_ 244.2, 1990, pp. 227–234  DOI: 10.1016/0370-2693(90)90060-J
  * [BF08] Roman Bezrukavnikov and Michael Finkelberg  “Equivariant Satake category and Kostant-Whittaker reduction”  In _Mosc. Math. J._ 8.1, 2008, pp. 39–72183  DOI: 10.17323/1609-4514-2008-8-1-39-72
  * [BK06] Jonathan Brundan and Alexander Kleshchev  “Shifted Yangians and finite $W$-algebras”  In _Adv. Math._ 200.1, 2006, pp. 136–195  DOI: 10.1016/j.aim.2004.11.004
  * [BRT07] E. Buffenoir, Ph. Roche and V. Terras  “Quantum dynamical coboundary equation for finite dimensional simple Lie algebras”  In _Adv. Math._ 214.1, 2007, pp. 181–229  DOI: 10.1016/j.aim.2007.02.001
  * [BRT12] E. Buffenoir, Ph. Roche and V. Terras  “Universal vertex-IRF transformation for quantum affine algebras”  In _J. Math. Phys._ 53.10, 2012, pp. 10351553  DOI: 10.1063/1.4754699
  * [CG90] Eugène Cremmer and Jean-Loup Gervais  “The quantum group structure associated with nonlinearly extended Virasoro algebras”  In _Comm. Math. Phys._ 134.3, 1990, pp. 619–632  URL: http://projecteuclid.org/euclid.cmp/1104201824
  * [Del90] P. Deligne  “Catégories tannakiennes”  In _The Grothendieck Festschrift, Vol. II_ 87, Progr. Math.  Birkhäuser Boston, Boston, MA, 1990, pp. 111–195
  * [Dri85] V.. Drinfel~ d  “Hopf algebras and the quantum Yang-Baxter equation”  In _Dokl. Akad. Nauk SSSR_ 283.5, 1985, pp. 1060–1064
  * [Dri87] V.. Drinfeld  “Quantum groups”  In _Proceedings of the International Congress of Mathematicians, Vol. 1, 2 (Berkeley, Calif., 1986)_ Amer. Math. Soc., Providence, RI, 1987, pp. 798–820
  * [EH00] Robin Endelman and Timothy J. Hodges  “Generalized Jordanian $R$-matrices of Cremmer-Gervais type”  In _Lett. Math. Phys._ 52.3, 2000, pp. 225–237  DOI: 10.1023/A:1007693603336
  * [ES01] Pavel Etingof and Olivier Schiffmann  “Lectures on the dynamical Yang-Baxter equations”  In _Quantum groups and Lie theory (Durham, 1999)_ 290, London Math. Soc. Lecture Note Ser.  Cambridge Univ. Press, Cambridge, 2001, pp. 89–129  arXiv:math/9908064
  * [ES02] Pavel Etingof and Olivier Schiffmann  “Lectures on quantum groups”, Lectures in Mathematical Physics  International Press, Somerville, MA, 2002, pp. xii+242
  * [Eti+15] Pavel Etingof, Shlomo Gelaki, Dmitri Nikshych and Victor Ostrik  “Tensor categories” 205, Mathematical Surveys and Monographs  American Mathematical Society, Providence, RI, 2015, pp. xvi+343  DOI: 10.1090/surv/205
  * [Eti02] Pavel Etingof  “On the dynamical Yang-Baxter equation”  In _Proceedings of the International Congress of Mathematicians, Vol. II (Beijing, 2002)_ Higher Ed. Press, Beijing, 2002, pp. 555–570  arXiv:math/0207008
  * [EV99] P. Etingof and A. Varchenko  “Exchange dynamical quantum groups”  In _Comm. Math. Phys._ 205.1, 1999, pp. 19–52  DOI: 10.1007/s002200050665
  * [Fel95] Giovanni Felder  “Elliptic quantum groups”  In _XIth International Congress of Mathematical Physics (Paris, 1994)_ Int. Press, Cambridge, MA, 1995, pp. 211–218  arXiv:hep-th/9412207
  * [GGS92] Murray Gerstenhaber, Anthony Giaquinto and Samuel D. Schack  “Quantum symmetry”  In _Quantum groups (Leningrad, 1990)_ 1510, Lecture Notes in Math.  Springer, Berlin, 1992, pp. 9–46  DOI: 10.1007/BFb0101176
  * [GK22] Victor Ginzburg and David Kazhdan  “Differential operators on $G/U$ and the Gelfand-Graev action”  In _Adv. Math._ 403, 2022, pp. Paper No. 10836848  DOI: 10.1016/j.aim.2022.108368
  * [GK75] I.. Gel~ fand and D.. Kajdan  “Representations of the group ${\mathrm{GL}}(n,K)$ where $K$ is a local field”  In _Lie groups and their representations (Proc. Summer School, Bolyai János Math. Soc., Budapest, 1971)_ Halsted, New York, 1975, pp. 95–118
  * [GN84] J.-L. Gervais and A. Neveu  “Novel triangle relation and absence of tachyons in Liouville string field theory”  In _Nuclear Phys. B_ 238.1, 1984, pp. 125–141  DOI: 10.1016/0550-3213(84)90469-3
  * [Hod95] Timothy J. Hodges  “On the Cremmer-Gervais quantizations of ${\rm SL}(n)$”  In _Internat. Math. Res. Notices_ , 1995, pp. 465–481  DOI: 10.1155/S107379289500033X
  * [Ioh96] Kenji Iohara  “Bosonic representations of Yangian double $\mathcal{D}Y_{\hbar}(\mathfrak{g})$ with $\mathfrak{g}=\mathfrak{gl}_{N},\mathfrak{sl}_{N}$”  In _J. Phys. A_ 29.15, 1996, pp. 4593–4621  DOI: 10.1088/0305-4470/29/15/027
  * [Jim85] Michio Jimbo  “A $q$-difference analogue of $U(\mathfrak{g})$ and the Yang-Baxter equation”  In _Lett. Math. Phys._ 10.1, 1985, pp. 63–69  DOI: 10.1007/BF00704588
  * [Kal21] Artem Kalmykov  “Geometric and categorical approaches to dynamical representation theory”  Zürich: Universität Zürich, 2021
  * [Kho04] S.. Khoroshkin  “An extremal projector and a dynamical twist”  In _Teoret. Mat. Fiz._ 139.1, 2004, pp. 158–176  DOI: 10.1023/B:TAMP.0000022749.42512.fd
  * [Kir63] A.. Kirillov  “Infinite-dimensional unitary representations of a second-order matrix group with elements in a locally compact field”  In _Dokl. Akad. Nauk SSSR_ 150, 1963, pp. 740–743
  * [Kos78] Bertram Kostant  “On Whittaker vectors and representation theory”  In _Invent. Math._ 48.2, 1978, pp. 101–184  DOI: 10.1007/BF01390249
  * [KS22] Artem Kalmykov and Pavel Safronov  “A categorical approach to dynamical quantum groups”  In _Forum Math. Sigma_ 10, 2022, pp. Paper No. e7657  DOI: 10.1017/fms.2022.68
  * [KST98] S.. Khoroshkin, A.. Stolin and V.. Tolstoy  “Deformation of Yangian $Y({\rm sl}_{2})$”  In _Comm. Algebra_ 26.4, 1998, pp. 1041–1055  DOI: 10.1080/00927879808826182
  * [Los10] Ivan Losev  “Finite W-algebras”  In _Proceedings of the International Congress of Mathematicians. Volume III_ Hindustan Book Agency, New Delhi, 2010, pp. 1281–1307
  * [MNO96] A. Molev, M. Nazarov and G. Olshanskii  “Yangians and classical Lie algebras”  In _Uspekhi Mat. Nauk_ 51.2(308), 1996, pp. 27–104  DOI: 10.1070/RM1996v051n02ABEH002772
  * [Mol07] Alexander Molev  “Yangians and classical Lie algebras” 143, Mathematical Surveys and Monographs  American Mathematical Society, Providence, RI, 2007, pp. xviii+400  DOI: 10.1090/surv/143
  * [Pre02] Alexander Premet  “Special transverse slices and their enveloping algebras” With an appendix by Serge Skryabin  In _Adv. Math._ 170.1, 2002, pp. 1–55  DOI: 10.1006/aima.2001.2063
  * [RS99] E. Ragoucy and P. Sorba  “Yangian realisations from finite $W$-algebras”  In _Comm. Math. Phys._ 203.3, 1999, pp. 551–572  DOI: 10.1007/s002200050034
  * [Sev00] A. Sevostyanov  “Quantum deformation of Whittaker modules and the Toda lattice”  In _Duke Math. J._ 105.2, 2000, pp. 211–238  DOI: 10.1215/S0012-7094-00-10522-4
  * [Zhe90] D.. Zhelobenko  “An introduction to the theory of $S$-algebras over reductive Lie algebras”  In _Representation of Lie groups and related topics_ 7, Adv. Stud. Contemp. Math.  GordonBreach, New York, 1990, pp. 155–221


# Density Estimation on the Binary Hypercube using Transformed Fourier-Walsh
Diagonalizations

Arthur C. Campello
Department of Applied Physics
Stanford University
Stanford, CA 94305
<EMAIL_ADDRESS>

###### Abstract

This article focuses on estimating distribution elements over a high-
dimensional binary hypercube from multivariate binary data. A popular approach
to this problem, optimizing Walsh basis coefficients, is made more
interpretable by an alternative representation as a "Fourier-Walsh"
diagonalization. Allowing monotonic transformations of the resulting matrix
elements yields a versatile binary density estimator: the main contribution of
this article. It is shown that the Aitchison and Aitken kernel emerges from a
constrained exponential form of this estimator, and that relaxing these
constraints yields a flexible variable-weighted version of the kernel that
retains positive-definiteness. Estimators within this unifying framework mix
together well and span over extremes of the speed-flexibility trade-off,
allowing them to serve a wide range of statistical inference and learning
problems.

_Keywords_ density estimation $\cdot$ binary hypercube $\cdot$ Walsh basis
$\cdot$ Aitchison and Aitken kernel $\cdot$ diagonalization $\cdot$ positive-
definite kernel

##  1 Introduction

Though less intuitive than continuous data, high-dimensional binary data
appear ubiquitously in modern statistical learning and artificial
intelligence. In medicine, critical pieces of information ranging from ocular
features [1] to drug trial data to gene expression [2] take binary forms.
Other common binary data types include connectivity and node activation in
social and epidemiological networks, survey data, and multivariate binary time
series [3]. In machine learning and artificial intelligence (AI) applications,
binary data often contain important feature information gleaned from more
complex data sets. Binary features have long been used in word image retrieval
[4] and are used in learning models for facial recognition [5] and fall
detection [6]. Progress in drawing insights from binary data advances both
areas dealing with direct and intermediate-form binary information.

This article addresses the estimation of multivariate binary densities from
independent observations of $n$-dimensional binary variables whose support is
the $\\{-1,1\\}^{n}$ hypercube. These densities elucidate complex dependencies
across variables and inform conditional probabilities that directly serve
statistical learning applications. While estimating all $2^{n}$ probabilities
over the hypercube becomes prohibitively expensive for large $n$, small
subsets can still provide strong insights about a dataset. From only two
density elements, for example, one can find the expected value of a binary
response variable conditional on a specific input from multiple regressor
variables.

In applied settings, the effectiveness of a density estimation scheme can be
assessed by three metrics: speed, interpretability, and flexibility. Although
these qualities often trade off, practical estimators should run in reasonable
time, make inferences in understandable ways, and adapt well to datasets of
varying sizes, dimensions, and sparsities. An early triumph in binary density
estimation with these attributes came from Aitchison and Aitken’s (AA) kernel
[1]. The AA kernel measures the proximity between two binary vectors by
scaling and exponentiating the number of agreeing indexes between them. For
dimension $n$ and number of observations $N$ The approach needs only $O(nN)$
time to estimate a density component. Furthermore, the AA kernel function is
shown to be a positive definite kernel over binary spaces, which means it is a
reproducing kernel Hilbert space (RKHS) method [7] and therefore can leverage
a representer theorem. The estimator has been shown to work well for sparse
data [8], but suffers from a flexibility limitation because it has one
smoothing parameter for all hypercube dimensions. This rigidity compromises
the estimator in cases where densities depend more on agreements along some
dimensions than others.

An alternative approach to binary density estimation uses a weighted sum of
orthogonal functions, typically Walsh functions, to find density components
[9, 10]. This approach estimates coefficients of a density function’s Fourier-
Walsh expansion. Since $2^{n}$ functions exist for $n$ dimensions, optimizing
every Walsh coefficient becomes impossible when $n$ is large. At the expense
of the method’s high flexibility, one may estimate coefficients sparingly or
in groups. A notable example of the latter involves using recursive block
thresholding to find coefficients in probabilistic polynomial time under
certain sparsity conditions [11]. Even at extremes of this trade-off, the
Fourier-Walsh approach requires more computation than the AA kernel estimator,
but proves more versatile due to its higher parameter count.

While the Fourier-Walsh and AA kernel estimators appear fundamentally
distinct, this article shows that a simple transformation of a restricted
version of the former yields the latter. It generalizes such transformations
with a guarantee of normalization, yielding a powerful binary density
estimator whose parameterizations place it on various points of the speed-
flexibility trade-off. The resulting estimator uses an interpretable kernel
that measures similarity between two $n$-length binary vectors by a signed and
weighted sum of $2^{n}$ variable products acted upon by a monotonic function.

Ways to enhance practical usage of Fourier-Walsh and AA kernel estimators also
become apparent. By transparently matching Fourier-Walsh coefficients to
corresponding variable products, the construction enables an interpretability-
first approach to prioritizing Walsh coefficient optimizations. For the AA
kernel, the general estimator’s form elucidates an extension of the method to
a more flexible dimension-weighted form without compromising its normalization
or positive definiteness.

The article is structured as follows: It first demonstrates an intuitive
derivation of naturally-ordered Walsh matrices based on how they translate
probabilities over the $\\{-1,1\\}^{n}$ hypercube – mapped to $2^{n}$-length
probability vector – onto expectation values of binary variables and products
among them. It justifies using shrinkage coefficients when estimating these
expectations from data and shows the proportionality of these to Walsh
coefficients. This yields a matrix diagonalization formulation of the Fourier-
Walsh estimator with interpretable Walsh coefficients as eigenvalues. From
here the article presents an element-wise monotonic transformation of this
diagonalization, using special Walsh matrix properties to guarantee the
resulting estimator’s normalization. This generalized form allows the
estimator to incorporate the wide range of activation functions common in
machine learning, by themselves or as mixtures. It is then shown that the
exponential transformation case with restricted Walsh coefficients yields an
AA kernel matrix. Relaxing constraints on the pre-transformation Walsh
coefficients introduces a variable-weighted extension of the AA kernel that
retains non-negativity and positive definiteness. Following this, the article
compares the times required to evaluate to different leave-one-out cross-
validation risk functions across variants of the general estimator. It
concludes with a discussion of regimes where variants of the general estimator
apply and future work invited by the presented estimation approach.

##  2 Estimation Theory

Binary density estimators take in, as inputs, observed binary data points in
$n$ dimensions assumed to be i.i.d samples of a binary random variable
$X=(X_{1},\dots,X_{n})$, $X_{i}\in\\{-1,1\\}$. The data are used to estimate
elements of the distribution of $X$, consisting of $2^{n}$ nonnegative
probability values that sum to one. Even without calculating estimates for all
density elements, an effective estimator should guarantee nonnegativity and
normalization of its complete output.

The naive approach to this problem simply estimates probabilities by their
relative frequencies in the data. These estimates often severely overfit data
and especially prove to be ineffective in high dimensions. Practical
estimators instead use data to make extrapolative inferences on density
elements beyond those corresponding to observations. To ensure reasonable
density extrapolations, one can impose typical constraints of neutrality and
symmetry. Neutrality means that the effects of new data on the density
estimates are independent of existing data. Given two datasets $D$ and
$D^{\prime}$, for example, an estimator $\hat{f}^{D}_{X}$ with the property

$\hat{f}^{D\cup
D^{\prime}}_{X}=\frac{1}{|D|+|D^{\prime}|}\left(|D|\hat{f}^{D}_{X}+|D^{\prime}|\hat{f}^{D^{\prime}}_{X}\right)$
(1)

adheres to neutrality. Noting that dataset $D$ could contain a single
observation, it becomes clear that this property restricts the estimator to a
discrete kernel form. The symmetry constraint means that, given two distinct
points in the hypercube $\\{-1,1\\}^{n}$, an observation at one point should
affect the density estimate at the other in the same way as in the case with
the points swapped. All estimators this article presents adhere to both
constraints.

###  2.1 Fourier-Walsh Estimation in Matrix Form

More than a hundred years ago, Joseph L. Walsh cleverly devised a complete set
of orthogonal functions $\phi_{k}$ for $k\in\mathbb{N}$ that yield basis
functions in discrete spaces of size $2^{n},\>n\in\mathbb{N}$ [12].
Specifically, Walsh functions allow one to represent a binary density $f$
defined for $x\in\\{0,1\\}^{n}$ as

$f(x)=\sum_{k\in\\{0,1\\}^{n}}c_{k}\phi_{k}(x),\qquad\phi_{k}(x)=(-1)^{\sum_{i}x_{i}k_{i}}.$
(2)

Here, the coefficients $c_{k}$ act similarly to the coefficients in a Fourier
series in that each encodes information pertaining to the entire distribution
rather than a single element. This means that estimating only a subset of the
involved $2^{n}$ Walsh coefficients can yield meaningful densities over the
entire binary hypercube. For this reason, early and recent research on binary
densities involves estimation methods using the Walsh basis [11, 9, 10]. This
article refers to estimators in this class as "Fourier-Walsh" estimators.

Although Walsh coefficients often appear as abstract parameters of equal
intrinsic importance in literature [11] they carry interpretable information
when the components of $X$ are themselves meaningful. Through an intuitive re-
derivation of the Walsh decomposition, it is shown that each corresponds to a
product of elements in a unique subset of $\\{X_{1},\dots,X_{n}\\}$ when using
the binary support $X_{i}\in\\{-1,1\\}$. Thus, some Walsh coefficients carry
meaning closely corresponding to input features, while others carry more
contrived information encoding products of possibly many features.

Suppose a random vector $\mathbf{r}^{(n)}_{X}\in\\{-1,1\\}^{2^{n}}$ containing
the products of all $2^{n}$ subsets (including the empty set) of
$\\{X_{1},\dots,X_{n}\\}$, generated recursively as

$\mathbf{r}^{(0)}_{X}=[1],\qquad\mathbf{r}^{(n+1)}_{X}=\begin{bmatrix}\mathbf{r}^{(n)}_{X}\\\
X_{n+1}\mathbf{r}^{(n)}_{X}\end{bmatrix}.$ (3)

From this construction, one can also recursively generate a matrix
$W^{(n)}\in\\{-1,1\\}^{2^{n}\times 2^{n}}$ whose columns contain the support
of $\mathbf{r}^{(n)}_{X}$. This takes the form

$W^{(0)}=[1],\qquad W^{(n+1)}=\begin{bmatrix}W^{(n)}&W^{(n)}\\\
W^{(n)}&-W^{(n)}\end{bmatrix}\implies
W^{(n)}=\bigotimes^{n}_{i=1}\begin{bmatrix}1&1\\\ 1&-1\end{bmatrix}.$ (4)

Interestingly, this matrix is a naturally ordered (or Haramard) Walsh matrix.
It is symmetric and has the important property
$W^{(n)}\left[W^{(n)}\right]^{\intercal}=\left[W^{(n)}\right]^{2}=2^{n}I_{2^{n}}$.
It follows from these definitions that
$\mathbb{E}\left[\mathbf{r}^{(n)}_{X}\right]=W^{(n)}\mathbf{p}$, where

$p_{j}=\mathbb{P}\left\\{\mathbf{r}^{(n)}_{X}=W^{(n)}_{:,j}\right\\}.$ (5)

Noting that the construction of $\mathbf{r}^{(n)}_{X}$ means
$\left[\mathbf{r}^{(n)}_{X}\right]_{2^{k-1}+1}=X_{k}$, one can also write

$p_{j}=\mathbb{P}\left\\{\bigcap^{n}_{k=1}\left(X_{k}=W^{(n)}_{2^{k-1}+1,j}\right)\right\\}.$
(6)

The vector $\mathbf{p}$ encodes the probability distribution of $X$ over the
hypercube as a vector and must satisfy the constraints
$\mathbf{1}^{\intercal}\mathbf{p}=1$ and $\mathbf{p}\succeq\mathbf{0}$. From
here forward, $\hat{\mathbf{p}}$ refers to the estimator of $\mathbf{p}$. This
vector mapping also defines a normalized "counts" vector $\mathbf{p}_{k}$ that
encodes observed instances of $X$.

Given these definitions, $\left[W^{(n)}\mathbf{p}_{k}\right]_{j}$ encodes the
sample mean of the product $\left[\mathbf{r}^{(n)}_{X}\right]_{j}$ and
$\left[W^{(n)}\hat{\mathbf{p}}\right]_{j}$ the expectation of
$\left[\mathbf{r}^{(n)}_{X}\right]_{j}$ associated with the estimation of the
binary density. For now, let $\left[W^{(n)}\hat{\mathbf{p}}\right]_{j}$ be
equal to $\left[W^{(n)}\mathbf{p}_{k}\right]_{j}$ multiplied by a shrinkage
factor $b_{j}\in[0,1]$. This means
$W^{(n)}\hat{\mathbf{p}}=\mathrm{diag}(\mathbf{b})W^{(n)}\mathbf{p}_{k}$,
where $\mathbf{b}\in[0,1]^{2^{n}}$ is now a shrinkage vector and produces the
estimator

$\hat{\mathbf{p}}=\frac{1}{2^{n}}W^{(n)}\mathrm{diag}(\mathbf{b})W^{(n)}\mathbf{p}_{k}.$
(7)

This form requires constraints on $\mathbf{b}$ to ensure $\hat{\mathbf{p}}$ is
a true density. Rearranging Equation 7 gives
$\mathbf{b}=\mathrm{diag}^{-1}(W^{(n)}\mathbf{p}_{k})W^{(n)}\hat{\mathbf{p}}.$
Since the first row of this matrix is trivially $\mathbf{1}^{\intercal}$, the
equality means $\mathbf{1}^{\intercal}\hat{\mathbf{p}}=1\iff b_{1}=1$. For
$\hat{\mathbf{p}}\succeq\mathbf{0}$ to hold, $\mathbf{b}$ must be a convex
combination of the columns of
$\left[\mathrm{diag}^{-1}(W^{(n)}\mathbf{p}_{k})W^{(n)}\right]$.
Geometrically, this means $\mathbf{b}_{2:2^{n}}$ must lie inside the simplex
formed by the columns of
$\left[\mathrm{diag}^{-1}(W^{(n)}\mathbf{p}_{k})W^{(n)}\right]$ with its first
row removed.

Equation 7 is equivalent to a Fourier-Walsh estimator with coefficients
proportional to elements $b_{j}\left[W^{(n)}\mathbf{p}_{k}\right]_{j}$. Here,
the constraint $b_{j}\in[0,1]$ is justified in addition to the aforementioned
constraints on $\mathbf{b}$. The chosen binary basis $X_{i}\in\\{-1,1\\}$
means $\left[\mathbf{r}^{(n)}_{X}\right]_{j}\in\\{-1,1\\}$ and
$\left[W^{(n)}\mathbf{p}_{k}\right]_{j}$ represents its mean from binary
samples. Suppose an analogous univariate random variable $Y\in\\{-1,1\\}$ with
$\mathbb{E}[Y]=q$ and $N$ observed outcomes of $Y$ with a sample mean
$\bar{y}$. If one estimates $q$ using a factor $b$ as $\hat{q}=b\bar{y}$, then
the square error minimizing $b^{*}$ given a true $q$ is

$b^{*}=\underset{b}{\mathrm{argmin}}\>\mathbb{E}_{q}\left[\left(b\bar{y}-q\right)^{2}\right]=q\frac{\mathbb{E}\left[\bar{y}\right]}{\mathbb{E}\left[\bar{y}^{2}\right]}=\frac{q^{2}}{\mathbb{E}\left[\bar{y}^{2}\right]}=\frac{Nq^{2}}{(N-1)q^{2}+1}.$
(8)

This equation constrains $b^{*}\in[0,1]\>\forall\>q\in[-1,1],N\in\mathbb{N}$.
As expected, $b^{*}(-1)=b^{*}(1)=1$ and $b^{*}(0)=0$; note that the optimal
$b^{*}$ varies substantially over $q$ even for large values of $N$. This
shrinkage technique works similarly to others in applied statistics, such as
the James–Stein estimator [13] and lasso and ridge regressions. Note that the
case $\mathbf{b}=\mathbf{1}$ corresponds to no regularization and reduces
Equation 7 to $\hat{\mathbf{p}}=\mathbf{p}_{k}$, that is, the data frequency
estimate. The fully regularized case of
$\mathbf{b}=[1\>\mathbf{0}^{\intercal}]^{\intercal}$ yields the uniform
estimate $\hat{\mathbf{p}}=\mathbf{1}/2^{n}$. A later section discusses using
cross-validation to optimize $\mathbf{b}$ within these extremes.

The derivation of Equation 7 elucidates that $b_{j}$ regularizes the
expectation of $\left[\mathbf{r}^{(n)}_{X}\right]_{j}$ used in the estimator.
This means that $n\choose k$ elements of $\mathbf{b}$ – and Walsh coefficients
– correspond to products of elements in subsets of $\\{X_{i},\dots,X_{n}\\}$
of size $k\in\mathbb{N}$. Trivially $b_{1}$ corresponds to the empty set whose
product is 1, further justifying setting $b_{1}=1$. In applications where
variables $X_{i}$ reflect interpretable information, therefore, elements of
$\mathbf{b}$ associated products of small subsets of $\\{X_{i},\dots,X_{n}\\}$
carry more meaning. This creates an intuition hierarchy of Walsh coefficients
that can inform optimization choices when $n$ is large. To specify this
hierarchy, define $S^{(n)}_{k}\subset\mathbb{N}$ to be the set of indexes of
$\mathbf{r}^{(n)}_{X}$ corresponding to products of $k$ variables. From the
construction of $\mathbf{r}^{(n)}_{X}$, it follows that

$S^{(n)}_{0}=\\{1\\},\qquad
S^{(n+1)}_{k}=S^{(n)}_{k}\cap\left\\{x+2^{n}\big{|}\>x\in
S^{(n)}_{k-1}\right\\}.$ (9)

Note that $S^{(n)}_{1}=\\{2^{x-1}+1\>|\>x\in[n]\\}$, which matches the indexes
of $W^{(n)}$ elements in Equation 6. An intuition-first approach to optimizing
$\mathbf{b}$ prescribes prioritizing indexes in sets $S^{(n)}_{k}$ of low $k$.
In a case where one considers any combination of more than three variables in
$\\{X_{i},\dots,X_{n}\\}$ uninterpretable, for example, only $(n^{3}+5n+6)/6$
out of the $2^{n}$ elements require optimization for an interpretable
estimator.

###  2.2 Monotonic Transformations of Fourier-Walsh Diagonalization Elements

At its core, the Fourier-Walsh estimator in matrix form defines a similarity
metric between two points on the $\\{-1,1\\}^{n}$ hypercube. For hypercube
points assigned to indexes $i$ and $j$ in $\mathbf{p}$, Equation 7 gives the
kernel $K_{ij}=\mathbf{b}^{\intercal}(W^{(n)}_{:,i}\odot
W^{(n)}_{:,j})/2^{n}$. This intuitive kernel sums elements of $\mathbf{b}$
with a factor $(\pm 1)$ on $b_{k}$ depending on whether the two input points
share the same $\left[\mathbf{r}^{(n)}_{X}\right]_{k}$; it then divides this
by the number of elements. In many cases, monotonic transformations of this
kernel, which preserve its interpretable ordering, can enhance it. For
example, transforming elements $K_{ij}$ using a nonnegative function
guarantees nonnegative density estimates without any restriction on
$\mathbf{b}$. Furthermore, some transformations allow the kernel to be
positive definite without the requirement $\mathbf{b}\succ\mathbf{0}$ as in
Equation 7.

Estimating binary densities using such transformed Fourier-Walsh matrices
requires a guarantee of normalization. Here s fact specific to Fourier-Walsh
matrices is proven to facilitate a normalized generalization of Equation 7
with monotonically transformed elements. Since products of Walsh functions are
themselves Walsh functions, the columns and rows of Walsh matrices must be
closed under element-wise multiplication. Given this fact, suppose a mapping
matrix $\mathcal{M}^{(n)}\in\mathbb{N}^{2^{n}\times 2^{n}}$ where
$W^{(n)}_{:,\mathcal{M}^{(n)}_{ij}}=W^{(n)}_{:,i}\odot W^{(n)}_{:,j}$.

######  Lemma 2.1.

$\mathcal{M}^{(n)}$ exists and each of its rows and columns contain unique
elements in $[2^{n}]\subset\mathbb{N}$.

###### Proof.

In the base case $n=0$, it is evident that
$W^{(0)}=[1]\implies\mathcal{M}^{(0)}=[1]$. From the recursive construction of
$W^{(n)}$, the following hold true for indexes $i,j\in\\{1,\dots,2^{n}\\}$:

$W^{(n)}_{:,i}\odot
W^{(n)}_{:,j}=W^{(n)}_{:,\mathcal{M}^{(n)}_{ij}}\implies\begin{cases}W^{(n+1)}_{:,i}\odot
W^{(n+1)}_{:,j}=W^{(n+1)}_{:,2^{n}+i}\odot
W^{(n+1)}_{:,2^{n}+j}=W^{(n+1)}_{:,\mathcal{M}^{(n)}_{ij}}\\\
W^{(n+1)}_{:,i}\odot W^{(n+1)}_{:,2^{n}+j}=W^{(n+1)}_{:,2^{n}+i}\odot
W^{(n+1)}_{:,j}=W^{(n+1)}_{:,2^{n}+\mathcal{M}^{(n)}_{ij}}\end{cases}.$ (10)

Thus,

$\mathcal{M}^{(n+1)}=\begin{bmatrix}\mathcal{M}^{(n)}&2^{n}J_{2^{n}}+\mathcal{M}^{(n)}\\\
2^{n}J_{2^{n}}+\mathcal{M}^{(n)}&\mathcal{M}^{(n)}\end{bmatrix},$ (11)

where $J_{m}$ is the $m\times m$ all-ones matrix. Note from Equation 11 that
if every row and every column of $\mathcal{M}^{(n)}$ contains all integers
$[2^{n}]$, then $\mathcal{M}^{(n+1)}$ will have the same property for integers
$[2^{n+1}]$. Because this is true for the base case $\mathcal{M}^{(0)}=[1]$ it
holds for all $\mathcal{M}^{(n)}$. This concludes the proof. ∎

With the mapping matrix $\mathcal{M}^{(n)}$ defined, one can write

$\left[W^{(n)}\mathrm{diag}(\mathbf{b})W^{(n)}\right]_{ij}=\sum^{2^{n}}_{k=1}b_{k}W^{(n)}_{ik}W^{(n)}_{jk}=\mathbf{b}^{\intercal}\left(W^{(n)}_{:,i}\odot
W^{(n)}_{:,j}\right)=\mathbf{b}^{\intercal}W^{(n)}_{:,\mathcal{M}^{(n)}_{ij}}.$
(12)

By Lemma 2.1, this means that all rows and columns of
$W^{(n)}\mathrm{diag}(\mathbf{b})W^{(n)}$ contain the same elements. This also
applies to element-wise transformations of
$W^{(n)}\mathrm{diag}(\mathbf{b})W^{(n)}$. Denoting $(Q)^{f}$ to mean the
element-wise action of $f:\mathbb{R}\to\mathbb{R}$ on matrix $Q$, This fact
means

$\left(W^{(n)}\mathrm{diag}(\mathbf{b})W^{(n)}\right)^{f}\mathbf{1}=\left[\left(W^{(n)}\mathrm{diag}(\mathbf{b})W^{(n)}\right)^{f}\mathbf{1}\right]_{1}\mathbf{1}.$
(13)

Since $W_{:,1}=\mathbf{1}$, it follows that

$\left[\left(W^{(n)}\mathrm{diag}(\mathbf{b})W^{(n)}\right)^{f}\mathbf{1}\right]_{1}=\sum^{2^{n}}_{j=1}f\left(\sum^{2^{n}}_{k=1}b_{k}W^{(n)}_{1k}W^{(n)}_{jk}\right)=\left(\mathbf{b}^{\intercal}W^{(n)}\right)^{f}\mathbf{1}.$
(14)

This normalization factor yields the complete general-form estimator

$\hat{\mathbf{p}}=\frac{\left(W^{(n)}\mathrm{diag}(\mathbf{b})W^{(n)}\right)^{f}}{\left(\mathbf{b}^{\intercal}W^{(n)}\right)^{f}\mathbf{1}}\mathbf{p}_{k}.$
(15)

Note that $f(x)=x$ and $b_{1}=1$ yield
$\left(\mathbf{b}^{\intercal}W^{(n)}\right)^{f}\mathbf{1}=\mathbf{b}^{\intercal}W^{(n)}\mathbf{1}=2^{n}$,
which is consistent with Equation 7.

Elements of the matrix in the numerator of Equation 15 are evaluated in time
proportional to the number of nonzero elements in $\mathbf{b}$ regardless of
$f$. However, the normalization factor for nontrivial $\mathbf{b}$ and
nonlinear $f$ typically takes $O(n2^{n})$ time to calculate using the Fast
Walsh Transform algorithm [14], regardless of the number of nonzero elements
in $\mathbf{b}$. In the cases of logistic and exponential transformations, a
certain restriction on $\mathbf{b}$ can greatly reduce the normalization times
to $O(1)$ and $O(n)$ respectively. These improvements are referenced in Table
1.

Define a vector $\mathbf{w}\in[0,1]^{n}$ and an associated
$\mathbf{b}_{\mathbf{w}}$ such that
$\mathbf{b}_{\mathbf{w}}=\sum^{n}_{k=1}w_{k}\hat{e}_{2^{k-1}+1}$; this means
$\mathbf{b}_{\mathbf{w}}$ has nonzero elements only at indexes in
$S^{(n)}_{1}$. Since all column vectors in $\\{W^{(n)}_{:,k}|k\in
S^{(n)}_{1}\\}$ are anti-symmetric – in the sense
$\mathbf{v}=-\mathrm{flip}(\mathbf{v})$ – it follows that
$W^{(n)}\mathbf{b}_{\mathbf{w}}$ is also anti-symmetric. From here, the
propriety of the sigmoid/logistic function $f(x)+f(-x)=1$ allows one to write

$f=\frac{1}{1+\gamma^{-x}}\implies\mathbf{1}^{\intercal}\left(W^{(n)}\mathbf{b}_{\mathbf{w}}\right)^{f}=\left(\mathbf{b}^{\intercal}_{\mathbf{w}}W^{(n)}\right)^{f}\mathbf{1}=2^{n}/2,$
(16)

$\forall\>\gamma\in\mathbb{R}_{+}$. This is evaluated in $O(1)$ time. Turning
to the exponential case, one notes that

$W^{(n)}\mathbf{b}_{\mathbf{w}}=\begin{bmatrix}1\\\ 1\end{bmatrix}\otimes
W^{(n-1)}\left[\mathbf{b}_{\mathbf{w}}\right]_{1:2^{n-1}}W^{(n-1)}+w_{n}\begin{bmatrix}\mathbf{1}\\\
-\mathbf{1}\end{bmatrix}.$ (17)

This means

$f(x)=\gamma^{x}\implies\left(W^{(n)}\mathbf{b}_{\mathbf{w}}\right)^{f}=\begin{bmatrix}\gamma^{w_{n}}\\\
\gamma^{-w_{n}}\end{bmatrix}\otimes\left(W^{(n-1)}\left[\mathbf{b}_{\mathbf{w}}\right]_{1:2^{n-1}}W^{(n-1)}\right)^{f}=\bigotimes^{n-1}_{i=0}\begin{bmatrix}\gamma^{w_{n-i}}\\\
\gamma^{-w_{n-i}}\end{bmatrix}$ (18)

and, finally,

$f(x)=\gamma^{x}\implies=\left(\mathbf{b}^{\intercal}_{\mathbf{w}}W^{(n)}\right)^{f}\mathbf{1}=\prod^{n}_{i=1}\left(\gamma^{w_{i}}+\gamma^{-w_{i}}\right).$
(19)

Equation 19 evaluates in $O(n)$ time.

The powerful flexibility of kernel transformation enables this binary density
estimator to employ the wide range of activation functions used in applied
machine learning. These include exponential, logistic/sigmoid, step, ReLU,
$\tanh$, ELU functions, and many others. The nonnegative natures of the first
four listed functions make them especially useful in guaranteeing nonnegative
density estimates. The choices among these functions can depend on the
performance of cross-validation, the context of the application, and the
desired evaluation speed.

An additional benefit of the presented matrix formulation is that convex
combinations of normalized matrices may also be used, allowing mixtures of
transformed kernel estimators. Such a mixed estimator would take the form

$\hat{\mathbf{p}}=\left[\sum^{m}_{i=1}c_{i}\frac{\left(W^{(n)}\mathrm{diag}(\mathbf{b}_{i})W^{(n)}\right)^{f_{i}}}{\left(\mathbf{b}^{\intercal}_{i}W^{(n)}\right)^{f_{i}}\mathbf{1}}\right]\mathbf{p}_{k},$
(20)

where $\sum^{m}_{i=1}c_{i}=1$, $c_{i}>0$.

###  2.3 Aitchison Aitken Kernel from Exponential Fourier-Walsh Matrix

This section demonstrates that using an exponential function in Equation 15
with $\mathbf{b}=\mathbf{b}_{\mathbf{w}}$ type restrictions yields the AA
kernel estimator; this is despite the seemingly fundamental differences
between the AA and Fourier-Walsh approaches to estimation. The AA kernel gives
a similarity metric between two $n$-length binary vectors $\mathbf{x}_{i}$ and
$\mathbf{x}_{j}$ with parameter $\lambda\in[1/2,1]$ as

$K(\mathbf{x}_{i},\mathbf{x}_{j};\lambda)=\lambda^{n-d(\mathbf{x}_{i},\mathbf{x}_{j})}(1-\lambda)^{d(\mathbf{x}_{i},\mathbf{x}_{j})},$

where $d(\mathbf{x}_{i},\mathbf{x}_{j})$ gives the number of elements where
$\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ differ. This kernel guarantees
normalization and nonnegative density estimation [1]. Supposing that
$\mathbf{x}_{i},\mathbf{x}_{j}\in\\{-1,1\\}^{n}$, one can write

$\mathbf{x}^{\intercal}_{i}\mathbf{x}_{j}=\left[n-d(\mathbf{x}_{i},\mathbf{x}_{j})\right]-d(\mathbf{x}_{i},\mathbf{x}_{j})\implies
d(\mathbf{x}_{i},\mathbf{x}_{j})=\frac{n-\mathbf{x}^{\intercal}_{i}\mathbf{x}_{j}}{2},\quad
n-d(\mathbf{x}_{i},\mathbf{x}_{j})=\frac{n+\mathbf{x}^{\intercal}_{i}\mathbf{x}_{j}}{2}.$
(21)

The AA kernel equation now becomes

$K(\mathbf{x}_{i},\mathbf{x}_{j};\lambda)=\sqrt{\lambda}^{n+\mathbf{x}^{\intercal}_{i}\mathbf{x}_{j}}\sqrt{1-\lambda}^{n-\mathbf{x}^{\intercal}_{i}\mathbf{x}_{j}}=\sqrt{\lambda(1-\lambda)}^{n}\sqrt{\frac{\lambda}{1-\lambda}}^{\mathbf{x}^{\intercal}_{i}\mathbf{x}_{j}}.$
(22)

Using the binary hypercube to $2^{n}$-vector mapping implicit in Equation 6,
one can write
$\mathbf{x}^{\intercal}_{i}\mathbf{x}_{j}=\left[W^{(n)}\mathrm{diag}(\mathbf{b}_{\mathbf{1}})W^{(n)}\right]_{ij}$,
where $\mathbf{b}_{\mathbf{1}}$ corresponds to $\mathbf{b}_{\mathbf{w}}$ in
the case $\mathbf{w}=\mathbf{1}$. Using Equation 19, one notes that

$f(x)=\sqrt{\frac{\lambda}{1-\lambda}}^{x}\implies\left[\left(\mathbf{b}^{\intercal}_{\mathbf{1}}W^{(n)}\right)^{f}\mathbf{1}\right]^{-1}=\left(\sqrt{\frac{\lambda}{1-\lambda}}+\sqrt{\frac{1-\lambda}{\lambda}}\right)^{-n}=\sqrt{\lambda(1-\lambda)}^{n},$
(23)

in agreement with the AA kernel normalization in Equation 22. Therefore, the
AA kernel estimator is equivalent to

$\hat{\mathbf{p}}_{\mathrm{AAK}}=\frac{\left(W^{(n)}\mathrm{diag}(\mathbf{b}_{\mathbf{1}})W^{(n)}\right)^{f}}{\left(\mathbf{b}^{\intercal}_{\mathbf{1}}W^{(n)}\right)^{f}\mathbf{1}}\mathbf{p}_{k},\qquad
f(x)=\sqrt{\frac{\lambda}{1-\lambda}}^{x},$ (24)

a restricted form of the estimator in Equation 15.

Given this equation, one may relax $\mathbf{b}$ from $\mathbf{b}_{\mathbf{1}}$
to $\mathbf{b}_{\mathbf{w}}$ to yield a variable-weighted version of the AA
kernel. This allows the elements of $\mathbf{w}$ to parameterize different
"smoothing" levels along the $n$ dimensions of the hypercube. A variable-
weighted kernel reflects realistic cases where similarities of two points in
some indexes matter more to overall similarity than similarities in other
indexes. For convenience, a reparameterization
$\gamma=\sqrt{\lambda/(1-\lambda)}$ is introduced. The bounds
$\lambda\in[1/2,1]$ correspond to $\gamma\in[1,\infty)$. Letting
$\mathbf{b}_{\mathbf{1}}\to\mathbf{b}_{\mathbf{w}}$ yields the weighted AA
kernel estimator

$\hat{\mathbf{p}}_{\mathrm{WAAK}}=\frac{\left(W^{(n)}\mathrm{diag}(\mathbf{b}_{\mathbf{w}})W^{(n)}\right)^{f}}{\left(\mathbf{b}^{\intercal}_{\mathbf{w}}W^{(n)}\right)^{f}\mathbf{1}}\mathbf{p}_{k},\qquad
f(x)=\gamma^{x}.$ (25)

It is now shown that one may write the numerator of Equation 25 as a series of
Kronecker products, easing computation and elucidating certain properties
about the kernel’s positive definiteness. Noting that the product
$W^{(n-1)}\mathrm{diag}\left([a\>\mathbf{0}^{\intercal}]\right)W^{(n-1)}$
gives $aJ_{2^{n-1}}$, the recursive form of $W^{(n)}$ means

$W^{(n)}\mathrm{diag}(\mathbf{b}_{\mathbf{w}})W^{(n)}=\begin{bmatrix}1&1\\\
1&1\end{bmatrix}\otimes
W^{(n-1)}\mathrm{diag}\left([\mathbf{b}_{\mathbf{w}}]_{1:2^{n-1}}\right)W^{(n-1)}+\begin{bmatrix}1&-1\\\
-1&1\end{bmatrix}w_{n}J_{2^{n-1}}.$ (26)

This means

$f(x)=\gamma^{x}\implies\left(W^{(n)}\mathrm{diag}(\mathbf{b}_{\mathbf{w}})W^{(n)}\right)^{f}=\begin{bmatrix}\gamma^{w_{n}}&\gamma^{-w_{n}}\\\
\gamma^{-w_{n}}&\gamma^{w_{n}}\end{bmatrix}\otimes\left(W^{(n-1)}\mathrm{diag}([\mathbf{b}_{\mathbf{w}}]_{1:2^{n-1}})W^{(n-1)}\right)^{f}.$
(27)

Continuing the recursion, the complete weighted AA kernel estimator can be
written as

$\hat{\mathbf{p}}_{\mathrm{WAAK}}=\left[\prod^{n}_{j=1}\left(\gamma^{w_{j}}+\gamma^{-w_{j}}\right)\right]^{-1}\bigotimes^{n-1}_{i=0}\begin{bmatrix}\gamma^{w_{n-i}}&\gamma^{-w_{n-i}}\\\
\gamma^{-w_{n-i}}&\gamma^{w_{n-i}}\end{bmatrix}\mathbf{p}_{k}.$ (28)

In this form, elements of the weighted AA kernel matrix can be evaluated in
$O(n)$ time. Note also that since a $2\times 2$ matrix with on-diagonal
elements $\gamma^{w}$ and off diagonal elements $\gamma^{-w}$ has eigenvalues
$\gamma^{w}-\gamma^{-w}$ and $\gamma^{w}+\gamma^{-w}$, the matrices in the
Kronecker product in Equation 28 are all positive definite when
$\mathbf{w}\succ\mathbf{0}$ and $\gamma>1$. Since a Kronecker product of two
positive definite matrices is also positive definite, it follows that the
weighted AA kernel estimator uses a positive definite kernel function. This
analysis rederives the known result that the AA kernel is positive definite
[7] and extends this property to its variable-weighted generalization.

##  3 Cross Validation

The flexibility of the presented general-form estimator is driven by its
variable, and possibly high, number of parameters. These include elements of
$\mathbf{b}$, parameters defining $f$, and weights $c_{i}$ applied to matrices
when mixing estimators. Practical optimization of these parameters typically
involves a cross-validation scheme. This section outlines the common leave-
one-out approach to cross-validation using the squared error (SE) and
Kullback-Leibler (KL) loss functions. It specifically focuses on risk function
evaluation times across variants of the general-form estimator.

For the optimizations described, suppose a "true" probability distribution
$\mathbf{p}$ and the estimator $\hat{\mathbf{p}}_{\lambda}$ parameterized by
$\lambda$. The optimal $\lambda^{*}$ minimizes the expectation of a loss
function $L(\mathbf{p},\hat{\mathbf{p}}_{\lambda})$ i.e., the risk. The SE and
KL loss functions are respectively defined as

$L_{\mathrm{SE}}(\mathbf{p},\hat{\mathbf{p}}_{\lambda})=||\hat{\mathbf{p}}_{\lambda}-\mathbf{p}||^{2}_{2},\qquad
L_{\mathrm{KL}}(\mathbf{p},\hat{\mathbf{p}}_{\lambda})=\sum^{2^{n}}_{j=1}\mathbf{p}_{j}\log\left(\mathbf{p}_{j}\big{/}\left[\hat{\mathbf{p}}_{\lambda}\right]_{j}\right).$
(29)

To implement the leave-one-out technique, let $K\in[2^{n}]$ be a multiset
encoding indexes of observations. Also, define
$\hat{\mathbf{p}}^{(k)}_{\lambda}$ as the estimate made without an observed
data point corresponding to $k\in K$. By the law of the unconscious
statistician,

$\mathbb{E}\left[\hat{\mathbf{p}}^{\intercal}_{\lambda}\mathbf{p}\right]=\frac{1}{|K|}\sum_{k\in
K}\mathbb{E}\left[\left(\hat{\mathbf{p}}^{(k)}_{\lambda}\right)^{\intercal}\hat{e}_{k}\right]\implies\arg\min_{\lambda}\mathbb{E}\left[L_{\mathrm{SE}}(\mathbf{p},\hat{\mathbf{p}}_{\lambda})\right]=\arg\min_{\lambda}\left[\hat{\mathbf{p}}^{\intercal}_{\lambda}\hat{\mathbf{p}}_{\lambda}-\frac{2}{|K|}\sum_{k\in
K}\left(\hat{\mathbf{p}}^{(k)}_{\lambda}\right)^{\intercal}\hat{e}_{k}\right].$
(30)

For the KL loss, the cross-validation optimization is given by

$\arg\min_{\lambda}\mathbb{E}\left[L_{\mathrm{KL}}(\mathbf{p},\hat{\mathbf{p}}_{\lambda})\right]=\arg\max_{\lambda}\sum_{k\in
K}\log\left[\left(\hat{\mathbf{p}}^{(k)}_{\lambda}\right)^{\intercal}\hat{e}_{k}\right],$
(31)

using a discrete adaptation of the known KL cross validation optimizer for
density estimation over continuous spaces [15].

One can now impose the familiar form
$\hat{\mathbf{p}}_{\lambda}=Q_{\lambda}\mathbf{p}_{k}=Q_{\lambda}\left[(1/|K|)\sum_{k\in
K}\hat{e}_{k}\right]$ for some generic symmetric estimator matrix
$Q_{\lambda}\in\mathbb{R}^{2^{n}\times 2^{n}}$. This means that evaluating
$\left(\hat{\mathbf{p}}^{(k)}_{\lambda}\right)^{\intercal}\hat{e}_{k}$
involves knowing $|K|-1$ elements of $Q_{\lambda}$. Therefore, the computation
of the KL risk function requires the evaluation of $|K|(|K|-1)/2$ elements of
$Q_{\lambda}$. In the SE risk case, the second term also involves knowing
$|K|(|K|-1)/2$ elements of $Q_{\lambda}$, while the first requires evaluating
$|K|(|K|+1)/2$ elements of $Q^{2}_{\lambda}$. Squared-matrix elements are
particularly simple to calculate for numerators of Equations 7 and 28 since

$\left(W^{(n)}\mathrm{diag}(\mathbf{b})W^{(n)}\right)^{2}=2^{n}\left(W^{(n)}\mathrm{diag}(\mathbf{b}\odot\mathbf{b})W^{(n)}\right)$
(32)

and

$f(x)=\gamma^{x}\implies\left[\left(W^{(n)}\mathrm{diag}(\mathbf{b}_{\mathbf{w}})W^{(n)}\right)^{f}\right]^{2}=\bigotimes^{n-1}_{i=0}\begin{bmatrix}\gamma^{2w_{n-i}}+\gamma^{-2w_{n-i}}&2\\\
2&\gamma^{2w_{n-i}}+\gamma^{-2w_{n-i}}\end{bmatrix},$ (33)

by the Kronecker mixed product property.

This section concludes with a tabulation of the computation times required to
normalize and evaluate matrix and squared-matrix elements of useful estimators
derived from restrictions of Equation 15. Table 1 shows these computation
times, where $b$ gives the number of nonzero (or non-constant) elements of
$\mathbf{b}$ and the fast Walsh transform [14] is used for listed operations
that take time $O(n2^{n})$.

Table 1: Evaluation times of normalization and finding one element of an estimator matrix and its square as functions of $n$, the hypercube dimension, and $b$, the number of non-constant diagonal elements. These are reported for the general-form estimator in Equation 15 for various monotonic functions and restrictions on $\mathbf{b}$. Estimator Restrictions | Normalization time | Matrix element time | Squared-Matrix element time
---|---|---|---
$f(x)=x,b_{1}=1$ | $O(1)$ | $O(b)$ | $O(b)$
$f(x)=\gamma^{x},\mathbf{b}=\mathbf{b}_{\mathbf{w}}$ | $O(n)$ | $O(n)$ | $O(n)$
$f(x)=\left(1+\gamma^{-x}\right)^{-1},\mathbf{b}=\mathbf{b}_{\mathbf{w}}$ | $O(1)$ | $O(n)$ | $O(n2^{n})$
No restriction | $O(n2^{n})$ | $O(b)$ | $O(n2^{n})$

##  4 Summary and Discussion

This article presents a powerful binary density estimator built on element-
wise monotonic transformations of Fourier-Walsh diagonalizations. To
accomplish this, the article first provides an intuitive rederivation of the
Fourier-Walsh decomposition in the form of a diagonalization. In this form,
Walsh coefficients are shown to relate to unique products of constituent
univariate binary variables. A specific property of Walsh matrices is then
shown that enables normalization of an estimator arising from any elementwise
transformation function. It is then elucidated how the AA kernel arises from
the above process with a generic exponential transformation, and a variable-
weighted extension of the kernel is introduced that retains its desirable
properties. Finally, the implementations of leave-one-out cross-validation
risk functions are outlined for squared error and Kullback-Leibler loss
functions and their computation times are compared across estimators. The
flexibility, speed, and interpretability of this new estimator under different
constraints make it an ideal candidate for use in a wide range of estimation
and learning applications.

The comparison made in Table 1 shows that variants of the proposed estimator
under different constraints serve best in different regimes of data science.
For problems of up to $n\approx 20$ dimensions – i.e. binary inputs –
computers can safely handle $O(n2^{n})$ time operations and the estimator in
Equation 15 may apply in its most general form. Learning in this setting could
involve iterating over a large number of transformation functions and
exploring mixtures of several different estimation matrices. Problems of
approximately 20 dimensions and 40 data points have been cited as typical in
applied binary density estimation [8]. At the other extreme, one could
consider a high-dimensional case of $n$ up to $n\approx 10^{4}$. Here, any
approach other than the introduced weighted AA kernel and untransformed
Fourier-Walsh diagonalization – with heavily restricted $\mathbf{b}$ – becomes
highly intractable.

These extremes not only showcase the high versatility of the general form
estimator, but also invite the possibility of variable selection when faced
with faced with a learning task. Suppose, for example, that 100 variables
encode five response variables and 95 regressors of varying inference
importance. One could first employ a direct Fourier-Walsh diagonalization
estimator and optimize only elements of $\mathbf{b}$ in $S^{(95)}_{1}$,
$S^{(95)}_{2}$, and $S^{(95)}_{3}$. From these optimized quantities, one could
find the set of 20 binary regressor variables most correlated with the
response variables and then apply less regularized and restricted estimator
variants using only these binary inputs. Such "variable search" approaches
made possible by the presented estimator can make it powerful in the realm of
machine learning over massive binary spaces. Methods to select hypercubes over
which to estimate practical densities could be the focus of exciting future
research.

## References

  * [1] J. Aitchison and C. G. G. Aitken.  Multivariate binary discrimination by the kernel method.  Biometrika, 63(3):413–420, 1976.
  * [2] I. Shmulevich and W. Zhang.  Binary analysis and optimization-based normalization of gene expression data.  Bioinformatics, 18(4):555–565, 2002.
  * [3] C. Jentsch and L. Reichmann.  Generalized binary time series models.  Econometrics, 7(4):47–73, 2019.
  * [4] B. Zhang, S. N. Srihari, and C. Huang.  Word image retrieval using binary features.  In Proc. SPIE 5296, Document Recognition and Retrieval XI, volume 5296, 2003.
  * [5] J. Lu, V. E. Liong, X. Zhou, and J. Zhou.  Learning compact binary face descriptor for face recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 10(1):2041–2056, 2015.
  * [6] M. Saleh and R. Le Bouquin Jeannes.  An efficient machine learning-based fall detection algorithm using local binary features.  In 2018 26th European Signal Processing Conference (EUSIPCO), 2018\.
  * [7] H. Mussa.  The aitchison and aitken kernel function revisited.  Journal of Mathematics Research, 5(1):22–25, 2013.
  * [8] B. Grund and P. Hall.  On the performance of kernel estimators for high-dimensional, sparse binary data.  Journal of Multivariate Analysis, 44(2):321–344, 1993.
  * [9] W. Liang and P. R. Krishnaiah.  Nonparametric iterative estimation of multivariate binary density.  Journal of Multivariate Analysis, 16(1):162–172, 1985.
  * [10] X. R. Chen, P. R. Krishnaiah, and W. Liang.  Estimation of multivariate binary density using orthogonal functions.  Journal of Multivariate Analysis, 31(2):178–186, 1989.
  * [11] M. Raginsky, J. G. Silva, S. Lazebnik, and R. Willett.  A recursive procedure for density estimation on the binary hypercube.  Electronic Journal of Statistics, 7:820–858, 2013.
  * [12] J. L. Walsh.  A closed set of normal orthogonal functions.  American Journal of Mathematics, 45(1):5–24, 1923.
  * [13] W. James and C. Stein.  Estimation with quadratic loss.  Berkeley Symp. on Math. Statist. and Prob., 4(1):361–379, 1961\.
  * [14] M. T. Hamood and S. Boussakta.  Fast walsh–hadamard–fourier transform algorithm.  IEEE Transactions on Signal Processing, 59(11):5627 – 5631, 2011\.
  * [15] P. Hall.  On kullback-leibler loss and density estimation.  Annals of Statistics, 15(4):1491–1519, 1987.