Skew Symplectic and Orthogonal Schur Functions

Skew Symplectic and Orthogonal Schur Functions

Naihuan JING a, Zhijun LI b and Danxia WANG b

N. Jing, Z. Li and D. Wang

a) Department of Mathematics, North Carolina State University, Raleigh, NC
27695, USA<EMAIL_ADDRESS>
b) School of Science, Huzhou University, Huzhou, Zhejiang 313000, P.R. China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Received August 28, 2023, in final form May 12, 2024; Published online May 21,
2024

Using the vertex operator representations for symplectic and orthogonal Schur
functions, we define two families of symmetric functions and show that they
are the skew symplectic and skew orthogonal Schur polynomials defined
implicitly by Koike and Terada and satisfy the general branching rules.
Furthermore, we derive the Jacobi–Trudi identities and Gelfand–Tsetlin
patterns for these symmetric functions. Additionally, the vertex operator
method yields their Cauchy-type identities. This demonstrates that vertex
operator representations serve not only as a tool for studying symmetric
functions but also offers unified realizations for skew Schur functions of
types A, C, and D.

skew orthogonal/symplectic Schur functions; Jacobi–Trudi identity;
Gelfand–Tsetlin patterns; vertex operators

05E05; 17B37

##  1 Introduction

For a partition $\lambda=(\lambda_{1},\lambda_{2},\ldots,\lambda_{n})$, Schur
functions, symplectic Schur functions, and orthogonal Schur functions in
finitely many variables are respectively defined by [19, 20, 24, 25]

$\displaystyle{\rm
s}_{\lambda}(x)=\frac{\det\big{(}x^{\lambda_{j}+n-j}_{i}\big{)}_{1\leq i,j\leq
n}}{\det\big{(}x^{n-j}_{i}\big{)}_{1\leq i,j\leq n}},$ (1.1)
$\displaystyle{\rm
sp}_{\lambda}\big{(}x^{\pm}\big{)}=\frac{\det\big{(}x^{\lambda_{j}+(n-j+1)}_{i}-x^{-\lambda_{j}-(n-j+1)}_{i}\big{)}^{n}_{i,j=1}}{\det\big{(}x^{n-j+1}_{i}-x^{-(n-j+1)}_{i}\big{)}^{n}_{i,j=1}},$
(1.2) $\displaystyle{\rm
o}_{\lambda}\big{(}x^{\pm}\big{)}=2(-1)^{\frac{n(n-1)}{2}}\frac{\det\big{(}x^{-\lambda_{j}+j-n}_{i}+\delta_{\lambda_{n}\neq
0}\delta_{j\neq
n}x^{\lambda_{j}-j+n}_{i}\big{)}^{n}_{i,j=1}}{\det\big{(}x^{-j+1}_{i}+x^{j-1}_{i}\big{)}^{n}_{i,j=1}},$
(1.3)

where $\delta_{a\neq b}$ equals 1 if $a\neq b$ and is $0$ otherwise. They
appear in various contexts of mathematics, especially in the representation
theory of classical Lie groups (general linear groups, symplectic groups and
even orthogonal groups) as characters of finite-dimensional irreducible
representations [14, 18, 25, 26]. They are also studied in generalized vertex
algebras and $q$-characters [5, 17].

With the advent of Schur symplectic and Schur orthogonal symmetric functions,
arises the inquiry into skew versions for these symmetric functions. Since the
family of symplectic Schur or orthogonal Schur functions indexed by partitions
does not constitute a complete basis in the ring of symmetric functions (in
fact they belong to the ring of Laurent polynomials), the traditional approach
of employing the adjoint operation is not viable for defining skew orthogonal
or symplectic Schur functions. Instead the skew symplectic and (even) skew
orthogonal symmetric functions were implicitly defined by Koike and Terada
[15] by restricting the irreducible characters of the classical groups into
their subgroups and were expressed by tableaux representations. In [2], the
tableaux representations were transformed into the Gelfand–Tsetlin pattern
representations by Ayyer and Fischer. It is natural to ask whether the skew
symplectic Schur and orthogonal Schur functions can be computed by
Jacobi–Trudi type formulas, which would then lead to a parallel formulism as
the Schur functions and skew Schur functions and also provide a lifting method
to define the skew Schur symplectic and orthogonal functions in infinitely
many variables.

The aim of this paper is to address these questions by the vertex operator
realization of the classical symmetric functions [9] and the symplectic Schur
and orthogonal Schur functions [3, 12]. The vertex algebraic method of
studying Schur polynomials can be traced back to the Kyoto school’s work on
integrable systems [6] where the tau functions of the KP hierarchy are in
terms of Schur polynomials. Skew Schur functions ${\rm s}_{\lambda/\mu}$
satisfy the celebrated decomposition formula, i.e., the general branching rule

$\displaystyle{\rm
s}_{\lambda}(x_{1},\dots,x_{n})=\sum_{\mu\subset\lambda}{\rm
s}_{\mu}(x_{1},\dots,x_{n-k}){\rm s}_{\lambda/\mu}(x_{n-k+1},\dots,x_{n}),$

which can be interpreted as the restriction of the irreducible representation
of $\mathrm{GL}(n,\mathbb{C})$ to
$\mathrm{GL}(n-k,\mathbb{C})\times\mathrm{GL}(k,\mathbb{C})$. We will define
skew (symplectic/orthogonal) Schur functions using vertex operators and derive
their combinatorial properties such as the Jacobi–Trudi identities, Cauchy-
type identities and verify they satisfy the Gelfand–Tsetlin pattern
representations found by [2]. We also show that our vertex operator
representations for skew-type symmetric functions comply with the general
branching rules in the agreement with Koike and Terada’s approach.

The construction of skew Schur functions and their generalizations can be
formulated in terms of the representation theory of infinite-dimensional Lie
algebras. One starts with the infinite-dimensional Heisenberg algebra [8] with
the center $1$ and defines the Fock space $\mathcal{M}^{*}$ and its completion
$\widetilde{\mathcal{M}^{*}}$ generated by the vacuum $\langle 0|$ (which is
the vacuum vector in the right module) and the Heisenberg generators with
positive modes. There are three families of special elements
($\langle\lambda|$, $\langle\lambda^{\rm sp}|$ and $\langle\lambda^{\rm o}|$
in the right module) defined by partitions $\lambda$, corresponding
respectively to the Schur, symplectic Schur, and orthogonal Schur cases. We
will prove that they are orthonormal by computing the inner product with the
elements ($|\lambda\rangle$, $|\lambda^{\rm sp}\rangle~{}and~{}|\lambda^{\rm
o}\rangle$ in the left module) in the Fock space $\mathcal{M}$ respectively
(see Theorem 2.3), which is key to construct (skew) symmetric functions and
also savages the problem of adjoint operation with skew versions. For the half
vertex operators $\Gamma_{+}(\\{x\\})$ and
$\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}$, we will show that

$\displaystyle\langle 0|\Gamma_{+}(\\{x\\})=\sum_{\lambda}\langle\lambda|{\rm
s}_{\lambda}(x),$ $\displaystyle\langle
0|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}=\sum_{\lambda}\langle\lambda^{\rm
sp}|{\rm sp}_{\lambda}\big{(}x^{\pm}\big{)}=\sum_{\lambda}\langle\lambda^{\rm
o}|{\rm o}_{\lambda}\big{(}x^{\pm}\big{)},$

where each summation is over (generalized) partitions
$\lambda=(\lambda_{1},\dots,\lambda_{N})$ with zeros parts allowed in the end.
The benefit of this approach is that we can easily obtain the well-known
classical Cauchy identities for (symplectic/orthogonal) Schur functions [14,
25]. This approach is logically independent of the Jacobi–Trudi formula.

It is well known that $\langle\mu|\Gamma_{+}(\\{x\\})|\lambda\rangle={\rm
s}_{\lambda/\mu}(x)$ [16, 21] using the half vertex operator
$\Gamma_{+}(\\{x\\})$. We extend the vertex operator method to derive two
families of symmetric functions

$\displaystyle{\rm sp}_{\lambda/\mu}\big{(}x^{\pm}\big{)}=\langle\mu^{\rm
sp}|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}|\lambda^{\rm
sp}\rangle,$ $\displaystyle{\rm
o}_{\lambda/\mu}\big{(}x^{\pm}\big{)}=\langle\mu^{\rm
o}|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}|\lambda^{\rm o}\rangle,$

which can be interpreted as the skew symplectic/orthogonal Schur functions
from the general branching rule. We also show that each skew function can be
written as a Jacobi–Trudi type determinant of complete homogeneous symmetric
functions with variables $x_{1},x^{-1}_{1},\dots,x_{N},x^{-1}_{N}$.

Exploiting the combinatorial properties of $\langle\mu^{\rm sp}|$ and
$\langle\mu^{\rm o}|$, we also obtain Gelfand–Tsetlin pattern representations
(generating functions of some Young tableaux in an equivalent form [24]) for
skew symplectic/orthogonal Schur functions, which provide alternative
definitions of symplectic/orthogonal Schur functions (see [2]). The vertex
operator method depending on the combinatorial properties of some orthogonal
elements of the Fock space $\widetilde{\mathcal{M}^{*}}$ may shed light on
specific descriptions of generalized shape $\pi$ symmetric functions [7].

This paper is organized as follows. In Section 2, we recall the vertex
operators related to fermionic Fock spaces $\mathcal{M}$ and $\mathcal{M}^{*}$
and prove the orthonormality for three distinguished sets of $\mathcal{M}$ and
the completion of $\mathcal{M}^{*}$. In Section 3, we define the skew
symplectic/orthogonal Schur functions by the vertex algebraic method and
derive their Jacobi–Trudi identities. We also show the vertex operator
realizations for these skew symmetric functions are consistent with the
general branching rule, therefore they agree with the skew symplectic and
orthogonal Schur polynomials introduced by Koike and Terada. In Section 4, we
give formulas of skew symplectic/orthogonal Schur functions in terms of
Gelfand–Tsetlin pattern representations, which agree with Ayyer and Fischer’s
results. In Section 5, we provide a vertex operator approach to the Cauchy
identities for the (symplectic/orthogonal) Schur functions, which is logically
independent of the Jacobi–Trudi formulas [13]. We also obtain Cauchy-type
identities for skew (symplectic/orthogonal) Schur functions.

##  2 Preliminaries

Let $\mathcal{H}$ be the Heisenberg algebra generated by $\\{a_{n}\mid n\neq
0\\}$ with the central element $c=1$ subject to the commutation relations [8]

$\displaystyle[a_{m},a_{n}]=m\delta_{m,-n}c,\qquad[a_{n},c]=0.$ (2.1)

The Fock space $\mathcal{M}$ (resp. $\mathcal{M}^{*}$) is generated by the
vacuum vector $|0\rangle$ (resp. dual vacuum vector $\langle 0|$) and subject
to

$\displaystyle a_{n}|0\rangle=\langle 0|a_{-n},\qquad n>0.$

In other words, $\mathcal{M}$ and $\mathcal{M}^{*}$ are respectively left and
right modules for $\mathcal{H}$. It is easy to see that $\mathcal{H}$ acts on
$\mathcal{M}$ (resp. $\mathcal{M}^{*}$) irreducibly. Moreover, $\mathcal{M}$
(resp. $\mathcal{M}^{*}$) is isomorphic to the symmetric algebra
$\mathbb{C}[a_{-1},a_{-2},\dots]$ (resp. $\mathbb{C}[a_{1},a_{2},\dots]$).
Note that $\mathcal{M}^{*}$ is a graded space with the gradation induced from
that of $\mathcal{H}$. Let $\big{\\{}\mathcal{M}^{*}_{n}\big{\\}}$ be the
filtration of the subspaces $\mathcal{M}^{*}_{n}$ spanned by homogeneous
elements with degree $\geq n$. Then we let $\widetilde{\mathcal{M}^{*}}$ be
the associated completion of the Fock space $\mathcal{M}^{*}$.

Consider the following vertex operators given in [3, 12] (see also [23])

$\displaystyle
X(z)=\exp\Biggl{(}\sum^{\infty}_{n=1}\frac{a_{-n}}{n}z^{n}\Biggr{)}\exp\Biggl{(}-\sum^{\infty}_{n=1}\frac{a_{n}}{n}z^{-n}\Biggr{)}=\sum_{n\in\mathbb{Z}}X_{n}z^{-n},$
$\displaystyle
X^{*}(z)=\exp\Biggl{(}-\sum^{\infty}_{n=1}\frac{a_{-n}}{n}z^{n}\Biggr{)}\exp\Biggl{(}\sum^{\infty}_{n=1}\frac{a_{n}}{n}z^{-n}\Biggr{)}=\sum_{n\in\mathbb{Z}}X^{*}_{n}z^{n},$
$\displaystyle
Y(z)=\exp\Biggl{(}\sum^{\infty}_{n=1}\frac{a_{-n}}{n}z^{n}\Biggr{)}\exp\Biggl{(}-\sum^{\infty}_{n=1}\frac{a_{n}}{n}(z^{-n}+z^{n})\Biggr{)}=\sum_{n\in\mathbb{Z}}Y_{n}z^{-n},$
$\displaystyle
Y^{*}(z)=\big{(}1-z^{2}\big{)}\exp\Biggl{(}-\sum^{\infty}_{n=1}\frac{a_{-n}}{n}z^{n}\Biggr{)}\exp\Biggl{(}\sum^{\infty}_{n=1}\frac{a_{n}}{n}(z^{-n}+z^{n})\Biggr{)}=\sum_{n\in\mathbb{Z}}Y^{*}_{n}z^{n},$
$\displaystyle
W(z)=\big{(}1-z^{2}\big{)}\exp\Biggl{(}\sum^{\infty}_{n=1}\frac{a_{-n}}{n}z^{n}\Biggr{)}\exp\Biggl{(}-\sum^{\infty}_{n=1}\frac{a_{n}}{n}(z^{-n}+z^{n})\Biggr{)}=\sum_{n\in\mathbb{Z}}W_{n}z^{-n},$
$\displaystyle
W^{*}(z)=\exp\Biggl{(}-\sum^{\infty}_{n=1}\frac{a_{-n}}{n}z^{n}\Biggr{)}\exp\Biggl{(}\sum^{\infty}_{n=1}\frac{a_{n}}{n}(z^{-n}+z^{n})\Biggr{)}=\sum_{n\in\mathbb{Z}}W^{*}_{n}z^{n}.$
(2.2)

######  Lemma 2.1.

The actions of the operators $X_{n}$, $X^{*}_{n}$, $Y_{n}$, $Y^{*}_{n}$,
$W_{n}$ and $W^{*}_{n}$ on the $($dual$)$ vacuum are given by

$\displaystyle X_{n}|0\rangle=X^{*}_{-n}|0\rangle=\langle 0|X_{-n}=\langle
0|X^{*}_{n}$
$\displaystyle\hphantom{X_{n}|0\rangle}{}=Y_{n}|0\rangle=Y^{*}_{-n}|0\rangle=W_{n}|0\rangle=W^{*}_{-n}|0\rangle=0\qquad\text{for}~{}n>0,$
(2.3) $\displaystyle\langle 0|Y^{*}_{n}=-\langle 0|Y^{*}_{-n+2},\quad\langle
0|Y_{n}=\langle 0|Y_{-n},\quad\langle 0|W_{n}=-\langle 0|W_{-n-2},\quad\langle
0|W^{*}_{n}=\langle 0|W^{*}_{-n}.$ (2.4)

###### Proof.

Relations (2.3) are obvious, and we only need to prove (2.4). Note the fact

$\displaystyle\langle
0|\exp\Biggl{(}\sum^{\infty}_{n=1}\frac{a_{n}}{n}(z^{-n}+z^{n})\Biggr{)}=\langle
0|\frac{1}{1-z^{2}}\exp\Biggl{(}\sum^{\infty}_{n=1}\frac{a_{-n}}{n}z^{n}\Biggr{)}Y^{*}(z)=\frac{1}{1-z^{2}}\langle
0|Y^{*}(z),$ $\displaystyle\langle
0|\exp\Biggl{(}\sum^{\infty}_{n=1}\frac{a_{n}}{n}(z^{-n}+z^{n})\Biggr{)}=\langle
0|\frac{1}{1-z^{-2}}\exp\Biggl{(}\sum^{\infty}_{n=1}\frac{a_{-n}}{n}z^{-n}\Biggr{)}Y^{*}\big{(}z^{-1}\big{)}$
$\displaystyle\hphantom{\langle
0|\exp\Biggl{(}\sum^{\infty}_{n=1}\frac{a_{n}}{n}(z^{-n}+z^{n})\Biggr{)}}{}=\frac{-z^{2}}{1-z^{2}}\langle
0|Y^{*}\big{(}z^{-1}\big{)},$

therefore

$\displaystyle\langle 0|Y^{*}(z)=-z^{2}\langle 0|Y^{*}\big{(}z^{-1}\big{)}.$
(2.5)

Taking the coefficient of $z^{n}$ of (2.5), we have

$\displaystyle\langle 0|Y^{*}_{n}=-\langle 0|Y^{*}_{-n+2}.$

Similarly, we can get the other commutation relations in (2.4). ∎

The following result can be found in [12, Theorem 3.4, Proposition 3.6].

######  Lemma 2.2.

The components of the vertex operators (2.2) satisfy the following commutation
relations

$\displaystyle X_{i}X_{j}+X_{j+1}X_{i-1}=0,\quad\ $ $\displaystyle
X^{*}_{i}X^{*}_{j}+X^{*}_{j-1}X^{*}_{i+1}=0,\quad\ $ $\displaystyle
X_{i}X^{*}_{j}+X^{*}_{j+1}X_{i+1}=\delta_{i,j},$ $\displaystyle
Y_{i}Y_{j}+Y_{j+1}Y_{i-1}=0,\quad\ $ $\displaystyle
Y^{*}_{i}Y^{*}_{j}+Y^{*}_{j-1}Y^{*}_{i+1}=0,\quad\ $ $\displaystyle
Y_{i}Y^{*}_{j}+Y^{*}_{j+1}Y_{i+1}=\delta_{i,j},$ (2.6) $\displaystyle
W_{i}W_{j}+W_{j+1}W_{i-1}=0,\quad\ $ $\displaystyle
W^{*}_{i}W^{*}_{j}+W^{*}_{j-1}W^{*}_{i+1}=0,\quad\ $ $\displaystyle
W_{i}W^{*}_{j}+W^{*}_{j+1}W_{i+1}=\delta_{i,j}.$

A ${\it partition}$ $\lambda=(\lambda_{1},\lambda_{2},\dots,\lambda_{l})$ is a
weakly decreasing sequence of positive integers, and a generalized partition
$\lambda=(\lambda_{1},\lambda_{2},\dots,\lambda_{n})$ is a weakly decreasing
finite sequence of nonnegative integers. The $\lambda_{i}$’s are called the
parts of $\lambda$, and the number $l(\lambda)$ of nonzero parts is the length
of $\lambda$. If $\lambda$, $\mu$ are partitions, we will write
$\mu\subset\lambda$ to mean that $\lambda_{i}\geq\mu_{i}$ for all $i\geq 1$.
Therefore, a generalized partition is a partition appended with a string of
finitely many zeros.111In the paper, generalized partitions should be used
always for the skew symplectic and orthogonal Schur functions. We remark that
the skew Schur functions indexed by generalized partitions coincide with those
indexed by ordinary partitions. For a given (generalized) partition
$\lambda=(\lambda_{1},\lambda_{2},\dots,\lambda_{l})$, let

$\displaystyle|\lambda\rangle=X_{-\lambda_{1}}X_{-\lambda_{2}}\cdots
X_{-\lambda_{l}}|0\rangle,\qquad$ $\displaystyle\langle\lambda|=\langle
0|X^{*}_{-\lambda_{l}}\cdots X^{*}_{-\lambda_{1}},$ (2.7)
$\displaystyle|\lambda^{\rm sp}\rangle=Y_{-\lambda_{1}}Y_{-\lambda_{2}}\cdots
Y_{-\lambda_{l}}|0\rangle,\qquad$ $\displaystyle\langle\lambda^{\rm
sp}|=\langle 0|Y^{*}_{-\lambda_{l}}\cdots Y^{*}_{-\lambda_{1}},$ (2.8)
$\displaystyle|\lambda^{\rm o}\rangle=W_{-\lambda_{1}}W_{-\lambda_{2}}\cdots
W_{-\lambda_{l}}|0\rangle,\qquad$ $\displaystyle\langle\lambda^{\rm
o}|=\langle 0|W^{*}_{-\lambda_{l}}\cdots W^{*}_{-\lambda_{1}}.$ (2.9)

We remark that the vectors $|\lambda\rangle$, $|\lambda^{\rm sp}\rangle$ and
$|\lambda^{\rm o}\rangle$ are in $\mathcal{M}$, $\langle\lambda|$ in
$\mathcal{M}^{*}$, while the vectors $\langle\lambda^{\rm sp}|$ and
$\langle\lambda^{\rm o}|$ belong to the completion
$\widetilde{\mathcal{M}^{*}}$. Due to the fact

$\displaystyle\langle 0|Y^{*}_{0}\neq\langle 0|,\qquad\langle
0|W^{*}_{0}\neq\langle 0|,$

the vector $\langle\lambda^{\rm sp}|$ (resp. $\langle\lambda^{\rm o}|$)
differs from the vector $\langle\mu^{\rm sp}|$ (resp. $\langle\mu^{\rm o}|$)
even if $\lambda$ and $\mu$ only differ by a string of zeros at the end. For
examples,

$\displaystyle\langle 0|Y^{*}_{0}Y^{*}_{-1}Y^{*}_{-4}\neq\langle
0|Y^{*}_{-1}Y^{*}_{-4}\neq\langle 0|Y^{*}_{0}Y^{*}_{0}Y^{*}_{-1}Y^{*}_{-4},$
$\displaystyle\langle 0|W^{*}_{0}Y^{*}_{-1}W^{*}_{-4}\neq\langle
0|W^{*}_{-1}W^{*}_{-4}\neq\langle 0|W^{*}_{0}W^{*}_{0}W^{*}_{-1}W^{*}_{-4}.$

For any $\langle u|$ in $\widetilde{\mathcal{M}^{*}}$ and $|v\rangle$ in
$\mathcal{M}^{*}$, one define $\langle u|v\rangle$ by

$\displaystyle\langle u|v\rangle=\langle u||v\rangle,$

where it is assumed that $\langle 0|1|0\rangle=1$. Now we have the following
orthonormality result for three distinguished sets of
$\widetilde{\mathcal{M}^{*}}$.

######  Theorem 2.3.

For $($generalized$)$ partitions $\mu=(\mu_{1},\mu_{2},\dots,\mu_{l})$ and
$\lambda=(\lambda_{1},\lambda_{2},\dots,\lambda_{l})$, one has that

$\displaystyle\langle\mu|\lambda\rangle=\delta_{\lambda\mu},$ (2.10)
$\displaystyle\langle\mu^{\rm sp}|\lambda^{\rm
sp}\rangle=\delta_{\lambda\mu},$ (2.11) $\displaystyle\langle\mu^{\rm
o}|\lambda^{\rm o}\rangle=\delta_{\lambda\mu},$ (2.12)

where the partitions in (2.10) are usual ones and the partitions in (2.11) and
(2.12) are generalized partitions.

###### Proof.

Using the commutation relations (2.6), we have

$\displaystyle\langle\mu|\lambda\rangle$ $\displaystyle=\langle
0|X^{*}_{-\mu_{l}}\cdots
X^{*}_{-\mu_{2}}X^{*}_{-\mu_{1}}X_{-\lambda_{1}}X_{-\lambda_{2}}\cdots
X_{-\lambda_{l}}|0\rangle$ $\displaystyle=\langle 0|X^{*}_{-\mu_{l}}\cdots
X^{*}_{-\mu_{2}}\big{(}\delta_{\mu_{1},\lambda_{1}}-X_{-\lambda_{1}-1}X^{*}_{-\mu_{1}-1}\big{)}X_{-\lambda_{2}}\cdots
X_{-\lambda_{l}}|0\rangle.$

The second summand is zero if $\mu_{1}\geq\lambda_{1}$, since the term
$X^{*}_{-\mu_{1}-1}X_{-\lambda_{2}}\cdots X_{-\lambda_{l}}$ can be rewritten
as $(-1)^{l-1}X_{-\lambda_{2}-1}\cdots X_{-\lambda_{l}-1}X^{*}_{-\mu_{1}-l}$
by (2.6), and thus it kills $|0\rangle$ according to (2.3). If
$\lambda_{1}>\mu_{1}$, we move $X_{-\lambda_{1}-1}$ to the left to kill
$\langle 0|$. Either way shows that the inner product is simplified to

$\displaystyle\langle\mu|\lambda\rangle=\delta_{\lambda_{1},\mu_{1}}\langle
0|X^{*}_{-\mu_{l}}\cdots X^{*}_{-\mu_{2}}X_{-\lambda_{2}}\cdots
X_{-\lambda_{l}}|0\rangle.$

Subsequently $\langle\mu|\lambda\rangle=\delta_{\lambda\mu}$.

In the case of $\langle\mu^{\rm sp}|\lambda^{\rm sp}\rangle$, for
$\mu_{1}\geq\lambda_{1}$ we also have that

$\displaystyle\langle\mu^{\rm sp}|\lambda^{\rm
sp}\rangle=\delta_{\mu_{1},\lambda_{1}}\langle 0|Y^{*}_{-\mu_{l}}\cdots
Y^{*}_{-\mu_{2}}Y_{-\lambda_{2}}\cdots Y_{-\lambda_{l}}|0\rangle.$

Now consider the subcase when $\mu_{1}<\lambda_{1}$. It follows from (2.3)
that

$\displaystyle\langle\mu^{\rm sp}|\lambda^{\rm sp}\rangle$
$\displaystyle\qquad=\begin{cases}0,\quad\mu_{1}+i-1\neq\lambda_{i}\ \text{for
all}\ 1\leq i\leq l,\\\ (-1)^{i-1}\langle 0|Y^{*}_{-\mu_{l}}\cdots
Y^{*}_{-\mu_{2}}Y_{-\lambda_{1}-1}\cdots
Y_{-\lambda_{i-1}-1}Y_{-\lambda_{i+1}}\cdots
Y_{-\lambda_{l}}|0\rangle,\quad\mu_{1}+i-1=\lambda_{i}.\end{cases}\\!$

Continuing the process, $\langle\mu^{\rm sp}|\lambda^{\rm sp}\rangle$ equals
to 0 or $(-1)^{\epsilon}\langle
0|Y^{*}_{-\mu_{l}}Y_{-\lambda_{1}-l+1}|0\rangle$, while

$\langle 0|Y^{*}_{-\mu_{l}}Y_{-\lambda_{1}-l+1}|0\rangle=-\langle
0|Y_{-\lambda_{1}-l}Y^{*}_{-\mu_{l}-1}|0\rangle=0$

since $-\mu_{l}-1<0$ and by (2.3). We therefore obtain (2.11) by combining the
relations between $\mu_{i}$ and $\lambda_{i}$. The relation (2.12) can be
proved similarly. ∎

######  Remark 2.4.

The equation (2.11) makes sense only when we fix a positive integer $l$ and
the length of generalized partition
$\mu=(\mu_{1},\dots,\mu_{m},\underbrace{0,\dots,0}_{l-m})$ is not bigger than
$l$, then

$\displaystyle\langle\mu^{\rm sp}|=\langle 0|\underbrace{Y^{*}_{0}\cdots
Y^{*}_{0}}_{l-m}Y^{*}_{-\mu_{m}}\cdots Y^{*}_{-\mu_{1}}.$

For example, it is easy to check that

$\displaystyle\langle
0|Y^{*}_{0}Y^{*}_{0}Y^{*}_{-1}Y^{*}_{-4}Y_{-4}Y_{-1}Y_{-1}Y_{-1}|0\rangle=0,\qquad\langle
0|Y^{*}_{-1}Y^{*}_{-4}Y_{-4}Y_{-1}Y_{-1}Y_{-1}|0\rangle=-1,$

thus for $\mu=(4,1,0,0)$

$\displaystyle\langle\mu^{\rm sp}|=\langle
0|Y^{*}_{0}Y^{*}_{0}Y^{*}_{-1}Y^{*}_{-4}.$

The same remark applies to the orthogonal case (2.12).

##  3 Skew symplectic/orthogonal Schur functions
and Jacobi–Trudi identities

This section first recalls the vertex operator construction of skew Schur
functions ${\rm s}_{\lambda/\mu}(x)$. Then we extend the method to realize
skew symplectic Schur functions ${\rm sp}_{\lambda/\mu}\big{(}x^{\pm}\big{)}$
and skew orthogonal Schur functions ${\rm
o}_{\lambda/\mu}\big{(}x^{\pm}\big{)}$. Their Jacobi–Trudi formulas are also
provided.

The Jacobi–Trudi identities for the Schur function ${\rm s}_{\lambda}(x)$, the
symplectic Schur function ${\rm sp}_{\lambda}\big{(}x^{\pm}\big{)}$, and the
orthogonal Schur function ${\rm o}_{\lambda}\big{(}x^{\pm}\big{)}$ can be
treated in the same manner by the vertex operator. Recall the classical
Jacobi–Trudi identity for the Schur functions [19, formula (3.4)]

$\displaystyle{\rm s}_{\nu}(x)=\det(h_{\lambda_{i}-i+j}(x))_{1\leq i,j\leq
N},$ (3.1)

where the complete homogeneous function $h_{n}(x)$ is defined by
$\prod^{N}_{i=1}\frac{1}{1-x_{i}z}=\sum_{n\in\mathbb{Z}}h_{n}(x)z^{n}$. For
the symplectic and the orthogonal cases, Schur function ${\rm
sp}_{\lambda}\big{(}x^{\pm}\big{)}$ and orthogonal Schur function ${\rm
o}_{\lambda}\big{(}x^{\pm}\big{)}$ admit the Jacobi–Trudi formulas [14,
Theorems 1.3.2 and 1.3.3] and [12, formulas (4.1) and (4.2)]

$\displaystyle{\rm
sp}_{\nu}\big{(}x^{\pm}\big{)}=\frac{1}{2}\det\big{(}h_{\nu_{i}-i+j}\big{(}x^{\pm}\big{)}+h_{\nu_{i}-i-j+2}\big{(}x^{\pm}\big{)}\big{)}_{1\leq
i,j\leq N},$ (3.2) $\displaystyle{\rm
o}_{\nu}\big{(}x^{\pm}\big{)}=\det\big{(}h_{\nu_{i}-i+j}\big{(}x^{\pm}\big{)}-h_{\nu_{i}-i-j}\big{(}x^{\pm}\big{)}\big{)}_{1\leq
i,j\leq N},$ (3.3)

where $h_{n}\big{(}x^{\pm}\big{)}$ [3, 12, 26] is defined by
$\prod^{N}_{i=1}\frac{1}{(1-x_{i}z)(1-x^{-1}_{i}z)}=\sum_{n\in\mathbb{Z}}h_{n}\big{(}x^{\pm}\big{)}z^{n}$.
It is obvious that $h_{n}(x)=h_{n}\big{(}x^{\pm}\big{)}=0$ for $n<0$.

Introduce the (half) vertex operators

$\displaystyle\Gamma_{+}(z)=\exp\Biggl{(}\sum^{\infty}_{n=1}\frac{a_{n}}{n}z^{n}\Biggr{)},\qquad\Gamma_{-}(z)=\exp\Biggl{(}\sum^{\infty}_{n=1}\frac{a_{-n}}{n}z^{n}\Biggr{)}.$
(3.4)

For $\\{x\\}=(x_{1},\dots,x_{N})$,
$\big{\\{}x^{\pm}\big{\\}}=\big{(}x_{1},x^{-1}_{1},\dots,x_{N},x^{-1}_{N}\big{)}$,
we denote

$\displaystyle\Gamma_{+}(\\{x\\})=\prod^{N}_{i=1}\Gamma_{+}(x_{i}),\qquad\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}=\prod^{N}_{i=1}\Gamma_{+}(x_{i})\Gamma_{+}\big{(}x^{-1}_{i}\big{)},\qquad\Gamma_{-}(\\{x\\})=\prod^{N}_{i=1}\Gamma_{-}(x_{i}).$

###  3.1 Skew Schur functions

For a partition $\mu=(\mu_{1},\dots,\mu_{l})$, one can rewrite
$X^{*}_{-\mu_{l}}\cdots X^{*}_{-\mu_{1}}$ by permuting the factors [10] and
(2.6):

$\displaystyle\varepsilon(\sigma)X^{*}_{-\mu_{l}}\cdots
X^{*}_{-\mu_{1}}=X^{*}_{-\mu_{\sigma(l)}+\sigma(l)-l}\cdots
X^{*}_{-\mu_{\sigma(i)}+\sigma(i)-i}\cdots
X^{*}_{-\mu_{\sigma(1)}+\sigma(1)-1},$ (3.5)

where $\sigma\in S_{l}$. The skew Schur function ${\rm s}_{\lambda/\mu}(x)$
generalizes the Schur function and has a determinantal expression [19, formula
(5.4)] for two partitions $\mu=(\mu_{1},\dots,\mu_{l})$ and
$\lambda=(\lambda_{1},\dots,\lambda_{k})$:

$\displaystyle{\rm
s}_{\lambda/\mu}(x)=\det(h_{\lambda_{i}-\mu_{j}-i+j})_{1\leq i,j\leq k}.$

######  Proposition 3.1.

Let $\mu=(\mu_{1},\dots,\mu_{l})$ and
$\lambda=(\lambda_{1},\dots,\lambda_{k})$ be two partitions. Then

$\displaystyle\langle\mu|\Gamma_{+}(\\{x\\})|\lambda\rangle={\rm
s}_{\lambda/\mu}(x).$ (3.6)

###### Proof.

Using the relations

$\displaystyle\Gamma_{+}(z)=\Gamma_{-}\big{(}z^{-1}\big{)}X^{*}\big{(}z^{-1}\big{)},$
$\displaystyle
X^{*}\big{(}z_{2}^{-1}\big{)}\Gamma_{-}\big{(}z^{-1}_{1}\big{)}=\bigg{(}1-\frac{z_{2}}{z_{1}}\bigg{)}^{-1}\Gamma_{-}\big{(}z^{-1}_{1}\big{)}X^{*}\big{(}z_{2}^{-1}\big{)},$
$\displaystyle\langle 0|\Gamma_{-}(z)=\langle 0|,$

we get that

$\displaystyle\langle 0|\Gamma_{+}(\\{x\\})={}$ $\displaystyle\prod_{1\leq
i<j\leq N}\bigg{(}1-\frac{x_{j}}{x_{i}}\bigg{)}^{-1}\langle
0|X^{*}\big{(}x_{N}^{-1}\big{)}\cdots X^{*}\big{(}x_{1}^{-1}\big{)}$
$\displaystyle={}$ $\displaystyle\prod_{1\leq i<j\leq
N}\bigg{(}1-\frac{x_{j}}{x_{i}}\bigg{)}^{-1}\sum_{n_{1},\dots,n_{N}\in\mathbb{Z}}\langle
0|X^{*}_{-n_{N}}\cdots X^{*}_{-n_{1}}x^{n_{N}}_{N}\cdots x^{n_{1}}_{1}.$

For $\nu=(\nu_{1},\dots,\nu_{N})$, the relation (3.5) says that the
coefficient of $\langle\nu|=\langle 0|X^{*}_{-\nu_{N}}\cdots X^{*}_{-\nu_{1}}$
in $\langle 0|\Gamma_{+}(\\{x\\})$ is

$\displaystyle\prod_{1\leq i<j\leq
N}\bigg{(}1-\frac{x_{j}}{x_{i}}\bigg{)}^{-1}\sum_{\sigma\in
S_{N}}\varepsilon(\sigma)x^{\nu_{\sigma(1)}-\sigma(1)+1}_{1}\cdots
x^{\nu_{\sigma(i)}-\sigma(i)+i}_{i}\cdots x^{\nu_{\sigma(N)}-\sigma(N)+N}_{N}$
$\displaystyle\quad\qquad=\prod_{1\leq i<j\leq
N}(x_{i}-x_{j})^{-1}\sum_{\sigma\in
S_{N}}\varepsilon(\sigma)x^{\nu_{\sigma(1)}-\sigma(1)+N}_{1}\cdots
x^{\nu_{\sigma(i)}-\sigma(i)+N}_{i}\cdots x^{\nu_{\sigma(N)}-\sigma(N)+N}_{N}$
$\displaystyle\quad\qquad=\frac{\det\big{(}x^{\nu_{j}+N-j}_{i}\big{)}_{1\leq
i,j\leq N}}{\prod_{1\leq i<j\leq
N}(x_{i}-x_{j})}=\frac{\det\big{(}x^{\nu_{j}+N-j}_{i}\big{)}_{1\leq i,j\leq
N}}{\det\big{(}x^{N-j}_{i}\big{)}_{1\leq i,j\leq N}}={\rm s}_{\nu}(x),$

where ${\rm s}_{\nu}(x)$ is the Schur function (1.1) associated with the
partition $\nu$. We therefore get the expansion

$\displaystyle\langle
0|\Gamma_{+}(\\{x\\})=\sum_{\begin{subarray}{c}l(\nu)\leq
N\end{subarray}}\langle\nu|{\rm s}_{\nu}(x),$ (3.7)

where the sum is over all partitions $\nu$ with length $\leq N$. It then
follows from (3.5) and the Jacobi–Trudi formula (3.1) that

$\displaystyle\langle 0|\Gamma_{+}(\\{x\\})=\sum_{n_{1},n_{2},\dots,n_{N}\geq
0}\langle 0|X^{*}_{-n_{N}}\cdots X^{*}_{-n_{1}}h_{n_{N}}(x)\cdots
h_{n_{1}}(x).$

By (2.2), we have that

$\displaystyle
X^{*}(z)\Gamma_{+}(\\{x\\})=\prod^{N}_{i=1}\frac{1}{(1-x_{i}z)}\Gamma_{+}(\\{x\\})X^{*}(z)=\sum_{i\geq
0}h_{i}(x)z^{i}\Gamma_{+}(\\{x\\})X^{*}(z),$

i.e.,

$\displaystyle X^{*}_{-n}\Gamma_{+}(\\{x\\})=\Gamma_{+}(\\{x\\})\sum_{i\geq
0}h_{i}(x)X^{*}_{-n-i}.$

It follows from (2.7) that

$\displaystyle\langle\mu|\Gamma_{+}(\\{x\\})={}$ $\displaystyle\langle
0|X^{*}_{-\mu_{l}}\cdots X^{*}_{-\mu_{1}}\Gamma_{+}(\\{x\\})$
$\displaystyle={}$ $\displaystyle\langle
0|\Gamma_{+}(\\{x\\})\sum_{i_{1},\dots,i_{l}\geq
0}X^{*}_{-\mu_{l}-i_{l}}\cdots X^{*}_{-\mu_{1}-i_{1}}h_{i_{l}}(x)\cdots
h_{i_{1}}(x)$ $\displaystyle={}$
$\displaystyle\sum_{i_{l+1},\dots,i_{l+N}\in\mathbb{Z}}\langle
0|X^{*}_{-i_{l+N}}\cdots X^{*}_{-i_{l+1}}h_{i_{l+N}}(x)\cdots h_{i_{l+1}}(x)$
$\displaystyle{}\times\sum_{i_{1},\dots,i_{l}\geq
0}X^{*}_{-\mu_{l}-i_{l}}\cdots X^{*}_{-\mu_{1}-i_{1}}h_{i_{l}}(x)\cdots
h_{i_{1}}(x)$ $\displaystyle={}$
$\displaystyle\sum_{i_{1},\dots,i_{l+N}\in\mathbb{Z}}\langle
0|X^{*}_{-i_{l+N}}\cdots X^{*}_{-i_{l+1}}X^{*}_{-\mu_{l}-i_{l}}\cdots
X^{*}_{-\mu_{1}-i_{1}}$ $\displaystyle{}\times h_{i_{l+N}}(x)\cdots
h_{i_{l+1}}(x)h_{i_{l}}(x)\cdots h_{i_{1}}(x).$ (3.8)

For partition $\lambda$ with $l\leq l(\lambda)\leq l+N$, the relations (3.5)
and (3.8) imply that the coefficient of $\langle\lambda|$ in
$\langle\mu|\Gamma_{+}(\\{x\\})$ is

$\displaystyle\sum_{\sigma\in
S_{l+N}}\epsilon(\sigma)\prod^{l+N}_{j=1}h_{\lambda_{\sigma(j)}-\mu_{j}-\sigma(j)+j}(x)=\det\big{(}h_{\lambda_{i}-\mu_{j}-i+j}(x)\big{)}_{1\leq
i,j\leq l+N}={\rm s}_{\lambda/\mu}(x),$

i.e.,

$\displaystyle\langle\mu|\Gamma_{+}(\\{x\\})=\sum_{\begin{subarray}{c}l\leq
l(\lambda)\leq l+N\end{subarray}}\langle\lambda|{\rm s}_{\lambda/\mu}(x).$

Subsequently (by the orthogonality (2.10))

$\displaystyle\langle\mu|\Gamma_{+}(\\{x\\})|\lambda\rangle={\rm
s}_{\lambda/\mu}(x).$

This completes the proof. ∎

######  Remark 3.2.

Okounkov [21] derived (3.6) by computing $\Gamma_{+}(\\{x\\})|\lambda\rangle$.
We can show (3.6) using the vertex operator approach (see Appendix A). Though
we do not know how to generalize either method to skew-type
orthogonal/symplectic cases, the method of Proposition 3.1 can be generalized
to both cases.

###  3.2 Skew symplectic Schur functions

It follows from (2.4) and (2.6) that

$\displaystyle\begin{aligned} \langle 0|Y^{*}_{-n_{1}}\cdots
Y^{*}_{-n_{i}}\cdots Y^{*}_{-n_{N}}&{}=(-1)^{i-1}\langle
0|Y^{*}_{-n_{i}-i+1}Y^{*}_{-n_{1}+1}\cdots
Y^{*}_{-n_{i-1}+1}Y^{*}_{-n_{i+1}}\cdots Y^{*}_{-n_{N}}\\\ &{}=(-1)^{i}\langle
0|Y^{*}_{n_{i}+i+1}Y^{*}_{-n_{1}+1}\cdots
Y^{*}_{-n_{i-1}+1}Y^{*}_{-n_{i+1}}\cdots Y^{*}_{-n_{N}}\\\ &{}=-\langle
0|Y^{*}_{-n_{1}}\cdots Y^{*}_{-n_{i-1}}Y^{*}_{n_{i}+2i}Y^{*}_{-n_{i+1}}\cdots
Y^{*}_{-n_{N}}.\end{aligned}$

In other words, if we replace the $i$th factor $Y^{*}_{-n_{i}}$ of $\langle
0|Y^{*}_{-n_{1}}\cdots Y^{*}_{-n_{i}}\cdots Y^{*}_{-n_{N}}$ by
$-Y^{*}_{n_{i}+2i}$, it remains the same. We use the notation $(^{a}_{b})$ to
mean either of $a$ or $b$. Then for a generalized partition
$\mu=(\mu_{1},\dots,\mu_{l})$,

$\displaystyle\langle 0|\begin{pmatrix}{Y^{*}_{-\mu_{l}}}\\\
{-Y^{*}_{\mu_{l}+2}}\end{pmatrix}\cdots\begin{pmatrix}{Y^{*}_{-\mu_{i}}}\\\
{-Y^{*}_{\mu_{i}+2(l+1-i)}}\end{pmatrix}\cdots\begin{pmatrix}{Y^{*}_{-\mu_{1}}}\\\
{-Y^{*}_{\mu_{1}+2l}}\end{pmatrix}=\langle\mu^{\rm sp}|.$

Note that there is a similar action of $S_{N}$ on the vectors $\langle
0|Y^{*}_{-n_{1}}\cdots Y^{*}_{-n_{i}}\cdots Y^{*}_{-n_{N}}$ as in (3.5) using
the commutation relation $Y^{*}_{i}Y^{*}_{j}=-Y^{*}_{j-1}Y^{*}_{i+1}$ (2.6).
Therefore, we have that

$\displaystyle\varepsilon(\sigma)\langle
0|\\!\begin{pmatrix}{Y^{*}_{-\mu_{\sigma(l)}+\sigma(l)-l}}\\\
{-Y^{*}_{\mu_{\sigma(l)}-\sigma(l)+l+2}}\end{pmatrix}\\!\cdots\\!\begin{pmatrix}{Y^{*}_{-\mu_{\sigma(i)}+\sigma(i)-i}}\\\
{-Y^{*}_{\mu_{\sigma(i)}-\sigma(i)+2(l+1)-i}}\end{pmatrix}\\!\cdots\\!\begin{pmatrix}{Y^{*}_{-\mu_{\sigma(1)}+\sigma(1)-1}}\\\
{-Y^{*}_{\mu_{\sigma(1)}-\sigma(1)+2l+1}}\end{pmatrix}\\!=\langle\mu^{\rm
sp}|.\\!$ (3.9)

We now define a family of symmetric functions by vertex operators.

######  Proposition 3.3.

Let $\mu=(\mu_{1},\dots,\mu_{l})$ and
$\lambda=(\lambda_{1},\dots,\lambda_{l+N})$ be two generalized partitions, and
$\big{\\{}x^{\pm}\big{\\}}=\big{\\{}x_{1},x_{1}^{-1}\ldots,x_{N},x_{N}^{-1}\big{\\}}$.
Then one has that

$\displaystyle\langle\mu^{\rm
sp}|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}|\lambda^{\rm
sp}\rangle=\det(a_{ij})_{1\leq i,j\leq l+N},$ (3.10)

where

$a_{ij}=\begin{cases}h_{\lambda_{i}-\mu_{j}-i+j}\big{(}x^{\pm}\big{)},&1\leq
j\leq l+1,\\\
h_{\lambda_{i}-i+j}\big{(}x^{\pm}\big{)}+h_{\lambda_{i}-i-j+2l+2}\big{(}x^{\pm}\big{)},&l+2\leq
j\leq l+N.\end{cases}$ (3.11)

We call the symmetric function (3.10) as the skew symplectic Schur function
${\rm sp}_{\lambda/\mu}\big{(}x^{\pm}\big{)}$ , i.e.,

$\displaystyle{\rm sp}_{\lambda/\mu}\big{(}x^{\pm}\big{)}=\det(a_{ij})_{1\leq
i,j\leq l+N}=\langle\mu^{\rm
sp}|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}|\lambda^{\rm
sp}\rangle.$ (3.12)

We will justify this by showing these symmetric functions obey the branching
rule, therefore they agree with the skew symplectic Schur functions defined by
Koike and Terada [15].

###### Proof.

By definition of the vertex operator $Y^{*}(z)$, one has that

$\displaystyle\langle 0|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}=$
$\displaystyle\prod_{1\leq i\leq j\leq N}\frac{1}{1-x_{i}x_{j}}\prod_{1\leq
i<j\leq N}\frac{1}{1-\frac{x_{i}}{x_{j}}}\langle 0|Y^{*}(x_{N})\cdots
Y^{*}(x_{1})$ $\displaystyle=$
$\displaystyle\frac{\prod^{N}_{i=1}x^{i-1-N}_{i}}{\det\big{(}x^{-(N-j+1)}_{i}-x^{N-j+1}_{i}\big{)}^{N}_{i,j=1}}\langle
0|Y^{*}(x_{N})\cdots Y^{*}(x_{1}),$

where we have used the Vandermonde type identity [12, formula (4.6)] and [26,
p. 229]:

$\displaystyle\det\big{(}x^{j-1}_{i}-x^{2N-j+1}_{i}\big{)}^{N}_{i,j=1}=\prod_{1\leq
i<j\leq N}(x_{j}-x_{i})\prod_{1\leq i\leq j\leq N}(1-x_{i}x_{j}).$

In view of (3.9) for partition $\nu=(\nu_{1},\dots,\nu_{N})$, the coefficient
of $\langle\nu^{\rm sp}|$ in $\langle 0|Y^{*}(x_{N})\cdots Y^{*}(x_{1})$ is

$\displaystyle\sum_{\sigma\in
S_{N}}\varepsilon(\sigma)\prod^{N}_{i=1}\big{(}x^{-\nu_{\sigma(i)}+\sigma(i)-i}_{i}-x^{\nu_{\sigma(i)}-\sigma(i)+2N+2-i}_{i}\big{)}=\det\big{(}x^{-\nu_{j}+j-i}_{i}-x^{\nu_{j}-j+2N+2-i}_{i}\big{)}^{N}_{i,j=1}.$

Then we have

$\displaystyle\langle 0|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}=$
$\displaystyle\sum_{\nu=(\nu_{1},\dots,\nu_{N})}\langle\nu^{\rm
sp}|\frac{\prod^{N}_{i=1}x^{i-1-N}_{i}\det\big{(}x^{-\nu_{j}+j-i}_{i}-x^{\nu_{j}-j+2N+2-i}_{i}\big{)}^{N}_{i,j=1}}{\det\big{(}x^{-(N-j+1)}_{i}-x^{N-j+1}_{i}\big{)}^{N}_{i,j=1}}$
$\displaystyle=$
$\displaystyle\sum_{\nu=(\nu_{1},\dots,\nu_{N})}\langle\nu^{\rm
sp}|\frac{\det\big{(}x^{-\nu_{j}-(N-j+1)}_{i}-x^{\nu_{j}+(N-j+1)}_{i}\big{)}^{N}_{i,j=1}}{\det\big{(}x^{-(N-j+1)}_{i}-x^{N-j+1}_{i}\big{)}^{N}_{i,j=1}}$
$\displaystyle=$
$\displaystyle\sum_{\nu=(\nu_{1},\dots,\nu_{N})}\langle\nu^{\rm
sp}|\frac{\det\big{(}x^{\nu_{j}+(N-j+1)}_{i}-x^{-\nu_{j}-(N-j+1\big{)}}_{i})^{N}_{i,j=1}}{\det\big{(}x^{N-j+1}_{i}-x^{-(N-j+1)}_{i}\big{)}^{N}_{i,j=1}}$
$\displaystyle=$
$\displaystyle\sum_{\nu=(\nu_{1},\dots,\nu_{N})}\langle\nu^{\rm sp}|{\rm
sp}_{\nu}\big{(}x^{\pm}\big{)},$ (3.13)

where the sum is over all generalized partitions $\nu=(\nu_{1},\dots,\nu_{N})$
and we have used the bialternant formula of the symplectic Schur function
${\rm sp}_{\nu}\big{(}x^{\pm}\big{)}$ (see (1.2)) associated to the partition
$\nu$. Invoking (3.9) again and also by the Jacobi–Trudi formula for the
symplectic Schur functions (3.2), we obtain another expression for $\langle
0|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}$:

$\displaystyle\langle 0|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}$
(3.14) $\displaystyle\qquad=\sum_{n_{1}\geq-N+1,n_{2}\geq-N+2,\dots,n_{N}\geq
0}\langle 0|Y^{*}_{-n_{N}}\cdots
Y^{*}_{-n_{1}}h_{n_{1}}\big{(}x^{\pm}\big{)}\prod^{N}_{i=2}\big{(}h_{n_{i}}\big{(}x^{\pm}\big{)}+h_{n_{i}-2i+2}\big{(}x^{\pm}\big{)}\big{)}.$

By definition of vertex operator $Y^{*}(z)$, we have

$\displaystyle{}Y^{*}(z)\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}$
$\displaystyle=\prod^{N}_{i=1}\frac{1}{(1-x_{i}z)\big{(}1-x^{-1}_{i}z\big{)}}\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}Y^{*}(z)$
$\displaystyle=\sum_{i\geq
0}h_{i}\big{(}x^{\pm}\big{)}z^{i}\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}Y^{*}(z),$

which implies

$\displaystyle
Y^{*}_{-n}\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}=\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}\sum_{i\geq
0}h_{i}\big{(}x^{\pm}\big{)}Y^{*}_{-n-i}.$

It then follows from (2.8), (3.14) and (3.9) that

$\displaystyle\langle\mu^{\rm
sp}|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}=\sum_{i_{1},\dots,i_{l}\geq
0}h_{i_{1}}\big{(}x^{\pm}\big{)}\cdots h_{i_{l}}\big{(}x^{\pm}\big{)}\langle
0|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}Y^{*}_{-\mu_{l}-i_{l}}\cdots
Y^{*}_{-\mu_{1}-i_{1}}$ $\displaystyle\hphantom{\langle\mu^{\rm
sp}|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}}{}=\sum_{\begin{subarray}{c}i_{1},\dots,i_{l}\geq
0\\\ i_{l+1}\geq-N+1,\dots,i_{l+N}\geq 0\end{subarray}}\langle
0|Y^{*}_{-i_{l+N}}\cdots Y^{*}_{-i_{l+1}}Y^{*}_{-\mu_{l}-i_{l}}\cdots
Y^{*}_{-\mu_{1}-i_{1}}\prod^{l+1}_{k=1}h_{i_{k}}\big{(}x^{\pm}\big{)}$
$\displaystyle\hphantom{\langle\mu^{\rm
sp}|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}=}{}\times\prod^{l+N}_{k=l+2}\big{(}h_{i_{k}}\big{(}x^{\pm}\big{)}+h_{i_{k}-2(k-l)+2}\big{(}x^{\pm}\big{)}\big{)}$
$\displaystyle\hphantom{\langle\mu^{\rm
sp}|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}}{}=\sum_{\lambda=(\lambda_{1},\dots,\lambda_{l+N})}\langle\lambda^{\rm
sp}|\sum_{\sigma\in
S_{l+N}}\prod^{l+1}_{k=1}h_{\lambda_{\sigma(k)}-\mu_{k}-\sigma(k)+k}\big{(}x^{\pm}\big{)}$
$\displaystyle\hphantom{\langle\mu^{\rm
sp}|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}=}{}\times\prod^{l+N}_{k=l+2}\big{(}h_{\lambda_{\sigma(k)}-\sigma(k)+k}\big{(}x^{\pm}\big{)}+h_{\lambda_{\sigma(k)}-\sigma(k)-k+2l+2}\big{(}x^{\pm}\big{)}\big{)}$
$\displaystyle\hphantom{\langle\mu^{\rm
sp}|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}}{}=\sum_{\lambda=(\lambda_{1},\dots,\lambda_{l+N})}\langle\lambda^{\rm
sp}|\det(a_{ij})_{1\leq i,j\leq l+N},$ (3.15)

where $a_{ij}$ are defined in (3.11) and in the last two equations
$\lambda_{\sigma(i)}-\sigma(i)+2(l+N+1)-i$ is greater than $l+N-i$. ∎

Using the orthogonality relation (2.11), for generalized partition
$\nu=(\nu_{1},\dots,\nu_{n})$, we have

$\displaystyle\langle\nu^{\rm
sp}|\sum_{\eta=(\eta_{1},\dots,\eta_{n})}|\eta^{\rm sp}\rangle\langle\eta^{\rm
sp}|=\langle\nu^{\rm
sp}|,\qquad\sum_{\eta=(\eta_{1},\dots,\eta_{n})}|\eta^{\rm
sp}\rangle\langle\eta^{\rm sp}||\nu^{\rm sp}\rangle=|\nu^{\rm sp}\rangle,$
(3.16)

where the sum is over all generalized partitions
$\eta=(\eta_{1},\dots,\eta_{n})$.

######  Proposition 3.4.

For $x^{\pm}=\big{(}x_{1},x^{-1}_{1},\dots,x_{n-k},x^{-1}_{n-k}\big{)}$ and
$y^{\pm}=\big{(}y_{1},y^{-1}_{1},\dots,y_{k},y^{-1}_{k}\big{)}$, the functions
${\rm sp}_{\lambda/\mu}$ satisfy the general branching rule

$\displaystyle{\rm
sp}_{\lambda}\big{(}x^{\pm};y^{\pm}\big{)}=\sum_{\mu=(\mu_{1},\dots,\mu_{n-k})}{\rm
sp}_{\mu}\big{(}x^{\pm}\big{)}{\rm sp}_{\lambda/\mu}\big{(}y^{\pm}\big{)}.$
(3.17)

###### Proof.

It follows from (3.10) that

$\displaystyle\langle 0|\Gamma_{+}\big{(}x^{\pm};y^{\pm}\big{)}|\lambda^{\rm
sp}\rangle={\rm sp}_{\lambda}\big{(}x^{\pm};y^{\pm}\big{)}.$ (3.18)

By (3.16),

$\displaystyle\langle 0|\Gamma_{+}\big{(}x^{\pm};y^{\pm}\big{)}|\lambda^{\rm
sp}\rangle$ $\displaystyle=\langle
0|\Gamma_{+}\big{(}x^{\pm}\big{)}\sum_{\mu=(\mu_{1},\dots,\mu_{n-k})}|\mu^{\rm
sp}\rangle\langle\mu^{\rm sp}|\Gamma_{+}\big{(}y^{\pm}\big{)}|\lambda^{\rm
sp}\rangle$ $\displaystyle=\sum_{\mu=(\mu_{1},\dots,\mu_{n-k})}{\rm
sp}_{\mu}\big{(}x^{\pm}\big{)}{\rm sp}_{\lambda/\mu}\big{(}y^{\pm}\big{)}.$
(3.19)

Combining (3.18) and (3.19), we obtain the result. ∎

######  Remark 3.5.

Since ${\rm sp}_{\lambda/\mu}$ satisfies the general branching rule (3.17) and
reduces to the symplectic Schur function when $\mu=\varnothing$, it is
rightfully called the skew symplectic Schur function. For $k=1$, the general
branching rule (3.17) reduces to the branching rule for Koike–Terada’s
symplectic Schur functions [15, Theorem 3.1]. In fact, ${\rm
sp}_{\lambda/\mu}$ agrees with that defined by Koike and Terada by using the
forthcoming Gelfand–Tsetlin patterns (4.2).

######  Remark 3.6.

We stress that the partitions $\mu$ of the skew symplectic Schur function
${\rm sp}_{\lambda/\mu}$ in (3.10) and (3.12) are general partitions, i.e.,
some parts can be zeros. For instance, the symmetric function $\langle
0|Y^{*}_{-1}\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}Y_{-2}Y_{-1}Y_{-1}|0\rangle$
is the skew symplectic Schur function ${\rm
sp}_{(2,1,1)/(1)}\big{(}x^{\pm}\big{)}$, and $\langle
0|Y^{*}_{0}Y^{*}_{-1}\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}Y_{-2}Y_{-1}Y_{-1}|0\rangle$
gives different expression ${\rm sp}_{(2,1,1,0)/(1,0)}\big{(}x^{\pm}\big{)}$.
The same statement applies to the skew orthogonal case (3.23).

######  Remark 3.7.

For two generalized partitions $\mu=(\mu_{1},\dots,\mu_{l})$ and
$\lambda=\big{(}\lambda_{1},\dots,\lambda_{l+1},0^{N-1}\big{)}$, the formula
(3.12) implies that ${\rm sp}_{\lambda/\mu}\big{(}x^{\pm}\big{)}={\rm
s}_{\lambda/\mu}\big{(}x^{\pm}\big{)}$ since the matrix $(a_{ij})$ is a block
upper-unipotent one. In particular, if $l=N$ or $l=N-1$, the skew symplectic
Schur function ${\rm
sp}_{(\lambda_{1},\dots,\lambda_{N},0^{l})/(0^{l})}\big{(}x^{\pm}\big{)}$ is
equal to the Schur function ${\rm
s}_{(\lambda_{1},\dots,\lambda_{N})}\big{(}x^{\pm}\big{)}$ for

$\big{\\{}x^{\pm}\big{\\}}=\big{(}x_{1},x^{-1}_{1},\dots,x_{N},x^{-1}_{N}\big{)}.$

One can also get the special cases of skew symplectic Schur functions via
Gelfand–Tsetlin patterns in the following.

###  3.3 Skew orthogonal Schur functions

We can treat the orthogonal Schur functions similarly. Using the commutation
relations

$\displaystyle\langle 0|W^{*}_{n}=\langle 0|W^{*}_{-n},\qquad
W^{*}_{i}W^{*}_{j}=-W^{*}_{j-1}W^{*}_{i+1},$

we see that the partition elements (2.9) with partitions
$\mu=(\mu_{1},\dots,\mu_{l})$ can be expressed as

$\displaystyle\varepsilon(\sigma)\langle
0|\begin{pmatrix}{W^{*}_{-\mu_{\sigma(l)}+\sigma(l)-l}}\\\
{\delta_{l}W^{*}_{\mu_{\sigma(l)}-\sigma(l)+l}}\end{pmatrix}\cdots\begin{pmatrix}{W^{*}_{-\mu_{\sigma(i)}+\sigma(i)-i}}\\\
{\delta_{i}W^{*}_{\mu_{\sigma(i)}-\sigma(i)+2l-i}}\end{pmatrix}\cdots\begin{pmatrix}{W^{*}_{-\mu_{\sigma(1)}+\sigma(1)-1}}\\\
{\delta_{1}W^{*}_{\mu_{\sigma(1)}-\sigma(1)+2l-1}}\end{pmatrix}=\langle\mu^{\rm
o}|,$ (3.20)

where $\delta_{i}$ denotes $\delta_{\mu_{l}\neq 0}\delta_{\sigma(i)\neq l}$.

Recall the Vandermonde type identity [12, formula (4.4)] and [26, p. 221] in
this case

$\displaystyle\det\big{(}x^{N-j}_{i}+x^{N+j-2}_{i}\big{)}^{N}_{i,j=1}=2\prod_{1\leq
i<j\leq N}(x_{i}-x_{j})(1-x_{i}x_{j}).$

Therefore,

$\displaystyle\langle 0|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}$
$\displaystyle\qquad=\sum_{\nu=(\nu_{1},\dots,\nu_{N})}\langle\nu^{\rm
o}|2(-1)^{\frac{N(N-1)}{2}}\frac{\prod^{N}_{i=1}x^{i-N}_{i}\det\big{(}x^{-\nu_{j}+j-i}_{i}+\delta_{j\neq
N}\delta_{\nu_{N}\neq
0}x^{\nu_{j}-j+2N-i}_{i}\big{)}^{N}_{i,j=1}}{\det\big{(}x^{-j+1}_{i}+x^{j-1}_{i}\big{)}^{N}_{i,j=1}}$
$\displaystyle\qquad=\sum_{\nu=(\nu_{1},\dots,\nu_{N})}\langle\nu^{\rm
o}|2(-1)^{\frac{N(N-1)}{2}}\frac{\det\big{(}x^{-\nu_{j}+j-N}_{i}+\delta_{j\neq
N}\delta_{\nu_{N}\neq
0}x^{\nu_{j}-j+N}_{i}\big{)}^{N}_{i,j=1}}{\det\big{(}x^{-j+1}_{i}+x^{j-1}_{i}\big{)}^{N}_{i,j=1}}$
$\displaystyle\qquad=\sum_{\nu=(\nu_{1},\dots,\nu_{N})}\langle\nu^{\rm o}|{\rm
o}_{\nu}\big{(}x^{\pm}\big{)},$ (3.21)

where the sum is over all partitions $\nu=(\nu_{1},\dots,\nu_{N})$ and ${\rm
o}_{\nu}\big{(}x^{\pm}\big{)}$ is the orthogonal Schur function associated to
partition $\nu$ (see (1.3)). It follows from (3.20) and the Jacobi–Trudi
formula (3.3) that

$\displaystyle\langle 0|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}$
$\displaystyle\quad\qquad{}=\sum_{n_{1}\geq-N+1,n_{2}\geq-N+2,\dots,n_{N}\geq
0}\langle 0|W^{*}_{-n_{N}}\cdots
W^{*}_{-n_{1}}\prod^{N}_{i=1}\big{(}h_{n_{i}}\big{(}x^{\pm}\big{)}-h_{n_{i}-2i}\big{(}x^{\pm}\big{)}\big{)}.$
(3.22)

The following proposition can be proved by the same method of Proposition 3.3,
(3.20), and (3.22).

######  Proposition 3.8.

For generalized partitions $\mu=(\mu_{1},\dots,\mu_{l})$ and
$\lambda=(\lambda_{1},\dots,\lambda_{l+N})$, and
$\big{\\{}x^{\pm}\big{\\}}=\big{\\{}x_{1},x_{1}^{-1},\ldots,x_{N},x_{N}^{-1}\big{\\}}$,
one has that

$\displaystyle\langle\mu^{\rm
o}|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}|\lambda^{\rm
o}\rangle=\det(b_{ij})^{l+N}_{i,j=1},$ (3.23)

where

$\displaystyle
b_{ij}=\begin{cases}h_{\lambda_{i}-\mu_{j}-i+j}\big{(}x^{\pm}\big{)},&1\leq
j\leq l,\\\
h_{\lambda_{i}-i+j}\big{(}x^{\pm}\big{)}-h_{\lambda_{i}-i-j+2l}\big{(}x^{\pm}\big{)},&l+1\leq
j\leq l+N.\end{cases}$

We also call the symmetric function in (3.23) the skew Schur orthogonal
function ${\rm o}_{\lambda/\mu}(x)$. In the following, we will justify the
definition.

Using the orthogonality relation (2.12), for generalized partition
$\nu=(\nu_{1},\dots,\nu_{n})$, one has

$\displaystyle\langle\nu^{\rm
o}|\sum_{\eta=(\eta_{1},\dots,\eta_{n})}|\eta^{\rm o}\rangle\langle\eta^{\rm
o}|=\langle\nu^{\rm o}|,\qquad\sum_{\eta=(\eta_{1},\dots,\eta_{n})}|\eta^{\rm
o}\rangle\langle\eta^{\rm o}||\nu^{\rm o}\rangle=|\nu^{\rm o}\rangle,$ (3.24)

where the sum is over all generalized partitions
$\eta=(\eta_{1},\dots,\eta_{n})$. Similar to the proof of Proposition 3.4, we
have the following statement.

######  Proposition 3.9.

For $x^{\pm}=\big{(}x_{1},x^{-1}_{1},\dots,x_{n-k},x^{-1}_{n-k}\big{)}$ and
$y^{\pm}=\big{(}y_{1},y^{-1}_{1},\dots,y_{k},y^{-1}_{k}\big{)}$, the functions
${\rm o}_{\lambda/\mu}$ satisfy the general branching rule

$\displaystyle{\rm
o}_{\lambda}\big{(}x^{\pm};y^{\pm}\big{)}=\sum_{\mu=(\mu_{1},\dots,\mu_{n-k})}o_{\mu}\big{(}x^{\pm}\big{)}{\rm
o}_{\lambda/\mu}\big{(}y^{\pm}\big{)}.$ (3.25)

######  Remark 3.10.

Based on the general branching rule for orthogonal Schur function (3.25), the
function ${\rm o}_{\lambda/\mu}$ (3.23) is rightfully called the skew
orthogonal Schur function. For $k=1$, the general branching rule (3.25)
reduces to the branching rule for the orthogonal Schur function [15, Theorem
3.2].

##  4 Gelfand–Tsetlin pattern representations
for three skew-type functions

In this section, we first derive the formulas for skew Schur functions in
Gelfand–Tsetlin patterns using vertex operators. We then do the same for skew
symplectic/orthogonal Schur functions.

We start by recalling a special case of skew Schur functions in terms of
vertex operators [11, Proposition 2.4]. We say a (generalized) partition
$\lambda$ interlaces another (generalized) partition $\nu$, denoted as
$\nu\prec\lambda$, if $\lambda_{i}\geq\nu_{i}\geq\lambda_{i+1}$ for all $i$.

######  Lemma 4.1 ([19, p. 72]).

For two interlacing partitions $\nu\prec\lambda$,
$\nu=(\nu_{1},\dots,\nu_{l})$ and $\lambda=(\lambda_{1},\dots,\lambda_{l+1})$,
one has that

$\displaystyle{\rm s}_{\lambda/\nu}(t)=t^{|\lambda|-|\nu|}.$

It follows from Lemma 4.1 that skew Schur functions can be written as a sum
over Gelfand–Tsetlin patterns: for partitions
$\mu=(\mu_{1},\dots,\mu_{l})\subset\lambda=(\lambda_{1},\dots,\lambda_{l+N})$

$\displaystyle{\rm s}_{\lambda/\mu}(x)=\sum_{\mu=z_{0}\prec
z_{1}\prec\dots\prec z_{N}=\lambda}\prod^{N}_{i=1}x^{|z_{i}|-|z_{i-1}|}_{i},$

summed over all sequences of partitions $\mu=z_{0}\prec z_{1}\prec\dots\prec
z_{N}=\lambda$, where $z_{i}=(z_{i,1},z_{i,2},\dots,\allowbreak z_{i,l+i})$
satisfy the Gelfand–Tsetlin pattern:

$\displaystyle z_{i+1,j}\leq z_{i,j-1}\leq z_{i+1,j-1}\qquad\text{for}\quad
2\leq j\leq l+i+1.$

Using the similar method as in [11], we can get the Gelfand–Tsetlin pattern
formulas for skew symplectic (orthogonal) Schur functions ${\rm
sp}_{\lambda/\mu}\big{(}x^{\pm}\big{)}$ (${\rm
o}_{\lambda/\mu}\big{(}x^{\pm}\big{)}$). We start by proving three lemmas.

######  Lemma 4.2.

For any integer $m$ and generalized partitions
$\alpha=(\alpha_{1},\dots,\alpha_{k})$, $\beta=(\beta_{1},\dots,\beta_{k+2})$,
one has

$\displaystyle\langle\alpha^{\rm sp}|\sum_{i\leq m,\,j\leq
m+1}Y^{*}_{i}Y^{*}_{j}t^{i+j}|\beta^{\rm sp}\rangle=0.$

###### Proof.

From (2.3) and (2.6), we know that $Y^{*}_{j}|\beta^{\rm sp}\rangle=0$ for
$j<-\beta_{1}$. Therefore,

$\displaystyle\sum_{i\leq m,\,j\leq m+1}Y^{*}_{i}Y^{*}_{j}t^{i+j}|\beta^{\rm
sp}\rangle$ $\displaystyle=\sum_{-\beta_{1}-1\leq i\leq m,\,-\beta_{1}\leq
j\leq m+1}Y^{*}_{i}Y^{*}_{j}t^{i+j}|\beta^{\rm sp}\rangle$
$\displaystyle=-\sum_{-\beta_{1}-1\leq i\leq m,\,-\beta_{1}\leq j\leq
m+1}Y^{*}_{j-1}Y^{*}_{i+1}t^{i+j}|\beta^{\rm sp}\rangle$
$\displaystyle=-\sum_{-\beta_{1}-1\leq a\leq m,\,-\beta_{1}\leq b\leq
m+1}Y^{*}_{a}Y^{*}_{b}t^{a+b}|\beta^{\rm sp}\rangle,$

therefore $\sum_{i\leq m,\,j\leq m+1}Y^{*}_{i}Y^{*}_{j}t^{i+j}|\beta^{\rm
sp}\rangle=0$, which completes the proof. ∎

######  Corollary 4.3.

For any integer $m$ and generalized partition
$\alpha=(\alpha_{1},\dots,\alpha_{k})$,

$\displaystyle\langle\alpha^{\rm sp}|\sum_{i\leq m,\,j\leq
m+1}Y^{*}_{i}Y^{*}_{j}t^{i+j}=0.$

######  Lemma 4.4.

For generalized partitions $\nu=(\nu_{1},\dots,\nu_{l})$ and
$\lambda=(\lambda_{1},\dots,\lambda_{l+1})$, one has

$\displaystyle{\rm
sp}_{\lambda/\nu}\big{(}t^{\pm}\big{)}=\sum_{\nu\prec\alpha\prec\lambda}t^{2|\alpha|-|\lambda|-|\nu|}$

summed over all generalized partitions $\alpha$ such that
$\nu\prec\alpha\prec\lambda$.

###### Proof.

It follows from equation (3.15) that

$\displaystyle\langle\nu^{\rm
sp}|\Gamma_{+}\big{(}t^{\pm}\big{)}=\sum_{i,i_{1},\dots,i_{l}\geq
0}h_{i}\big{(}t^{\pm}\big{)}\prod^{l}_{j=1}h_{i_{j}}\big{(}t^{\pm}\big{)}\langle
0|Y^{*}_{-i}Y^{*}_{-\nu_{l}-i_{l}}\cdots Y^{*}_{-\nu_{1}-i_{1}},$ (4.1)

where
$h_{i}\big{(}t^{\pm}\big{)}=t^{-i}+t^{-i+2}+\dots+t^{i-2}+t^{i}=\frac{t^{-i}-t^{i+2}}{1-t^{2}}$.
Since $Y^{*}_{j-1}Y^{*}_{j}=0$ and Corollary 4.3, we have

$\displaystyle\sum_{i\geq\nu_{l}+1,j\geq\nu_{l}}\langle
0|Y^{*}_{-i}Y^{*}_{-j}\bigl{(}t^{-i}-t^{i+2}\bigr{)}\bigl{(}t^{\nu_{l}-j}-t^{j-\nu_{l}+2}\bigr{)}$
$\displaystyle\qquad\quad{}=\sum_{i\geq\nu_{l}+1,j\geq\nu_{l}}\langle
0|Y^{*}_{-i}Y^{*}_{-j}\bigl{(}-t^{i+2+\nu_{l}-j}-t^{j-\nu_{l}+2-i}\bigr{)}$
$\displaystyle\qquad\quad{}=\sum_{j\geq i\geq\nu_{l}+1}\langle
0|Y^{*}_{-i}Y^{*}_{-j}\bigl{(}-t^{i+2+\nu_{l}-j}-t^{j-\nu_{l}+2-i}\bigr{)}$
$\displaystyle\qquad\qquad{}+\sum_{i-1>j\geq\nu_{l}}\langle
0|Y^{*}_{-i}Y^{*}_{-j}\bigl{(}-t^{i+2+\nu_{l}-j}-t^{j-\nu_{l}+2-i}\bigr{)}$
$\displaystyle\qquad\quad{}=\sum_{j\geq i\geq\nu_{l}+1}\langle
0|Y^{*}_{-i}Y^{*}_{-j}\bigl{(}-t^{i+2+\nu_{l}-j}-t^{j-\nu_{l}+2-i}\bigr{)}$
$\displaystyle\qquad\qquad{}+\sum_{i-1>j\geq\nu_{l}}\langle
0|Y^{*}_{-j-1}Y^{*}_{-i+1}\bigl{(}t^{i+2+\nu_{l}-j}+t^{j-\nu_{l}+2-i}\bigr{)}$
$\displaystyle\qquad\quad{}=\sum_{j\geq i\geq\nu_{l}+1}\langle
0|Y^{*}_{-i}Y^{*}_{-j}\bigl{(}-t^{i+2+\nu_{l}-j}-t^{j-\nu_{l}+2-i}\bigr{)}$
$\displaystyle\qquad\qquad{}+\sum_{n\geq m\geq\nu_{l}+1}\langle
0|Y^{*}_{-m}Y^{*}_{-n}\bigl{(}t^{n+4+\nu_{l}-m}+t^{m-\nu_{l}-n}\bigr{)}$
$\displaystyle\qquad\quad{}=\sum_{j\geq i\geq\nu_{l}+1}\langle
0|Y^{*}_{-i}Y^{*}_{-j}\bigl{(}t^{j+4+\nu_{l}-i}+t^{i-\nu_{l}-j}-t^{i+2+\nu_{l}-j}-t^{j-\nu_{l}+2-i}\bigr{)}$
$\displaystyle\qquad\quad{}=\sum_{j\geq i\geq\nu_{l}+1}\langle
0|Y^{*}_{-i}Y^{*}_{-j}\bigl{(}t^{i-j}-t^{j-i+2}\bigr{)}\bigl{(}t^{-\nu_{l}}-t^{2+\nu_{l}}\bigr{)},$

where we have split $i\geq\nu_{l}+1$, $j\geq\nu_{l}$ into $i-j=1$, $i-j>1$ and
$i-j\leq 0$, in other words, $i=j+1$, $i-1>j\geq\nu_{l}$, $j\geq
i\geq\nu_{l}+1$. Thus

$\displaystyle\sum_{i,i_{l}\geq
0}h_{i}\big{(}t^{\pm}\big{)}h_{i_{l}}\big{(}t^{\pm}\big{)}\langle
0|Y^{*}_{-i}Y^{*}_{-\nu_{l}-i_{l}}$
$\displaystyle\quad{}=\frac{1}{\big{(}1-t^{2}\big{)}^{2}}\sum_{i,i_{l}\geq
0}\langle
0|Y^{*}_{-i}Y^{*}_{-\nu_{l}-i_{l}}\big{(}t^{-i}-t^{i+2}\big{)}\big{(}t^{-i_{l}}-t^{i_{l}+2}\big{)}$
$\displaystyle\quad{}=\sum_{0\leq i\leq\nu_{l},\,i_{l}\geq 0}\langle
0|Y^{*}_{-i}Y^{*}_{-\nu_{l}-i_{l}}h_{i}\big{(}t^{\pm}\big{)}h_{i_{l}}\big{(}t^{\pm}\big{)}$
$\displaystyle\qquad{}+\frac{1}{\big{(}1-t^{2}\big{)}^{2}}\sum_{i\geq\nu_{l}+1,\,i_{l}\geq
0}\langle
0|Y^{*}_{-i}Y^{*}_{-\nu_{l}-i_{l}}\big{(}t^{-i}-t^{i+2}\big{)}\big{(}t^{-i_{l}}-t^{i_{l}+2}\big{)}$
$\displaystyle\quad{}=\sum_{0\leq i\leq\nu_{l},\,i_{l}\geq 0}\langle
0|Y^{*}_{-i}Y^{*}_{-\nu_{l}-i_{l}}h_{i}\big{(}t^{\pm}\big{)}h_{i_{l}}\big{(}t^{\pm}\big{)}+\sum_{\nu_{l}+i_{l}\geq
i\geq\nu_{l}+1}\langle
0|Y^{*}_{-i}Y^{*}_{-\nu_{l}-i_{l}}h_{\nu_{l}}\big{(}t^{\pm}\big{)}h_{\nu_{l}+i_{l}-i}\big{(}t^{\pm}\big{)}.$

Setting $\gamma_{1}=\nu_{l}+i_{l}\geq\gamma_{2}=i\geq 0$, then the above
equation becomes

$\displaystyle\sum_{i,i_{l}\geq
0}h_{i}\big{(}t^{\pm}\big{)}h_{i_{l}}\big{(}t^{\pm}\big{)}\langle
0|Y^{*}_{-i}Y^{*}_{-\nu_{l}-i_{l}}$
$\displaystyle\qquad\quad{}=\sum_{\max\\{\gamma_{2},\nu_{l}\\}\leq\gamma_{1}}\langle
0|Y^{*}_{-\gamma_{2}}Y^{*}_{-\gamma_{1}}h_{\min\\{\gamma_{2},\nu_{l}\\}}\big{(}t^{\pm}\big{)}h_{\gamma_{1}-\max\\{\gamma_{2},\nu_{l}\\}}\big{(}t^{\pm}\big{)}.$

Continuing the process, equation (4.1) becomes

$\displaystyle\langle\nu^{\rm sp}|\Gamma_{+}\big{(}t^{\pm}\big{)}$
$\displaystyle\qquad\quad{}=\sum_{\lambda}\langle\lambda^{\rm
sp}|h_{\min\\{\lambda_{l+1},\nu_{l}\\}}\big{(}t^{\pm}\big{)}\Bigg{(}\prod^{l}_{j=2}h_{\min\\{\lambda_{j},\nu_{j-1}\\}-\max\\{\lambda_{j+1},\nu_{j}\\}}\big{(}t^{\pm}\big{)}\Bigg{)}h_{\lambda_{1}-\max\\{\lambda_{2},\nu_{1}\\}}\big{(}t^{\pm}\big{)}$
$\displaystyle\qquad\quad{}=\sum_{\lambda}\langle\lambda^{\rm
sp}|h_{\min\\{\lambda_{l+1},\nu_{l}\\}}\big{(}t^{\pm}\big{)}t^{\min\\{\lambda_{l+1},\nu_{l}\\}}\Bigg{(}\prod^{l}_{j=2}h_{\min\\{\lambda_{j},\nu_{j-1}\\}-\max\\{\lambda_{j+1},\nu_{j}\\}}\big{(}t^{\pm}\big{)}$
$\displaystyle\qquad\quad\hphantom{=\sum_{\lambda}}{}\times
t^{\min\\{\lambda_{j},\nu_{j-1}\\}+\max\\{\lambda_{j+1},\nu_{j}\\}}\Bigg{)}h_{\lambda_{1}-\max\\{\lambda_{2},\nu_{1}\\}}\big{(}t^{\pm}\big{)}t^{\lambda_{1}+\max\\{\lambda_{2},\nu_{1}\\}}t^{-|\lambda|-|\nu|}$
$\displaystyle\qquad\quad{}=\sum_{\lambda}\langle\lambda^{\rm
sp}|\sum_{\nu\prec\alpha\prec\lambda}t^{2|\alpha|-|\lambda|-|\nu|},$

where we use the fact that

$|\lambda|+|\nu|=\min\\{\lambda_{l+1},\nu_{l}\\}+\sum^{l}_{j=2}\min\\{\lambda_{j},\nu_{j-1}\\}+\sum^{l}_{j=2}\max\\{\lambda_{j+1},\nu_{j}\\}+\lambda_{1}+\max\\{\lambda_{2},\nu_{1}\\}$

and the notations

$\displaystyle
0\leq\alpha_{l+1}\leq\min\\{\lambda_{l+1},\nu_{l}\\},\qquad\max\\{\lambda_{j+1},\nu_{j}\\}\leq\alpha_{j}\leq\min\\{\lambda_{j},\nu_{j-1}\\},$
$\displaystyle\max\\{\lambda_{2},\nu_{1}\\}\leq\alpha_{1}\leq\lambda_{1}.$

Thus the proof is completed. ∎

######  Lemma 4.5.

For generalized partitions $\nu=(\nu_{1},\dots,\nu_{l})$ and
$\lambda=(\lambda_{1},\dots,\lambda_{l+1})$, one has

$\displaystyle{\rm
o}_{\lambda/\nu}\big{(}t^{\pm}\big{)}=\sum_{\nu\prec\alpha\prec\lambda}(1+\delta_{\lambda_{l+1}>0}\delta_{\nu_{l},0})t^{2|\alpha|-|\lambda|-|\nu|}$

summed over all partitions $\alpha$ such that $\nu\prec\alpha\prec\lambda$ and
$\alpha_{l+1}\in\\{0,\min\\{\lambda_{l+1},\nu_{l}\\}\\}$.

###### Proof.

Using the same method for Lemma 4.4, we have

$\displaystyle\langle\nu^{\rm
o}|\Gamma_{+}\big{(}t^{\pm}\big{)}=\sum_{\lambda}\langle\lambda^{\rm
o}|\big{(}t^{-\min\\{\lambda_{l+1},\nu_{l}\\}}+t^{\min\\{\lambda_{l+1},\nu_{l}\\}}\big{)}$
$\displaystyle\quad{}\times\Bigg{(}\prod^{l}_{j=2}h_{\min\\{\lambda_{j},\nu_{j-1}\\}-\max\\{\lambda_{j+1},\nu_{j}\\}}\big{(}t^{\pm}\big{)}\Bigg{)}h_{\lambda_{1}-\max\\{\lambda_{2},\nu_{1}\\}}\big{(}t^{\pm}\big{)}$
$\displaystyle=\sum_{\lambda}\langle\lambda^{\rm
o}|\big{(}1+t^{2\min\\{\lambda_{l+1},\nu_{l}\\}}\big{)}\Bigg{(}\prod^{l}_{j=2}h_{\min\\{\lambda_{j},\nu_{j-1}\\}-\max\\{\lambda_{j+1},\nu_{j}\\}}\big{(}t^{\pm}\big{)}t^{\min\\{\lambda_{j},\nu_{j-1}\\}+\max\\{\lambda_{j+1},\nu_{j}\\}}\Bigg{)}$
$\displaystyle\quad\times
h_{\lambda_{1}-\max\\{\lambda_{2},\nu_{1}\\}}\big{(}t^{\pm}\big{)}t^{\lambda_{1}+\max\\{\lambda_{2},\nu_{1}\\}}t^{-|\lambda|-|\nu|}$
$\displaystyle=\sum_{\lambda}\langle\lambda^{\rm
o}|\sum_{\nu\prec\alpha\prec\lambda}\big{(}1+\delta_{\lambda_{l+1}>0}\delta_{\nu_{l},0}\big{)}t^{2|\alpha|-|\lambda|-|\nu|},$

summed over $\alpha$ such that $\nu\prec\alpha\prec\lambda$ and
$\alpha_{l+1}\in\\{0,\min\\{\lambda_{l+1},\nu_{l}\\}\\}$. ∎

The following theorem directly follows from Lemmas 4.4 and 4.5.

######  Theorem 4.6.

For generalized partitions
$\mu=(\mu_{1},\dots,\mu_{l})\subset\lambda=(\lambda_{1},\dots,\lambda_{l+N})$,
one has

$\displaystyle{\rm sp}_{\lambda/\mu}\big{(}x^{\pm}\big{)}=\sum_{\mu=z_{0}\prec
z_{1}\prec\dots\prec
z_{2N}=\lambda}\prod^{N}_{i=1}x^{2|z_{2i-1}|-|z_{2i}|-|z_{2i-2}|}_{i},$ (4.2)
$\displaystyle{\rm
o}_{\lambda/\mu}\big{(}x^{\pm}\big{)}=\sum_{\mu=z^{\prime}_{0}\prec
z^{\prime}_{1}\prec\dots\prec
z^{\prime}_{2N}=\lambda}\prod^{N}_{i=1}\big{(}1+\delta_{z^{\prime}_{2i,l+i}>0}\delta_{z^{\prime}_{2i-2,l+i-1},0}\big{)}x^{2|z^{\prime}_{2i-1}|-|z^{\prime}_{2i}|-|z^{\prime}_{2i-2}|}_{i},$
(4.3)

where generalized partitions
$z_{k}=\big{(}z_{k,1},\dots,z_{k,l+\lceil\frac{k}{2}\rceil}\big{)}$ satisfy
the symplectic Gelfand–Tsetlin pattern

$\displaystyle z_{k+1,j}\leq z_{k,j-1}\leq z_{k+1,j-1}\qquad\text{for}\quad
2\leq j\leq l+\biggl{\lceil}\frac{k+1}{2}\biggr{\rceil}$

and generalized partitions
$z^{\prime}_{k}=\big{(}z^{\prime}_{k,1},\dots,z^{\prime}_{k,l+\lceil\frac{k}{2}\rceil}\big{)}$
satisfy the orthogonal Gelfand–Tsetlin pattern

$\displaystyle z^{\prime}_{k+1,j}\leq z^{\prime}_{k,j-1}\leq
z^{\prime}_{k+1,j-1}\qquad\text{for}\quad 2\leq j\leq
l+\biggl{\lceil}\frac{k+1}{2}\biggr{\rceil}$

subject to
$z^{\prime}_{2i-1,l+i}\in\big{\\{}0,\min\big{\\{}z^{\prime}_{2i,l+i},z^{\prime}_{2i-2,l+i-1}\big{\\}}\big{\\}}$.

######  Remark 4.7.

One can find an equivalent statement of (4.2) in [2], which defined the
symplectic Gelfand–Tsetlin pattern by induction. For $\mu=\varnothing$, (4.2)
recovers the classical Gelfand–Tsetlin patterns for symplectic/orthogonal
Schur functions [22, p. 310], and (4.3) reduces to the multi-orthogonal
Gelfand–Tsetlin pattern [22, p. 324].

##  5 Cauchy-type identities and general branching rules

In this section, we give a new proof of Cauchy identities for
(symplectic/orthogonal) Schur functions by the method of vertex operators.
This approach is logically independent of the Jacobi–Trudi formula. The method
can be further extended to skew Schur functions as well as skew
symplectic/orthogonal Schur functions.

######  Lemma 5.1.

For $x=(x_{1},x_{2},\dots,x_{N})$, we have

$\displaystyle\Gamma_{-}(\\{x\\})|0\rangle=\sum_{\lambda}{\rm
s}_{\lambda}(x)|\lambda\rangle,$ (5.1)
$\displaystyle\Gamma_{-}(\\{x\\})|0\rangle=\prod_{1\leq i<j\leq
N}(1-x_{i}x_{j})^{-1}\sum_{\lambda}{\rm s}_{\lambda}(x)|\lambda^{\rm
sp}\rangle,$ $\displaystyle\Gamma_{-}(\\{x\\})|0\rangle=\prod_{1\leq i\leq
j\leq N}(1-x_{i}x_{j})^{-1}\sum_{\lambda}{\rm s}_{\lambda}(x)|\lambda^{\rm
o}\rangle,$

where the sum is over all partitions $\lambda$ with $l(\lambda)\leq N$.

###### Proof.

We first prove (5.1) by a similar method as in Proposition 3.1, which is
different from that of [11]. Using the relations

$X(z)\Gamma_{+}\big{(}z^{-1}\big{)}=\Gamma_{-}(z)\qquad\text{and}\qquad\Gamma_{+}\big{(}z^{-1}_{1}\big{)}\Gamma_{-}(z_{2})=\left(1-\frac{z_{2}}{z_{1}}\right)^{-1}\Gamma_{-}(z_{2})\Gamma_{+}\big{(}z^{-1}_{1}\big{)},$

one has

$\displaystyle\Gamma_{-}(\\{x\\})|0\rangle$ $\displaystyle{}=\prod_{1\leq
i<j\leq N}\bigg{(}1-\frac{x_{j}}{x_{i}}\bigg{)}^{-1}X(x_{1})\cdots
X(x_{N})|0\rangle$ $\displaystyle{}=\prod_{1\leq i<j\leq
N}\bigg{(}1-\frac{x_{j}}{x_{i}}\bigg{)}^{-1}\sum_{n_{1},\dots,n_{N}}x^{n_{1}}_{1}\cdots
x^{n_{N}}_{N}X_{-n_{1}}\cdots X_{-n_{N}}|0\rangle.$ (5.2)

Using $X_{i}X_{j}=-X_{j+1}X_{i-1}$, the factors can be shuffled into the
following expression:

$\displaystyle\varepsilon(\sigma)X_{-\lambda_{1}}\cdots
X_{-\lambda_{N}}=X_{-\lambda_{\sigma(1)}+\sigma(1)-1}\cdots
X_{-\lambda_{\sigma(i)}+\sigma(i)-i}\cdots
X_{-\lambda_{\sigma(N)}+\sigma(N)-N},$ (5.3)

where $\sigma\in S_{N}$. Thus the coefficient of
$|\lambda\rangle=X_{-\lambda_{1}}\cdots X_{-\lambda_{N}}|0\rangle$ in
$\Gamma_{-}(\\{x\\})|0\rangle$ (5.2) is

$\displaystyle\prod_{1\leq i<j\leq
N}\bigg{(}1-\frac{x_{j}}{x_{i}}\bigg{)}^{-1}\sum_{\sigma}\varepsilon(\sigma)x^{\lambda_{\sigma(1)}-\sigma(1)+1}_{1}\cdots
x^{\lambda_{\sigma(i)}-\sigma(i)+i}_{i}\cdots
x^{\lambda_{\sigma(N)}-\sigma(N)+N}_{N}$
$\displaystyle\qquad\quad{}=\prod_{1\leq i<j\leq
N}(x_{i}-x_{j})^{-1}\sum_{\sigma}\varepsilon(\sigma)x^{\lambda_{\sigma(1)}-\sigma(1)+N}_{1}\cdots
x^{\lambda_{\sigma(i)}-\sigma(i)+N}_{i}\cdots
x^{\lambda_{\sigma(N)}-\sigma(N)+N}_{N}$
$\displaystyle\qquad\quad{}=\frac{\det\big{(}x^{\lambda_{j}+N-j}_{i}\big{)}_{1\leq
i,j\leq N}}{\prod_{1\leq i<j\leq
N}(x_{i}-x_{j})}=\frac{\det\big{(}x^{\lambda_{j}+N-j}_{i}\big{)}_{1\leq
i,j\leq N}}{\det\big{(}x^{N-j}_{i}\big{)}_{1\leq i,j\leq N}}={\rm
s}_{\lambda}(x).$

Then the relation (2.10) gives
$\langle\lambda|\Gamma_{-}(\\{x\\})|0\rangle={\rm s}_{\lambda}(x)$ for any
$\lambda$, subsequently

$\displaystyle\Gamma_{-}(\\{x\\})|0\rangle=\sum_{\begin{subarray}{c}\lambda\\\
l(\lambda)\leq N\end{subarray}}{\rm s}_{\lambda}(x)|\lambda\rangle.$

The other two relations can be proved similarly with the help of

$\displaystyle\varepsilon(\sigma)Y_{-\lambda_{1}}\cdots
Y_{-\lambda_{N}}=Y_{-\lambda_{\sigma(1)}+\sigma(1)-1}\cdots
Y_{-\lambda_{\sigma(i)}+\sigma(i)-i}\cdots
Y_{-\lambda_{\sigma(N)}+\sigma(N)-N},$
$\displaystyle\varepsilon(\sigma)W_{-\lambda_{1}}\cdots
W_{-\lambda_{N}}=W_{-\lambda_{\sigma(1)}+\sigma(1)-1}\cdots
W_{-\lambda_{\sigma(i)}+\sigma(i)-i}\cdots
W_{-\lambda_{\sigma(N)}+\sigma(N)-N},$ (5.4)

where $\sigma\in S_{N}$. ∎

By direct calculation, we have the following two lemmas.

######  Lemma 5.2.

From the commutation relations (2.1) and (3.4), we have

$\displaystyle\langle
0|\Gamma_{+}(\\{x\\})\Gamma_{-}(\\{y\\})|0\rangle=\prod^{N}_{i=1}\prod^{K}_{j=1}\big{(}1-x_{i}y_{j}\big{)}^{-1},$

where $x=(x_{1},\dots,x_{N}),~{}y=(y_{1},\dots,y_{K})$.

Now we study some cases of the inner product $\langle
0|\Gamma_{+}(\\{x\\})\Gamma_{-}(\\{y\\})|0\rangle$ related to symmetric
functions. Due to (2.10)–(2.12), (3.7), (3.2), (3.21) and Lemma 5.1, we have
the following result.

######  Lemma 5.3.

For $x=(x_{1},\dots,x_{N})$, $y=(y_{1},\dots,y_{N})$,
$x^{\pm}=\big{(}x^{\pm}_{1},\dots,x^{\pm}_{N}\big{)}$, we have

$\displaystyle\langle
0|\Gamma_{+}(\\{x\\})\Gamma_{-}(\\{y\\})|0\rangle=\sum_{\begin{subarray}{c}\lambda\\\
l(\lambda)\leq N\end{subarray}}{\rm s}_{\lambda}(x){\rm s}_{\lambda}(y),$
$\displaystyle\langle
0|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}\Gamma_{-}(\\{y\\})|0\rangle=\prod_{1\leq
k<l\leq N}(1-y_{k}y_{l})^{-1}\sum_{\begin{subarray}{c}\lambda\\\
l(\lambda)\leq N\end{subarray}}{\rm sp}_{\lambda}\big{(}x^{\pm}\big{)}{\rm
s}_{\lambda}(y),$ $\displaystyle\langle
0|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}\Gamma_{-}(\\{y\\})|0\rangle=\prod_{1\leq
k\leq l\leq N}(1-y_{k}y_{l})^{-1}\sum_{\begin{subarray}{c}\lambda\\\
l(\lambda)\leq N\end{subarray}}{\rm o}_{\lambda}\big{(}x^{\pm}\big{)}{\rm
s}_{\lambda}(y).$

Combining Lemmas 5.2 and 5.3, we can prove the classical Cauchy identities
associated to Schur functions, symplectic and orthogonal Schur functions
respectively.

######  Theorem 5.4 ([4, 14, 25]).

Let $x=(x_{1},\dots,x_{N})$, $y=(y_{1},\dots,y_{N})$,
$x^{\pm}=\big{(}x^{\pm}_{1},\dots,x^{\pm}_{N}\big{)}$. Then

$\displaystyle\sum_{\begin{subarray}{c}\lambda\\\ l(\lambda)\leq
N\end{subarray}}{\rm s}_{\lambda}(x){\rm
s}_{\lambda}(y)=\prod^{N}_{i,j=1}(1-x_{i}y_{j})^{-1},$
$\displaystyle\sum_{\begin{subarray}{c}\lambda\\\ l(\lambda)\leq
N\end{subarray}}{\rm sp}_{\lambda}\big{(}x^{\pm}\big{)}{\rm
s}_{\lambda}(y)=\prod^{N}_{i,j=1}\bigl{[}(1-x_{i}y_{j})\big{(}1-x^{-1}_{i}y_{j}\big{)}\bigr{]}^{-1}\prod_{1\leq
k<l\leq N}(1-y_{k}y_{l}),$ $\displaystyle\sum_{\begin{subarray}{c}\lambda\\\
l(\lambda)\leq N\end{subarray}}{\rm o}_{\lambda}\big{(}x^{\pm}\big{)}{\rm
s}_{\lambda}(y)=\prod^{N}_{i,j=1}\big{[}(1-x_{i}y_{j})\big{(}1-x^{-1}_{i}y_{j}\big{)}\big{]}^{-1}\prod_{1\leq
k\leq l\leq N}(1-y_{k}y_{l}),$

where each summation is over partitions $\lambda$ with $l(\lambda)\leq N$.

The following is a generalization of Lemma 5.1.

######  Lemma 5.5.

For a generalized partition $\mu=(\mu_{1},\dots,\mu_{l})$ and
$x=(x_{1},x_{2},\dots,x_{N})$, we have

$\displaystyle\Gamma_{-}(\\{x\\})|\mu\rangle=\sum_{\begin{subarray}{c}\lambda\\\
l(\lambda)\leq l+N\end{subarray}}{\rm s}_{\lambda/\mu}(x)|\lambda\rangle,$
(5.5) $\displaystyle\Gamma_{-}(\\{x\\})|\mu^{\rm sp}\rangle=\prod_{1\leq
i<j\leq
N}(1-x_{i}x_{j})^{-1}\sum_{\lambda=(\lambda_{1},\dots,\lambda_{l+N})}{\rm
S}^{*}_{\lambda/\mu}(x)|\lambda^{\rm sp}\rangle,$ (5.6)
$\displaystyle\Gamma_{-}(\\{x\\})|\mu^{\rm o}\rangle=\prod_{1\leq i\leq j\leq
N}(1-x_{i}x_{j})^{-1}\sum_{\lambda=(\lambda_{1},\dots,\lambda_{l+N})}{\rm
S}^{*}_{\lambda/\mu}(x)|\lambda^{\rm o}\rangle,$

where the second and third sums are over generalized partitions
$\lambda=(\lambda_{1},\dots,\lambda_{l+N})$ with zeros parts allowed in the
end, and

$\displaystyle{\rm
S}^{*}_{\lambda/\mu}(x)=\frac{\det(c_{ij})^{l+N}_{i,j=1}}{\det\big{(}x^{N-j}_{i}\big{)}_{1\leq
i,j\leq N}}$

with

$c_{ij}=\begin{cases}x^{\lambda_{j}-j+N}_{i},&1\leq i\leq N,\\\
h_{\mu_{i-N}-\lambda_{j}+j-i}\big{(}x^{\pm}\big{)},&N+1\leq i\leq
N+l.\end{cases}$

###### Proof.

We only prove (5.6), as the others can be handled similarly. By definition
(2.2), we have

$\displaystyle\Gamma_{-}(\\{x\\})=\frac{1}{\prod_{1\leq i<j\leq
N}(1-x_{i}x_{j})\big{(}1-x^{-1}_{i}x_{j}\big{)}}Y(x_{1})\cdots
Y(x_{N})\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)},$

where $x^{\pm}=\big{(}x^{\pm}_{1},\dots,x^{\pm}_{N}\big{)}$. It is easy to
check that (by vertex operator calculus)

$\displaystyle\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}Y(z)=\frac{1}{\prod^{N}_{i=1}(1-x_{i}z)\big{(}1-x^{-1}_{i}z\big{)}}Y(z)\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)},$

thus

$\displaystyle\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}Y_{n}=\sum_{i\geq
0}h_{i}\big{(}x^{\pm}\big{)}Y_{n+i}\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}.$

Then

$\displaystyle\prod_{1\leq i<j\leq
N}(1-x_{i}x_{j})\big{(}1-x^{-1}_{i}x_{j}\big{)}\Gamma_{-}(\\{x\\})|\mu^{\rm
sp}\rangle$ $\displaystyle\qquad{}=Y(x_{1})\cdots
Y(x_{N})\sum_{k_{N+1},\dots,k_{N+l}\geq
0}h_{k_{N+1}}\big{(}x^{\pm}\big{)}\cdots
h_{k_{N+l}}\big{(}x^{\pm}\big{)}Y_{-\mu_{1}+k_{N+1}}\cdots
Y_{-\mu_{l}+k_{N+l}}|0\rangle$
$\displaystyle\qquad{}=\sum_{k_{1},\dots,k_{N}\in\mathbb{Z}}x^{k_{1}}_{1}\cdots
x^{k_{N}}_{N}\sum_{k_{N+1},\dots,k_{N+l}\geq
0}h_{k_{N+1}}\big{(}x^{\pm}\big{)}\cdots h_{k_{N+l}}\big{(}x^{\pm}\big{)}$
$\displaystyle\qquad\quad{}\times Y_{-k_{1}}\cdots
Y_{-k_{N}}Y_{-\mu_{1}+k_{N+1}}\cdots Y_{-\mu_{l}+k_{N+l}}|0\rangle.$

By (5.4), the coefficient of $|\lambda^{\rm sp}\rangle$ in $\prod_{1\leq
i<j\leq
N}(1-x_{i}x_{j})\big{(}1-x^{-1}_{i}x_{j}\big{)}\Gamma_{-}(\\{x\\})|\mu^{\rm
sp}\rangle$ is

$\displaystyle\det(c_{ij})^{l+N}_{i,j=1}$

with

$c_{ij}=\begin{cases}x^{\lambda_{j}-j+i}_{i},&1\leq i\leq N,\\\
h_{\mu_{i-N}-\lambda_{j}+j-i}\big{(}x^{\pm}\big{)},&N+1\leq i\leq
N+l.\end{cases}$

By means of the Vandermonde identity, we can get (5.6). ∎

######  Remark 5.6.

The symmetric rational function ${\rm S}^{*}_{\lambda/\mu}(x)$ makes sense
without the restriction $\mu\subset\lambda$, while ${\rm s}_{\lambda/\mu}(x)$
is zero unless $\mu\subset\lambda$. Unlike the skew symplectic Schur functions
${\rm sp}_{\lambda/\mu}\big{(}x^{\pm}\big{)}$, the symmetric polynomials ${\rm
S}^{*}_{\lambda/\mu}(x_{1},\dots,x_{N})$ do not change if we add zeroes to
$\lambda$ and $\mu$ simultaneously. More precisely,

${\rm
S}^{*}_{(\lambda_{1},\dots,\lambda_{M+N-1},\lambda_{M+N})/(\mu_{1},\dots,\mu_{M-1},0)}(x_{1},\dots,x_{N})=0,$

unless $\lambda_{M+N}=0$, and

${\rm
S}^{*}_{(\lambda_{1},\dots,\lambda_{M+N-1},0)/(\mu_{1},\dots,\mu_{M-1},0)}(x_{1},\dots,x_{N})={\rm
S}^{*}_{(\lambda_{1},\dots,\lambda_{M+N-1})/(\mu_{1},\dots,\mu_{M-1})}(x_{1},\dots,x_{N}).$

Moreover, ${\rm
S}^{*}_{(\lambda_{1},\dots,\lambda_{M+N})/(0^{M})}(x_{1},\dots,x_{N})=0$
unless $\lambda_{N+1}=\cdots=\lambda_{M+N}=0$, and

${\rm
S}^{*}_{(\lambda_{1},\dots,\lambda_{N},0^{M})/(0^{M})}(x_{1},\dots,x_{N})={\rm
S}^{*}_{(\lambda_{1},\dots,\lambda_{N})/\varnothing}(x_{1},\dots,x_{N})={\rm
s}_{(\lambda_{1},\dots,\lambda_{N})}(x_{1},\dots,x_{N}).$

We now offer a vertex algebraic proof of the Cauchy-type identity for skew
Schur functions [19, p. 93], which can be seen as a generalized Cauchy
identity for Schur functions.

######  Theorem 5.7.

For two partitions $\lambda$ and $\mu$,

$\sum_{\rho}{\rm s}_{\rho/\lambda}(x){\rm
s}_{\rho/\mu}(y)=\prod_{i,j}(1-x_{i}y_{j})^{-1}\sum_{\nu}{\rm
s}_{\mu/\tau}(x){\rm s}_{\lambda/\tau}(y),$

where the sum is over all partitions $\rho$ and $\tau$ such that the skew
Schur functions are defined.

###### Proof.

From (2.10), we can easily deduce that

$\displaystyle\langle\lambda|\sum_{\rho}|\rho\rangle\langle\rho|=\langle\lambda|,\qquad\sum_{\rho}|\rho\rangle\langle\rho||\lambda\rangle=|\lambda\rangle,$

where the sum is over all partitions $\rho$. Thus

$\displaystyle\langle\lambda|\Gamma_{+}(\\{x\\})\Gamma_{-}(\\{y\\})|\mu\rangle=\langle\lambda|\Gamma_{+}(\\{x\\})\sum_{\rho}|\rho\rangle\langle\rho|\Gamma_{-}(\\{y\\})|\mu\rangle=\sum_{\rho}{\rm
s}_{\rho/\lambda}(x){\rm s}_{\rho/\mu}(y),$ (5.7)

where we have used (3.6). We also have the following relation:

$\displaystyle\langle\lambda|\Gamma_{+}(\\{x\\})\Gamma_{-}(\\{y\\})|\mu\rangle$
$\displaystyle{}=\prod_{i,j}(1-x_{i}y_{j})^{-1}\langle\lambda|\Gamma_{-}(\\{y\\})\Gamma_{+}(\\{x\\})|\mu\rangle$
$\displaystyle{}=\prod_{i,j}(1-x_{i}y_{j})^{-1}\langle\lambda|\Gamma_{-}(\\{y\\})\sum_{\tau}|\tau\rangle\langle\tau|\Gamma_{+}(\\{x\\})|\mu\rangle$
$\displaystyle{}=\prod_{i,j}(1-x_{i}y_{j})^{-1}\sum_{\nu}{\rm
s}_{\mu/\tau}(x){\rm s}_{\lambda/\tau}(y).$ (5.8)

Comparing (5.7) and (5.8), we get the identity. ∎

We also extend the vertex algebraic approach to derive the Cauchy-type
identities for skew symplectic/orthogonal Schur functions.

######  Theorem 5.8.

For generalized partition $\mu=(\mu_{1},\dots,\mu_{l})$ and
$\lambda=(\lambda_{1},\dots,\lambda_{l+N})$, one has

$\displaystyle\sum_{\rho=(\rho_{1},\dots,\rho_{l+N})}{\rm
sp}_{\rho/\mu}\big{(}x^{\pm}\big{)}{\rm S}^{*}_{\rho/\lambda}(y){}$
$\displaystyle\qquad\quad{}=\prod^{N}_{i=1}\prod^{K}_{j=1}\frac{1}{(1-x_{i}y_{j})(1-x^{-1}_{i}y_{j})}\sum_{\tau=(\tau_{1},\dots,\tau_{l})}{\rm
sp}_{\lambda/\tau}\big{(}x^{\pm}\big{)}{\rm S}^{*}_{\mu/\tau}(y),$ (5.9)
$\displaystyle\sum_{\rho=(\rho_{1},\dots,\rho_{l+N})}{\rm
o}_{\rho/\mu}\big{(}x^{\pm}\big{)}{\rm S}^{*}_{\rho/\lambda}(y)$
$\displaystyle\qquad\quad{}=\prod^{N}_{i=1}\prod^{K}_{j=1}\frac{1}{(1-x_{i}y_{j})(1-x^{-1}_{i}y_{j})}\sum_{\tau=(\tau_{1},\dots,\tau_{l})}{\rm
o}_{\lambda/\tau}\big{(}x^{\pm}\big{)}{\rm S}^{*}_{\mu/\tau}(y),$ (5.10)

where $x^{\pm}=\big{(}x^{\pm}_{1},\dots,x^{\pm}_{N}\big{)}$,
$y=(y_{1},\dots,y_{K})$.

###### Proof.

For generalized partition $\nu=(\nu_{1},\dots,\nu_{n})$, recall the relations
(3.16) and (3.24)

$\displaystyle\langle\nu^{\rm
sp}|\sum_{\eta=(\eta_{1},\dots,\eta_{n})}|\eta^{\rm sp}\rangle\langle\eta^{\rm
sp}|=\langle\nu^{\rm sp}|,\qquad$
$\displaystyle\sum_{\eta=(\eta_{1},\dots,\eta_{n})}|\eta^{\rm
sp}\rangle\langle\eta^{\rm sp}||\nu^{\rm sp}\rangle=|\nu^{\rm sp}\rangle,$
$\displaystyle\langle\nu^{\rm
o}|\sum_{\eta=(\eta_{1},\dots,\eta_{n})}|\eta^{\rm o}\rangle\langle\eta^{\rm
o}|=\langle\nu^{\rm o}|,\qquad$
$\displaystyle\sum_{\eta=(\eta_{1},\dots,\eta_{n})}|\eta^{\rm
o}\rangle\langle\eta^{\rm o}||\nu^{\rm o}\rangle=|\nu^{\rm o}\rangle,$

where the sum is over all generalized partitions
$\eta=(\eta_{1},\dots,\eta_{n})$. On the one hand,

$\displaystyle\langle\mu^{\rm
sp}|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}\Gamma_{-}(\\{y\\})|\lambda^{\rm
sp}\rangle=~{}$ $\displaystyle\langle\mu^{\rm
sp}|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}\sum_{\rho=(\rho_{1},\dots,\rho_{l+N})}|\rho^{\rm
sp}\rangle\langle\rho^{\rm sp}|\Gamma_{-}(\\{y\\})|\lambda^{\rm sp}\rangle$
$\displaystyle=~{}$ $\displaystyle\sum_{\rho=(\rho_{1},\dots,\rho_{l+N})}{\rm
sp}_{\rho/\mu}\big{(}x^{\pm}\big{)}{\rm S}^{*}_{\rho/\lambda}(y),$ (5.11)

where we used (3.10) and (5.6). On the other hand,

$\displaystyle\langle\mu^{\rm
sp}|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}\Gamma_{-}(\\{y\\})|\lambda^{\rm
sp}\rangle$
$\displaystyle\qquad\quad{}=\prod^{N}_{i=1}\prod^{K}_{j=1}\frac{1}{(1-x_{i}y_{j})\big{(}1-x^{-1}_{i}y_{j}\big{)}}\langle\mu^{\rm
sp}|\Gamma_{-}(\\{y\\})\sum_{\tau=(\tau_{1},\dots,\tau_{l})}|\tau^{\rm
sp}\rangle\langle\tau^{\rm
sp}|\Gamma_{+}\big{(}\big{\\{}x^{\pm}\big{\\}}\big{)}|\lambda^{\rm sp}\rangle$
$\displaystyle\qquad\quad{}=\prod^{N}_{i=1}\prod^{K}_{j=1}\frac{1}{(1-x_{i}y_{j})\big{(}1-x^{-1}_{i}y_{j}\big{)}}\sum_{\tau=(\tau_{1},\dots,\tau_{l})}{\rm
sp}_{\lambda/\tau}\big{(}x^{\pm}\big{)}{\rm S}^{*}_{\mu/\tau}(y).$ (5.12)

Comparing (5) with (5.12), we can get (5.9). The relation (5.10) can be proved
similarly. ∎

######  Remark 5.9.

The generalized partition $\tau$ in the right-hand side of (5.12) is contained
in the generalized partition $\lambda$. For $\mu=\big{(}0^{N}\big{)}$,
$\lambda=\big{(}0^{N}\big{)}$, relation (5.12) recovers the classical Cauchy
identity for Schur functions

$\displaystyle\sum_{\rho=(\rho_{1},\dots,\rho_{N})}{\rm
s}_{\rho}\big{(}x_{1},x^{-1}_{1},\dots,x_{N},x^{-1}_{N}\big{)}{\rm
s}_{\rho}(y_{1},\dots,y_{N})=\frac{1}{\prod^{N}_{i=1}\prod^{N}_{j=1}(1-x_{i}y_{j})\big{(}1-x^{-1}_{i}y_{j}\big{)}}$

due to Remark 3.7 and the fact that the generalized partition $\tau$ have to
be $\big{(}0^{N}\big{)}$.

######  Remark 5.10.

In [16], Lam studied some general symmetric functions via a boson-fermion
correspondence. It will be interesting to study and realize the skew symmetric
functions using vertex operators.

Added in proof. In the current paper posted on arXiv in 2022, among other
things we have derived the Jacobi–Trudi formulas for the skew Schur symplectic
functions and the skew Schur orthogonal functions in terms of the
(generalized) homogeneous symmetric functions. We recently noticed that [1]
has found a combinatorial proof of the alternative Jacobi–Trudi formulas in
terms of elementary symmetric functions in 2023. We will also give a vertex
algebraic proof of the latter in a forthcoming paper.

##  Appendix A Another proof of Proposition 3.1

In this appendix, we give a simple vertex algebraic proof of Proposition 3.1
and (5.5).

###### Proof.

From the definition of $X(z)$, we have

$\displaystyle\Gamma_{+}(\\{x\\})X(z)=\prod^{N}_{i=1}\frac{1}{1-x_{i}z}X(z)\Gamma_{+}(\\{x\\}),$

i.e.,

$\displaystyle\Gamma_{+}(\\{x\\})X_{n}=\sum_{i\geq
0}h_{i}(x)X_{n+i}\Gamma_{+}(\\{x\\}).$

Thus

$\displaystyle\Gamma_{+}(\\{x\\})|\lambda\rangle=\Gamma_{+}(\\{x\\})X_{-\lambda_{1}}\dots
X_{-\lambda_{k}}|0\rangle$ $\displaystyle=\sum_{i_{1},\dots,i_{k}\geq
0}h_{i_{1}}(x)\cdots h_{i_{k}}(x)X_{-\lambda_{1}+i_{1}}\dots
X_{-\lambda_{k}+i_{k}}|0\rangle$ $\displaystyle=\sum_{\nu}\\!\sum_{\sigma\in
S_{k}}\\!\\!\epsilon(\sigma)h_{\lambda_{1}-\nu_{\sigma(1)}+\sigma(1)-1}(x)\\!\cdots
h_{\lambda_{k}-\nu_{\sigma(k)}+\sigma(k)-k}(x)X_{-\nu_{\sigma(1)}+\sigma(1)-1}\\!\cdots
X_{-\nu_{\sigma(k)}+\sigma(k)-k}|0\rangle$
$\displaystyle=\sum_{\nu}\det(h_{\lambda_{i}-\nu_{j}-i+j}(x))^{k}_{i,j=1}|\nu\rangle=\sum_{\nu}{\rm
s}_{\lambda/\nu}(x)|\nu\rangle,$

where we have used (5.3) and the fact
$\Gamma_{+}(\\{x\\})|0\rangle=|0\rangle$. By (2.10), we have shown Proposition
3.1. Similarly, using (3.5) and (2.10), we can get

$\displaystyle\langle\lambda|\Gamma_{-}(\\{x\\})|\mu\rangle={\rm
s}_{\lambda/\mu}(x).$

It is clear that

$\displaystyle\Gamma_{-}(\\{x\\})|\mu\rangle=\sum_{\mu\subset\lambda}{\rm
s}_{\lambda/\mu}(x)|\lambda\rangle.$ ∎

### Acknowledgements

We extend our heartfelt appreciation to the anonymous referees for their
invaluable constructive feedback and suggestions, which have enhanced the
quality of the paper. The research is supported by the Simons Foundation
(grant no. MP-TSM-00002518), NSFC (grant nos. 12171303, 12101231, 12301033),
and NSF of Huzhou (grant no. 2022YZ47).

## References

  * [1] Albion S.P., Fischer I., Höngesberg H., Schreier-Aigner F., Skew symplectic and orthogonal characters through lattice paths, arXiv:2305.11730.
  * [2] Ayyer A., Fischer I., Bijective proofs of skew Schur polynomial factorizations, J. Combin. Theory Ser. A 174 (2020), 105241, 40 pages, arXiv:1905.05226.
  * [3] Baker T.H., Vertex operator realization of symplectic and orthogonal $S$-functions, J. Phys. A 29 (1996), 3099–3117.
  * [4] Betea D., Correlations for symplectic and orthogonal Schur measures, arXiv:1804.08495.
  * [5] Cuenca C., Gorin V., $q$-deformed character theory for infinite-dimensional symplectic and orthogonal groups, Selecta Math. (N.S.) 26 (2020), 40, 55 pages, arXiv:1812.06523.
  * [6] Date E., Kashiwara M., Jimbo M., Miwa T., Transformation groups for soliton equations, in Nonlinear Integrable Systems – Classical Theory and Quantum Theory (Kyoto, 1981), World Scientific Publishing, Singapore, 1983, 39–119.
  * [7] Fauser B., Jarvis P.D., King R.C., Plethysms, replicated Schur functions and series, with applications to vertex operators, J. Phys. A 43 (2010), 405202, 30 pages, arXiv:1006.4266.
  * [8] Frenkel I.B., Kac V.G., Basic representations of affine Lie algebras and dual resonance models, Invent. Math. 62 (1980), 23–66.
  * [9] Jing N., Vertex operators, symmetric functions, and the spin group $\Gamma_{n}$, J. Algebra 138 (1991), 340–398.
  * [10] Jing N., Symmetric polynomials and $U_{q}(\widehat{\rm sl}{}_{2})$, Represent. Theory 4 (2000), 46–63, arXiv:math.QA/9902109.
  * [11] Jing N., Li Z., A note on Cauchy’s formula, Adv. in Appl. Math. 153 (2024), 102630, 14 pages, arXiv:2202.11175.
  * [12] Jing N., Nie B., Vertex operators, Weyl determinant formulae and Littlewood duality, Ann. Comb. 19 (2015), 427–442, arXiv:1308.2792.
  * [13] Jing N., Rozhkovskaya N., Vertex operators arising from Jacobi–Trudi identities, Comm. Math. Phys. 346 (2016), 679–701, arXiv:1411.4725.
  * [14] Koike K., Terada I., Young-diagrammatic methods for the representation theory of the classical groups of type $B_{n}$, $C_{n}$, $D_{n}$, J. Algebra 107 (1987), 466–511.
  * [15] Koike K., Terada I., Young diagrammatic methods for the restriction of representations of complex classical Lie groups to reductive subgroups of maximal rank, Adv. Math. 79 (1990), 104–135.
  * [16] Lam T., A combinatorial generalization of the Boson-Fermion correspondence, Math. Res. Lett. 13 (2006), 377–392, arXiv:math.CO/0507341.
  * [17] Li H., Tan S., Wang Q., A certain Clifford-like algebra and quantum vertex algebras, Israel J. Math. 216 (2016), 441–470, arXiv:1505.07165.
  * [18] Littlewood D.E., The theory of group characters and matrix representations of groups, Oxford University Press, New York, 1940.
  * [19] Macdonald I.G., Symmetric functions and Hall polynomials, 2nd ed., Oxford Math. Monogr., The Clarendon Press, Oxford University Press, New York, 1995.
  * [20] Meckes E.S., The random matrix theory of the classical compact groups, Cambridge Tracts in Math., Vol. 218, Cambridge University Press, Cambridge, 2019.
  * [21] Okounkov A., Infinite wedge and random partitions, Selecta Math. (N.S.) 7 (2001), 57–81, arXiv:math.RT/9907127.
  * [22] Proctor R.A., Young tableaux, Gel’fand patterns, and branching rules for classical groups, J. Algebra 164 (1994), 299–360.
  * [23] Shimozono M., Zabrocki M., Deformed universal characters for classical and affine algebras, J. Algebra 299 (2006), 33–61, arXiv:math.CO/0404288.
  * [24] Stanley R.P., Enumerative combinatorics. Vol. 2, Cambridge Stud. Adv. Math., Vol. 62, Cambridge University Press, Cambridge, 1999.
  * [25] Sundaram S., Tableaux in the representation theory of the classical Lie groups, in Invariant Theory and Tableaux (Minneapolis, MN, 1988), IMA Vol. Math. Appl., Vol. 19, Springer, New York, 1990, 191–225.
  * [26] Weyl H., The classical groups. Their invariants and representations, Princeton Math. Ser., Princeton University Press, Princeton, NJ, 1946.