# Automorphisms of Hilbert schemes of Cayley’s K3 surfaces

Kwangwoo Lee  Department of Mathematics, Ewha Womans University, 52
Ewhayeodae-gil, Seodaemun-gu, Seoul, Republic of Korea<EMAIL_ADDRESS>
###### Abstract.

We prove that the automorphism group of Hilbert square of a Cayley’s K3
surface of Picard number $2$ is the free product of three cyclic groups of
order two. The generators are three Beauville involutions.

###### Key words and phrases:

K3 surface, Hilbert scheme, Automorphism, Beauville involution, Fuchsian group

###### 1991 Mathematics Subject Classification:

Primary: 14C05, 14J28, 14J50; Secondary: 20H10

The author was supported by National Research Foundation of
Korea(NRF-2020R1I1A1A01073662) and National Research Foundation of
Korea(NRF-2021R1A4A3033098) grant funded by the Korea government(MSIT)

##  1\. Introduction

Let $X$ be a nonsingular projective surface over ${\mathbb{C}}$, and let
$X^{[m]}$ be the Hilbert scheme of $m$ points on $X$, the space parametrizing
$0$-dimensional subschemes $Z\subset X$ of length $m$. By the result of
Beauville, the Hilbert scheme of points of a K3 surface is a
hyperk$\ddot{\text{a}}$hler manifold ([3]). A hyperk$\ddot{\text{a}}$hler
manifold is a simply connected compact K$\ddot{\text{a}}$hler manifold $Y$
such that $H^{0}(Y,\Omega_{Y}^{2})=\mathbb{C}\sigma_{Y}$, where $\sigma_{Y}$
is an everywhere non-degenerate holomorphic $2$-form on $Y$. In particular, a
K3 surface is a $2$-dimensional hyperk$\ddot{\text{a}}$hler manifold. For a K3
surface $X$, the Hilbert scheme $X^{[m]}$ of length $m$ of $X$ is a
hyperk$\ddot{\text{a}}$hler manifold of dimension $2m$ with Beauville-
Bogomolov-Fujiki’s form ([3]) on $H^{2}(X^{[m]},\mathbb{Z})$ and we have a
natural Hodge isometry

(1.1) $H^{2}(X^{[m]},\mathbb{Z})\cong H^{2}(X,\mathbb{Z})\oplus\mathbb{Z}[e],$

where $e:=E/2$ with self-intersection number $(e^{2})=-2m+2$ and $E$ is the
exceptional divisor of the Hilbert-Chow morphism

(1.2) $\epsilon:X^{[m]}\rightarrow X^{(m)}=X^{m}/S_{m}.$

Here $X^{(m)}$ is the $m$-th symmetric product of $X$ defined by the quotient
of Cartesian product $X^{m}$ by the symmetric group $S_{m}$. Moreover, we have
the natural injections of automorphisms

(1.3) $\operatorname{Aut}(X)\hookrightarrow\operatorname{Aut}(X^{[m]})$

and

(1.4) $\operatorname{Aut}(X^{[m]})\hookrightarrow
O(H^{2}(X^{[m]},\mathbb{Z}),q_{X^{[m]}}),\hskip 14.22636ptf\mapsto f^{*},$

where $O(H^{2}(X^{[m]},\mathbb{Z}),q_{X^{[m]}})$ is the isometry group of
$H^{2}(X^{[m]},\mathbb{Z})$ with the quadratic form $q_{X^{[m]}}$.

In [18], Oguiso considered a K3 surface $S$ with Picard lattice
$NS(S)={\mathbb{Z}}h_{1}\oplus{\mathbb{Z}}h_{2}$ where the intersection matrix

(1.5) $[(h_{i},h_{j})]=\begin{bmatrix}4&2\\\ 2&-4\end{bmatrix}.$

He constructed an isometry of $NS(S)$ given by the multiplication of
$\eta^{6}$ on $NS(S)$, where $h_{2}=\eta:=\frac{1+\sqrt{5}}{2}$ is a basis of
the lattice $NS(S)$ together with $h_{1}:=1$. Then by the global Torelli
theorem for K3 surfaces, it induces an automorphism $g$ of $S$ with positive
entropy without fixed points. It is also known as the Cayley-Oguiso
automorphism in [10]. This was introduced by Cayley. In [9], he constructed a
K3 surface with an automorphism of infinite order. Let $X$ be a Cayley’s K3
surface: the determinantal quartic surface in ${\mathbb{P}}^{3}$ with
homogeneous coordinates $\textbf{x}=[x_{0}:x_{1}:x_{2}:x_{3}]$ defined by

(1.6) $X:=(\det M(\textbf{x})=0)\subset{\mathbb{P}}^{3}_{\textbf{x}},$

where $M(\textbf{x})$ is the generic $4\times 4$ matrix whose $(k,j)$ entry is
$\Sigma_{i}a_{ijk}x_{i}$. By a genericity assumption of $M$,
$\operatorname{rank}M(\textbf{x})=3$ for all $\textbf{x}\in X$ and $X$ is a
smooth quartic K3 surface. In particular, let $X$ be a Cayley’s K3 surface
with the rank of $NS(X)=2$. Then by [10, Proposition 2.2], $X\cong S$.

For a very ample divisor $D_{n}:=\eta^{2n}$ with $n\in{\mathbb{Z}}$ on $X$, we
have an isomorphism
$\phi_{D_{n}}:X\xrightarrow{\cong}X_{n}\subset{\mathbb{P}}^{3}$ (cf. [15,
§5.1]). Then $\phi_{D_{n}}$ defines a Beauville birational involution
$\iota_{n}$ on the Hilbert scheme $X^{[2]}$ of $X$. Since $X$ has no line,
$\iota_{n}$ is a biregular involution of $X^{[2]}$ by [2]. In [15, Theorem 2],
the author showed that all Beauville involutions are generated by
$\iota_{0},\iota_{1},\iota_{2}$ and the natural automorphism $g$ induced from
$X$, i.e., an image of (1.3).

In this paper, we will show the follow theorem:

######  Theorem 1.1.

$\operatorname{Aut}(X^{[2]})\cong\langle\iota_{0},\iota_{1},\iota_{2}\rangle\cong{\mathbb{Z}}_{2}\ast{\mathbb{Z}}_{2}\ast{\mathbb{Z}}_{2}$.

The first equivalence in Theorem 1.1 is also known in [19] without proof. Our
strategies are the followings. For the first equivalence, we use the
characterization of natural automorphisms. Next we use hyperbolic geometry to
prove the second equivalence.

######  Remark 1.2.

For Hilbert squares of a K3 surface with Picard number one, the automorphism
group is either trivial or ${\mathbb{Z}}_{2}$ generated by a non-symplectic
involution [5, 8].

##  2\. Preliminaries

In this section we recall some useful results on Hilbert squares of projective
K3 surfaces and on hyperbolic geometry which we will use in the following
sections.

###  2.1. Hilbert squares of K3 surfaces

For a K3 surface $X$, the Hilbert scheme $X^{[m]}$ of length $m$ of $X$ is a
hyperk$\ddot{\text{a}}$hler manifold of dimension $2m$ by Beauville [3]. There
exists an injective morphism $i:H^{2}(X,{\mathbb{C}})\rightarrow
H^{2}(X^{[m]},{\mathbb{C}})$ such that

(2.1)
$H^{2}(X^{[m]},{\mathbb{C}})=i(H^{2}(X,{\mathbb{C}}))\oplus{\mathbb{C}}[E]$

with a quadratic form $q_{X^{[m]}}$ on $X^{[m]}$ satisfying
$q_{X^{[m]}}(i(\alpha))=q(\alpha)$ for $\alpha\in H^{2}(X,{\mathbb{C}})$,
where $q$ is the quadratic form on $X$. Note that $E$ is the exceptional
divisor of (1.2). For $e:=E/2$,
$H^{2}(X^{[m]},\mathbb{Z})=i(H^{2}(X,{\mathbb{Z}}))\oplus{\mathbb{Z}}[e]$.

For any line bundle $L$ on $X$, the tensor product
$\otimes_{i=1}^{m}p_{i}^{*}(L)$ has a natural $S_{m}$-invariant structure, and
taking $S_{m}$-invariants defines a line bundle $L(m)$ on $X^{(m)}$, where
$p_{i}:X^{m}\rightarrow X$ is the projection onto the $i$-th factor. We also
define the pullback via the Hilbert-Chow morphism (1.2) to the Hilbert scheme:

(2.2) $L[m]:=\epsilon^{*}L(m).$

By construction we have the first Chern class $c_{1}(L[m])=i(c_{1}(L))$.

For any $0$-dimensional subscheme $(Z,\mathcal{O}_{Z})$ of length
$h^{0}(\mathcal{O}_{Z})=k+1$, if the restriction map $H^{0}(L)\rightarrow
H^{0}(L\otimes\mathcal{O}_{Z})$ is surjective, then $L$ is called k-very ample
for an integer $k\geq 0$ ([6]). By definition, $L$ is $0$-very ample if and
only if $L$ is generated by its global sections, and $L$ is $1$-very ample if
and only if $L$ is very ample. For the Grassmannian
$\operatorname{Grass}(m,H^{0}(L))$ of all $m$-dimensional quotients of
$H^{0}(L)$, we have a rational map

(2.3) $\phi_{L}:X^{[m]}\dashrightarrow\operatorname{Grass}(m,H^{0}(L)),$

sending $(Z,{\mathcal{O}}_{Z})\in X^{[m]}$ to the quotient
$H^{0}(L)\rightarrow H^{0}(L\otimes{\mathcal{O}}_{Z})$. If $L$ is $(m-1)$-very
ample, then $\phi_{L}$ is a morphism and $\phi_{L}$ is embedding if and only
if $L$ is $m$-very ample (cf. [7]).

Let $X$ be a K3 surface with Picard lattice
$NS(X)={\mathbb{Z}}h_{1}\oplus{\mathbb{Z}}h_{2}$ whose intersection matrix is
given in (1.5). Since $X$ has no $(-2)$-curve, the ample cone of $X$ is just
the positive cone, i.e.,

(2.4) $\begin{array}[]{lcl}\operatorname{Amp}(X)&=&\\{(x,y)\in
NS(X)\otimes{\mathbb{R}}\mid(x,y)^{2}>0\text{ and }x>0\\}\\\ &=&\\{(x,y)\in
NS(X)\otimes{\mathbb{R}}\mid x>0\text{ and
}\frac{1-\sqrt{5}}{2}<\frac{y}{x}<\frac{1+\sqrt{5}}{2}\\}.\\\ \end{array}$

For an ample line bundle $L$ on $X$, $\otimes_{i=1}^{m}p_{i}^{*}(L)$ is ample
on the Cartesian product $X^{m}$, hence $L(m)$ is ample on the symmetric
product $X^{(m)}$. Since the Hilbert-Chow morphism is birational, the induced
line bundle $L[m]$ is big and nef on $X^{[m]}$. However, since $L[m]$ has
degree zero on the fibers of the Hilbert-Chow morphism, $L[m]$ is not ample.
Hence, the line bundle $L[m]$ lies on the boundary of the nef cone of
$X^{[m]}$.

######  Lemma 2.1.

For any ample line bundle $L$ on the K3 surface $X$ with Picard lattice
$NS(X)={\mathbb{Z}}h_{1}\oplus{\mathbb{Z}}h_{2}$, the line bundle $L[2]-e$ is
ample on $X^{[2]}$.

###### Proof.

Let $L$ be an ample line bundle on $X$. In [14, Theorem 1.1], Knutsen showed
that $L$ is $k$-very ample if and only if $L^{2}\geq 4k$ and there exists no
effective divisor $D$ satisfying the conditions $(*)$ below:

(*) $\begin{array}[]{lcl}&2D^{2}\overset{(i)}{\leq}L.D\leq
D^{2}+k+1\overset{(ii)}{\leq}2k+2\\\ &\text{ with equality in }(i)\text{ if
and only if }L\sim 2D\text{ and }L^{2}\leq 4k+4,\\\ &\text{ and equality in
}(ii)\text{ if and only if }L\sim 2D\text{ and }L^{2}=4k+4.\\\ \end{array}$

Since any effective divisor $D$ on $X$ has $D^{2}\geq 4$, for $k=2$,
$L^{2}\geq 8$ implies that $L$ is $2$-very ample.

If $L$ is $2$-very ample line bundle, then $\phi_{L}$ in (2.3) is an
embedding, hence $\phi_{L}^{*}{\mathcal{O}}_{{\mathbb{G}}}(1)$ is ample on
$X^{[2]}$, where ${\mathbb{G}}:=\operatorname{Grass}(2,H^{0}(L))$. As
explained in [4, §2], it is $L[2]-e$. Next since $L^{2}>0$ and $L^{2}\equiv
0\mod 4$, we suppose that $L^{2}=4$. In [15, Lemma 12], we showed that any
such $L=D_{n}:=\eta^{2n}$ ($n\in{\mathbb{Z}}$) and it is very ample, hence it
defines an embedding $X\hookrightarrow{\mathbb{P}}^{3}$. By this embedding we
identify $\phi_{L}:X^{[2]}\rightarrow\operatorname{Grass}(2,H^{0}(L))$ with

(2.5)
$\phi_{L}:X^{[2]}\rightarrow\operatorname{Grass}(1,{\mathbb{P}}^{3}),Z\mapsto\langle
Z\rangle$

where $\langle Z\rangle$ is the one dimensional span of $Z$ in
${\mathbb{P}}^{3}$. Moreover, since $X$ has no line, $\phi_{L}$ is a finite
morphism onto its image. Therefore,
$\phi^{*}{\mathcal{O}}_{{\mathbb{G}}}(1)=L[2]-e$ is ample on $X^{[2]}$. ∎

######  Theorem 2.2.

[11, Theorem 1] Let $F$ be a projective algebraic variety deformation
equivalent to Hilbert schemes of a K3 surface. Fix an ample divisor $H$ on
$F$. A divisor $C$ on $F$ is ample if $C.D>0$ for each divisor class $D$
satisfying $H.D>0$ and either

  1. (1)

$D^{2}\geq-2$; or

  2. (2)

$D^{2}=-10$ and $(D,H^{2}(F,{\mathbb{Z}}))=2{\mathbb{Z}}$.

######  Lemma 2.3.

For $X$ in Lemma 2.1, there is no divisor $D$ of square $-10$ on $X^{[2]}$.

###### Proof.

For $D=(x,z,y)$, suppose that $D^{2}=4(x^{2}+xy-y^{2})-2z^{2}=-10$. Then since
$z$ is odd, let $z=2z^{\prime}+1$ and then we have
$x^{2}+xy-y^{2}=2z^{\prime}(z^{\prime}+1)-2$, i.e., $x^{2}+xy-y^{2}\equiv
2\mod 4.$ Now if $x$ ($y$, resp.) is even, then $y$ ($x$, resp.) is also even,
hence a contradiction. So both $x,y$ are odd, but then $x^{2}+xy-y^{2}$ is
odd, a contradiction. ∎

In [19, Theorem 6.6], Oguiso showed that
$g=\iota_{0}\circ\iota_{1}\circ\iota_{2}$ without proof. Here we used the same
notation $g$ for the induced automorphism on $X^{[2]}$ from $X$ via (1.3). In
[15, Remark 2], we confirmed this relation. In [15, Theorem 2], the author
also found the relations between Beauville involutions and the natural
automorphism $g$ as follows.

######  Theorem 2.4.

[15, Theorem 2] For $k=0,1,$ or $2$ and $l\in{\mathbb{Z}}$,

(2.6) $\iota_{3l+k}=g^{l}\circ\iota_{k}\circ g^{-l}.$

Boissi$\grave{\text{e}}$re and Sarti proved that if $X$ is a K3 surface, then
an automorphism $f\in\operatorname{Aut}(X^{[n]})$ is natural if and only if it
preserves the diagonal (cf. [6]).

###  2.2. Hyperbolic geometry

Recall that the $n$-dimensional hyperbolic space, usually denoted by
${\mathbb{H}}^{n}$, is the unique simply connected, $n$-dimensional complete
Riemannian manifold with a constant negative sectional curvature equal to
$-1$. One of the models of hyperbolic plane is the upper half-plane

(2.7) ${\mathbb{H}}^{2}=\\{(x,y)\in{\mathbb{R}}^{2}\mid y>0\\}.$

Recall that the group of all orientation-preserving isometries of
${\mathbb{H}}^{2}$ is

(2.8)
$\text{M}\ddot{\text{o}}\text{b}^{+}({\mathbb{H}}^{2}):=\\{g(z)=\frac{az+b}{cz+d}\mid
a,b,c,d\in{\mathbb{R}},ad-bc=1\\},$

where $z=x+\sqrt{-1}y$, a complex number. Note that such an isometry
corresponds to a matrix

(2.9) $\begin{bmatrix}a&b\\\ c&d\end{bmatrix}\in{\rm SL}(2,{\mathbb{R}}).$

Moreover, since two matrices in ${\rm SL}(2,{\mathbb{R}})$ induce the same
isometry if and only if their matrices differ by a factor of $\pm 1$, hence
$\text{M}\ddot{\text{o}}\text{b}^{+}({\mathbb{H}}^{2})\cong{\rm
PSL}(2,{\mathbb{R}})$.

Any orientation-preserving isometry of ${\mathbb{H}}^{2}$ is one of the
following three types.

######  Definition 2.5.

Let $g(\neq\operatorname{id})$ be any M$\ddot{\text{o}}$bius transformation in
(2.8). Then

  1. (1)

$g$ is elliptic if and only if $g$ has a unique fixed point in
${\mathbb{H}}^{2}$;

  2. (2)

$g$ is hyperbolic if and only if $g$ has two fixed points on
$\partial{\mathbb{H}}^{2}$;

  3. (3)

$g$ is parabolic if and only if $g$ has a unique fixed point on
$\partial{\mathbb{H}}^{2}$,

where $\partial{\mathbb{H}}^{2}$ means the set of boundary points of
${\mathbb{H}}^{2}$.

Two M$\ddot{\text{o}}$bius transformations $f,g$ are conjugate if there is a
M$\ddot{\text{o}}$bius transformation $h$ such that $g=hfh^{-1}$. Any
M$\ddot{\text{o}}$bius transformation is conjugate to one of the above three
types. Indeed, a parabolic isometry is conjugate to $g(z)=z+s$,
$s\in{\mathbb{R}}$. An elliptic one is conjugate to
$g(z)=\frac{s}{-1/sz}=-s^{2}/z$, $s\in{\mathbb{R}}\setminus{0}$. A hyperbolic
one is conjugate to $g(z)=\frac{sz}{1/s}=s^{2}z$,
$s\in{\mathbb{R}}\setminus{0}$. Moreover, the types of hyperbolic isometries
are characterized by the trace.

######  Theorem 2.6.

[1, Theorem 4.3.4] Let $g(\neq\operatorname{id})$ be any
M$\ddot{\text{o}}$bius transformation in (2.8) with
$\operatorname{trace}(g):=a+d$. Then

  1. (1)

$g$ is parabolic if and only if $\operatorname{trace}^{2}(g)=4$;

  2. (2)

$g$ is elliptic if and only if $\operatorname{trace}^{2}(g)\in[0,4)$;

  3. (3)

$g$ is hyperbolic if and only if $\operatorname{trace}^{2}(g)\in(4,+\infty)$.

Let $f,g,h$ be elliptic isometries of order two with distinct fixed points
$u,v,w$ respectively. We assume that $u,v$ and $w$ are not collinear. For the
hyperbolic triangle with vertices $u,v$ and $w$, let $\alpha,\beta$ and
$\gamma$ be the angles and let $a,b$ and $c$ be the lengths of the opposite
sides respectively. It is known as the Sine rule for hyperbolic triangles that

(2.10) $\frac{\sinh a}{\sin\alpha}=\frac{\sinh b}{\sin\beta}=\frac{\sinh
c}{\sin\gamma}.$

Let $\lambda=\sinh a\sinh b\sin\gamma(=\sinh b\sinh c\sin\alpha=\sinh c\sinh
a\sin\beta$).

######  Theorem 2.7.

[1, Theorem 11.5.1] The absolute value of the trace of any of the isometires

$fgh,hfg,ghf,hgf,fhg,gfh$

is equal to $2\lambda$.

######  Definition 2.8.

[1, §5.1. and §9.4.][13, §3.2.]
$G\subset\text{M}\ddot{\text{o}}\text{b}^{+}({\mathbb{H}}^{2})$ is elementary
if there is a finite $G$-orbit in
${\mathbb{H}}^{2}\cup\partial{\mathbb{H}}^{2}$. A discrete subgroup of ${\rm
PSL}(2,{\mathbb{R}})$ is called Fuchsian group. For any Fuchsian group $G$,
the Dirichlet polygon $D(p)$ with center $p$ is the set

(2.11) $D(p)=\\{z\in{\mathbb{H}}^{2}\mid d(z,p)<d(z,g(p))\text{ for all }g\in
G\setminus\\{\operatorname{id}\\}\\},$

where $d$ is the hyperbolic metric on ${\mathbb{H}}^{2}$ and $p$ is any point
in ${\mathbb{H}}^{2}$ that is not fixed by any elliptic element of $G$.

Let $G$ be a finitely generated non-elementary Fuchsian group. Any Dirichlet
polygon $D(p)$ for $G$ is finite sided and topologically, $(D\cup\partial
D)/G$ is a compact surface $S$ of some genus, say $g$, with a certain number
of holes removed. It is independent of the choice of $p$, provided that $p$ is
not an elliptic fixed-point of $G$. The genus $g$ does not depend on the
choice of $D$. Let $r,s,t$ be the number of conjugacy classes of maximal
elliptic cyclic subgroups with orders $m_{1},\cdots,m_{r}$, the number of
conjugacy classes of maximal parabolic cyclic subgroups, and the number of
conjugacy classes of maximal hyperbolic boundary cyclic subgroups,
respectively.

######  Definition 2.9.

[1, Definition 10.4.1] The symbol

$(g:m_{1},\cdots,m_{r};s;t)$

is called the signature of $G$: each parameter is a non-negative integer and
$m_{j}\geq 2$.

It is known that every Fuchsian group has a presentation of the following form
(cf.[12],[16]).

(2.12)
$\begin{array}[]{lcl}\text{Generators}:&a_{1},b_{1},\cdots,a_{g},b_{g}&\text{(Hyperbolic)}\\\
&x_{1},x_{2},\cdots,x_{r}&\text{(Elliptic)}\\\
&p_{1},\cdots,p_{s}&\text{(Parabolic)}\\\
&h_{1},\cdots,h_{t}&\text{(Hyperbolic boundary elements)}\\\ \end{array}$
(2.13)
$\text{Relations}:x_{1}^{m_{1}}=x_{2}^{m_{2}}=\cdots=x_{r}^{m_{r}}=\prod_{i=1}^{g}a_{i}b_{i}a_{i}^{-1}b_{i}^{-1}\prod_{j=1}^{r}x_{j}\prod_{k=1}^{s}p_{k}\prod_{l=1}^{t}h_{l}=1.$

If $s+t>0$, the Fuchsian group is a free product of cyclic groups (cf.
[12],[16, §3]):

(2.14) $G\cong{\mathbb{Z}}_{m_{1}}\ast\cdots\ast{\mathbb{Z}}_{m_{r}}\ast
F_{v},$

where $F_{v}$ is a free group of rank $v=2g+s+t-1$.

######  Theorem 2.10.

[1, Theorem 11.5.2] Let $f,g$ and $h$ be elliptic involutions which generate a
non-elementary group $G$ and let $\lambda$ be given by Theorem 2.7.

  1. (1)

If $\lambda>1$ then $G$ is discrete and has signature $(0:2,2,2;0;1)$.

  2. (2)

If $\lambda=1$ then $G$ is discrete and has signature $(0:2,2,2;1;0)$.

  3. (3)

If $\lambda<1$ then $G$ is discrete only if $\lambda$ is one of the values

$\cos(\pi/q),q\geq 3;\hskip 5.69054pt\cos(2\pi/q),q\geq 5;\hskip
5.69054pt\cos(3\pi/q),q\geq 7:$

the possible signatures for $G$ are

$(0:2,2,2,q;0;0),(0:2,3,q;0;0),(0:2,4,q;0;0).$

For two elliptic isometries, we have the following theorem.

######  Theorem 2.11.

[17, §5] Let $A$ and $B$ be the elliptic generators of finite orders of
$\mathfrak{A}$ and $\mathfrak{B}$, respectively. If $AB\neq\operatorname{id}$
and $AB$ has real fixed points, then the group $G$ generated by $A$ and $B$ is
the free product of $\mathfrak{A}$ and $\mathfrak{B}$.

######  Example 2.12.

Let $f$ and $g$ are elliptic isometries of order $2$ without common fixed
points. If $fg$ is hyperbolic, then the group $G=\langle
f,g\rangle\cong{\mathbb{Z}}_{2}\ast{\mathbb{Z}}_{2}$.

##  3\. Proof

###  3.1. Proof of Theorem 1.1

First of all, we will show that
$\operatorname{Aut}(X^{[2]})\cong\langle\iota_{0},\iota_{1},\iota_{2}\rangle$.

Recall that the natural automorphisms leave invariant the half-exceptional
divisor $e$. Whereas for the non-natural automorphisms, we have the following
lemma:

######  Lemma 3.1.

Let $f\in\operatorname{Aut}(X^{[2]})$ be a non-natural automorphism of
$X^{[2]}$. Then $f^{*}(e)=xh_{1}+\lambda e+yh_{2}$ with
$xh_{1}+yh_{2}\in\operatorname{Amp}(X)$ and $\lambda<0$.

###### Proof.

Let $f^{*}(e)=(x,\lambda,y)$ with respect to the basis $\\{h_{1},e,h_{2}\\}$
of $NS(X^{[2]})$. Since $f^{*}(2e)=f^{*}(E)\neq E$ and $f^{*}(E)$ and $E$ are
both effective and irreducible, $(f^{*}(e),e)=-2\lambda\geq 0$. Since
$(f^{*}(e))^{2}=(e)^{2}=-2$, $4(x^{2}+xy-y^{2})-2\lambda^{2}=-2$, hence
$\lambda\neq 0$, i.e., $\lambda<0$. Moreover,
$(x,y)^{2}=4(x^{2}+xy-y^{2})=2(\lambda^{2}-1)>0$, i.e., either
$(x,y)\in\operatorname{Amp}(X)$ or $-(x,y)\in\operatorname{Amp}(X)$ by (2.4).
Moreover, since $(1,0),(1,1)\in\operatorname{Amp}(X)$, $(1,-1,0)$ and
$(1,-1,1)$ are ample on $X^{[2]}$ by Lemma 2.1, hence intersections with these
imply that $x>0$, i.e., $(x,y)$ is an ample divisor of $X$. ∎

For a divisor $D\in NS(X^{[2]})$, let $[D]_{e}$ be the $e$-coordinate with
respect to the basis $\\{h_{1},e,h_{2}\\}$.

######  Theorem 3.2.

$\operatorname{Aut}(X^{[2]})\cong\langle\iota_{0},\iota_{1},\iota_{2}\rangle.$

###### Proof.

Let $f\in\operatorname{Aut}(X^{[2]})$ be a non-natural automorphism of
$X^{[2]}$. By Lemma 3.1, we have that $f^{*}(e)=(x,\lambda,y)$ with
$(x,y)\in\operatorname{Amp}(X)$ and $\lambda<0$. If
$[\iota_{k}^{*}f^{*}(e)]_{e}\geq 0$ for some $k\in{\mathbb{Z}}$, then again by
Lemma 3.1, $\iota_{k}^{*}f^{*}(e)=e$, i.e., $\iota_{k}^{*}f^{*}=g^{n*}$ for
some $n\in{\mathbb{Z}}$. Now by (1.4), $g^{n}=f\iota_{k}$ in
$\operatorname{Aut}(X^{[2]})$. Since $g=\iota_{0}\iota_{1}\iota_{2}$ and by
Theorem 2.13, $f\in\langle\iota_{0},\iota_{1},\iota_{2}\rangle.$ Next, suppose
that for all $k\in{\mathbb{Z}}$, $[\iota_{k}^{*}f^{*}(e)]_{e}<0$. Then we
claim the following.

######  Claim.

For some $m\in{\mathbb{Z}}$, $[\iota_{m}^{*}f^{*}(e)]_{e}>\lambda$.

######  Proof of Claim.

Suppose that for all $k\in{\mathbb{Z}}$,
$[\iota_{k}^{*}f^{*}(e)]_{e}\leq\lambda$. In [15, Example 1],
$\iota_{0}^{*},\iota_{1}^{*},\iota_{2}^{*}\in O(H^{2}(X^{[2]},{\mathbb{Z}}))$
are given as follows with respect to the basis $\\{h_{1},e,h_{2}\\}$:

(3.1) $\iota_{0}^{*}=\begin{bmatrix}3&2&2\\\ -4&-3&-2\\\
0&0&-1\end{bmatrix},\hskip 5.69054pt\iota_{1}^{*}=\begin{bmatrix}5&2&-2\\\
-6&-3&2\\\ 6&2&-3\end{bmatrix},\hskip
5.69054pt\iota_{2}^{*}=\begin{bmatrix}27&4&-16\\\ -14&-3&8\\\
42&6&-25\end{bmatrix}.$

In particular, $[\iota_{0}^{*}f^{*}(e)]_{e}=-4x-3\lambda-2y\leq\lambda$, i.e.,
$2x+y\geq-2\lambda$. Similarly,
$[\iota_{1}^{*}f^{*}(e)]_{e}=-6x-3\lambda+2y\leq\lambda$, i.e.,
$3x-y\geq-2\lambda$ and
$[\iota_{2}^{*}f^{*}(e)]_{e}=-14x-3\lambda+8y\leq\lambda$, i.e.,
$7x-4y\geq-2\lambda$. Moreover, by Theorem 2.13,
$\iota_{3l+k}^{*}=g^{-l*}\circ\iota_{k}^{*}\circ g^{l*}$ for
$l\in{\mathbb{Z}}$ and $k=0,1,2$. Since $g$ is natural, $g^{*}(e)=e$, hence
$[\iota_{3l+k}^{*}f^{*}(e)]_{e}=[\iota_{k}^{*}g^{l*}f^{*}(e)]_{e}$, where
$k=0,1,$ or $2$. Let $g^{l*}f^{*}(e)=(x^{\prime},\lambda,y^{\prime})$ with
$(x^{\prime},y^{\prime})\in\operatorname{Amp}(X)$. If we let
$r^{\prime}=y^{\prime}/x^{\prime}$, then $(g^{l*}f^{*}(e))^{2}=(e)^{2}=-2$
gives

(3.2) $4x^{\prime 2}(1+r^{\prime}-r^{\prime 2})=2\lambda^{2}-2.$

Now for $k=0$,
$[\iota_{0}^{*}g^{l*}f^{*}(e)]_{e}=-4x^{\prime}-3\lambda-2y^{\prime}\leq\lambda$
gives that $2x^{\prime}+y^{\prime}\geq-2\lambda$. By taking the squares, we
have that $(x^{\prime}(2+2r^{\prime}))^{2}\geq 4\lambda^{2}$. By substituting
(3.2), we have that $x^{\prime 2}(4+4r^{\prime}+r^{\prime 2})\geq 8x^{\prime
2}(1+r^{\prime}-r^{\prime 2})+4$ or $x^{\prime 2}(9r^{\prime
2}-4r^{\prime}-4)\geq 4$. However, the left hand side is negative if
$\frac{2-2\sqrt{10}}{9}<r^{\prime}<\frac{2+2\sqrt{10}}{9}$.

Similarly for $k=1$, we have $x^{\prime 2}(9r^{\prime 2}-14r^{\prime}+1)\geq
4$, and the left hand side is negative if
$\frac{7-2\sqrt{10}}{9}<r^{\prime}<\frac{7+2\sqrt{10}}{9}$. For $k=2$, we have
$x^{\prime 2}(24r^{\prime 2}-64r^{\prime}+41)\geq 4$, and the left hand side
is negative if $\frac{16-\sqrt{10}}{12}<r^{\prime}<\frac{16+\sqrt{10}}{12}$.
Since the ample cone of $X$ is given in (2.4), for some power $l$, at least
one of three inequalities induces a contradiction, hence
$[\iota_{m}^{*}f^{*}(e)]_{e}>\lambda$.

If we continue this process at least $|\lambda|-1$ times, we get a
contradiction to $[\iota_{k}^{*}f^{*}(e)]_{e}<0$ for all $k\in{\mathbb{Z}}$.
Hence, we may assume that for some
$\iota^{*}=\iota_{k_{1}}^{*}\iota_{k_{2}}^{*}\cdots\iota_{k_{j}}^{*}$,
$[\iota^{*}f^{*}(e)]_{e}\geq 0$. Then again by Lemma 3.1,
$\iota^{*}f^{*}(e)=e$, i.e., $f\iota=g^{l}$ in $\operatorname{Aut}(X^{[2]})$
for some $l\in{\mathbb{Z}}$. Again by $g=\iota_{0}\iota_{1}\iota_{2}$ and by
Theorem 2.13, $f\in\langle\iota_{0},\iota_{1},\iota_{2}\rangle.$ ∎

### 3.2.

Next, we will show that
$\langle\iota_{0},\iota_{1},\iota_{2}\rangle\cong{\mathbb{Z}}_{2}\ast{\mathbb{Z}}_{2}\ast{\mathbb{Z}}_{2}$.

Note that the Picard lattice $NS(X^{[2]})$ of Hilbert square of the K3 surface
$X$ has signature $(1,2)$. For an ample divisor $H$ on $X^{[2]}$, we consider
the light cone of $NS(X^{[2]})$:

(3.3) $\mathcal{L}^{+}:=\\{D\in\mathop{\rm
NS}\nolimits(X^{[2]})\otimes{\mathbb{R}}\mid D^{2}>0,D.H>0\\}.$

If we let $H=(1,-1,0)$ with respect to the basis $\\{h_{1},e,h_{2}\\}$, then

(3.4) $\mathcal{L}^{+}=\\{(x,\lambda,y)\in\mathop{\rm
NS}\nolimits(X^{[2]})\otimes{\mathbb{R}}\mid
2(x^{2}+xy-y^{2})>\lambda^{2},2x+y+\lambda>0\\}.$

Since the signature of $\mathop{\rm NS}\nolimits(X^{[2]})$ is $(1,2)$,
$\mathcal{L}^{+}/{\mathbb{R}}^{+}$ with distance $|AB|$ defined by

(3.5) $\|A\|\|B\|\cosh|AB|=(A.B)$

is isomorphic to $2$-dimensional hyperbolic geometry $\mathbb{H}^{2}$, where
$\|A\|=\sqrt{(A.A)}$.

Note that for each $k\in{\mathbb{Z}}$, $\iota_{k}^{*}$ preserves distances of
${\mathcal{L}}^{+}/{\mathbb{R}}^{+}$, hence
$\iota_{k}^{*}\in\operatorname{Aut}({\mathbb{H}}^{2})$ and it is an
involution. Fixed points of $\iota_{0}^{*}$, $\iota_{1}^{*}$ and
$\iota_{2}^{*}$ are $[1,-1,0]$, $[1,-1,1]$, and $[2,-1,3]$, respectively. Here
$[x,\lambda,y]$ denotes a class of points modulo ${\mathbb{R}}^{+}$. Note that
since these fixed points correspond to interior points of ${\mathbb{H}}^{2}$,
$\iota_{0}^{*},\iota_{1}^{*},\iota_{2}^{*}$ are all elliptic isometries.

######  Remark 3.3.

Note that fixed points $[1,-1,0],[1,-1,1],[2,-1,3]$ correspond to ample
divisors of square $2$ on $X^{[2]}$ with respect to the Beauville-Bogomolov-
Fujiki’s form.

In the rest, we will show that there is no relation between
$\iota_{0}^{*},\iota_{1}^{*},\iota_{2}^{*}$.

It is known that if $\gamma_{1},\gamma_{2}$ are two geodesics in
${\mathbb{H}}^{2}$ and $z_{i}\in\gamma_{i}$ are two points on them, then there
exists an isometry $g\in M\ddot{\text{o}}b^{+}({\mathbb{H}}^{2})$ such that
$g(\gamma_{1})=\gamma_{2}$ and $g(z_{1})=z_{2}$. Hence we may assume that two
points of $[1,-1,0],[1,-1,1],[2,-1,3]$ map to $(0,1),(0,k)\in{\mathbb{H}}^{2}$
with $k>1$ on the $y$-axis of ${\mathbb{H}}^{2}$, a hyperbolic line. Then the
remaining point maps to $(b,m)\in{\mathbb{H}}^{2}$ with $b\neq 0$. Moreover,
$\iota_{0}^{*},\iota_{1}^{*},\iota_{2}^{*}$ correspond to the elliptic
isometries $T,U,V$ whose fixed points are $(b,m),(0,1),(0,k)$ respectively.
Hence these M$\ddot{\text{o}}$bius transformations correspond to the following
matrices in ${\rm SL}(2,{\mathbb{R}})$;

(3.6) $U=\begin{bmatrix}0&1\\\ -1&0\end{bmatrix},\hskip
5.69054ptV=\begin{bmatrix}0&k\\\ -1/k&0\end{bmatrix}$

and

(3.7) $T=\begin{bmatrix}1&b\\\ 0&1\end{bmatrix}\begin{bmatrix}0&m\\\
-1/m&0\end{bmatrix}\begin{bmatrix}1&-b\\\
0&1\end{bmatrix}=\begin{bmatrix}b/m&-(b^{2}+m^{2})/m\\\
1/m&-b/m\end{bmatrix}.$

Note that all traces of $T,U,V$ are zero, i.e, elliptic isometries with fixed
points $(b,m),(0,1),(0,k)$ respectively. Note also that the first matrix in
(3.7) corresponds to the translation $z\mapsto z+b$.

Consider the group $G$ generated by $T,U,V$. Note that any two of three
generates a free group ${\mathbb{Z}}_{2}\ast{\mathbb{Z}}_{2}$ (Example 2.12).
Hence any element in $G$ can be written as $TX_{1}TX_{2}\cdots TX_{k}$, where
each $X_{i}$ is of the form

(3.8) $(UV)^{n},(UV)^{n}U,(VU)^{s},(VU)^{s}V$

in the free group ${\mathbb{Z}}_{2}\ast{\mathbb{Z}}_{2}$ generated by $U,V$,
where $n,s\in{\mathbb{Z}}$.

######  Lemma 3.4.

$TX_{i}$ is hyperbolic for any $X_{i}$ in (3.8).

###### Proof.

We use the classification of a hyperbolic isometry by the trace (Theorem 2.6).
First,

(3.9) $TU=\begin{bmatrix}(b^{2}+m^{2})/m&b/m\\\ b/m&1/m\end{bmatrix}$

and its trace is $b^{2}/m+m+1/m>2$, hence it is hyperbolic. Similarly,

(3.10) $T(UV)^{s}U=TU(VU)^{s}=\begin{bmatrix}\frac{b^{2}+m^{2}}{m}k^{s}&*\\\
*&\frac{1}{m}k^{-s}\end{bmatrix}$

and its trace is $\frac{b^{2}}{m}k^{s}+mk^{s}+(mk^{s})^{-1}>2$, hence it is
hyperbolic. Moreover,

(3.11)
$T(VU)^{s-1}V=TU(UV)^{s}=\begin{bmatrix}\frac{b^{2}+m^{2}}{m}k^{-s}&*\\\
*&\frac{1}{m}k^{s}\end{bmatrix}$

and its trace is $\frac{b^{2}}{m}k^{-s}+mk^{-s}+(mk^{-s})^{-1}>2$, hence it is
also hyperbolic.

Next,

(3.12) $T(VU)^{s}=TU(UV)^{s}U=\begin{bmatrix}-\frac{b}{m}k^{s}&*\\\
*&\frac{b}{m}k^{-s}\end{bmatrix}$

and its trace is $|\frac{b}{m}(k^{s}-k^{-s})|>2$ by induction. Indeed for
$s=1$, $TVU$ corresponds to one of $\iota_{i}^{*}\iota_{j}^{*}\iota_{k}^{*}$
where $i,j,k\in\\{0,1,2\\}$ are distinct. It is enough to show that
$\iota_{2}^{*}\iota_{1}^{*}\iota_{0}^{*}$ is hyperbolic by Theorem 2.7.

(3.13) $\iota_{2}^{*}\iota_{1}^{*}\iota_{0}^{*}=\begin{bmatrix}5&0&8\\\
0&1&0\\\ 8&0&13\end{bmatrix}$

has two fixed points

(3.14) $((-1+\sqrt{5})/2,0,1)\hskip 5.69054pt\text{ and }\hskip
5.69054pt(-(1+\sqrt{5})/2,0,1)$

on the boundary of ${\mathcal{L}}^{+}$, hence they correspond to two points on
$\partial{\mathbb{H}}^{2}$. Thus $TVU$ is hyperbolic, i.e., the absolute value
of its trace $|\frac{b}{m}(k-k^{-1})|>2$. For $s=2$,

(3.15)
$\begin{array}[]{lcl}|\frac{b}{m}(k^{2}-k^{-2})|&=&|\frac{b}{m}(k-k^{-1})|(k+k^{-1})\\\
&>&|\frac{b}{m}(k-k^{-1})|\\\ &>&2.\\\ \end{array}$

Now suppose that $|k^{n}-k^{-n}|\geq|k^{n-1}-k^{-n+1}|$. Then

(3.16)
$\begin{array}[]{lcl}|k^{n+1}-k^{-n-1}|&=&|(k^{n}-k^{-n})(k+k^{-1})-(k^{n-1}-k^{-n+1})|\\\
&\geq&|(k^{n}-k^{-n})(k+k^{-1}-1)|\\\ &\geq&|k^{n}-k^{-n}|.\\\ \end{array}$

Hence the trace $|\frac{b}{m}(k^{s}-k^{-s})|>2$ and $T(VU)^{s}$ is hyperbolic.

Similarly,

(3.17) $T(UV)^{s+1}=TU(VU)^{s}V=\begin{bmatrix}-\frac{b}{m}k^{-s-1}&*\\\
*&\frac{b}{m}k^{s+1}\end{bmatrix}$

and its trace is $|\frac{b}{m}(k^{s+1}-k^{-s-1})|>2$, hence $T(UV)^{s+1}$ is
hyperbolic. ∎

Now by Theorem 2.10, $G$ has the signature $(0;2,2,2;0,1)$ with
$g=0,m_{1}=m_{2}=m_{3}=2,s=0,$ and $t=1$. Since $s+t>0$, $G$ is a free product
of three elliptic elements by (2.14), i.e., $G=\langle
T,U,V\rangle\cong\langle\iota_{0}^{*},\iota_{1}^{*},\iota_{2}^{*}\rangle\cong{\mathbb{Z}}_{2}\ast{\mathbb{Z}}_{2}\ast{\mathbb{Z}}_{2}$.
Now by (1.4), in $\operatorname{Aut}(X^{[2]})$,
$\iota_{0},\iota_{1},\iota_{2}$ has no relation, hence
$\operatorname{Aut}(X^{[2]})\cong\langle\iota_{0},\iota_{1},\iota_{2}\rangle\cong{\mathbb{Z}}_{2}\ast{\mathbb{Z}}_{2}\ast{\mathbb{Z}}_{2}$.

### Acknowledgements

The author would like to thank Professor Keiji Oguiso for letting me to read
his preprint.

## References

  * [1] Beardon, A.F. (1983). The geometry of discrete groups. Graduate texts in mathematics 91. New York: Springer.
  * [2] Beauville, A. (1983). Some remarks on Kähler manifolds with $c_{1}=0$. Progr. Math. 39:1-26.
  * [3] Beauville, A. (1983). Varietes kahleriennes dont la premiere classe de Chern est nulle. J. Differential Geometry. 18:755-782.
  * [4] Bertram, A., Coskun, I. (2013). The birational geometry of the Hilbert scheme of points on surfaces. in Birational geometry, rational curves, and arithmetic. New York: Springer. 15–55.
  * [5] Boissière, S., Cattaneo, A., Nieper-Wisskirchen, M., Sarti, A. (2016). The automorphism group of the Hilbert scheme of two points on a generic projective K3 surface. In K3 surfaces and Their Moduli. Progr. Math. 315:1-15.
  * [6] Boissière, S., Sarti, A. (2012). A note on automorphisms and birational transformations of holomorphic symplectic manifolds. Proc. Amer. Math. Soc. 140(12):4053-4062.
  * [7] Catanese, F., G$\ddot{\text{o}}$ttsche, L. (1990). d-very-ample line bundles and embeddings of Hilbert schemes of $0$-cycles, Manuscripta Math. 68(3):337–341.
  * [8] Cattaneo, A. (2019). Automorphisms of Hilbert schemes of points on a generic projective K3 surface. Math. Nach. 292:2137-2152.
  * [9] Cayley, A. (1869-71). A memoir on quartic surfaces. Proc. London. Math. Soc. 3:19-69.
  * [10] Festi, D., Garbagnati, A., van Geemen, B., van Luijk, R. (2013). The Cayley–Oguiso automorphism of positive entropy on a K3 surface. J. Mod. Dyn. 7(1):75–97.
  * [11] Hassett, B., Tschinkel, Y. (2009). Moving and ample cones of holomorphic symplectic fourfolds. Geom. Funct. Anal. 19:1065-1080.
  * [12] Harpe, P. De La, Voiculescu, D. (1995). A problem on the $II_{1}$-factors of Fuchsian groups. Astérisque 232:155-158.
  * [13] Katok, S. (1992). Fuchsian Groups. University of Chicago Press. Chicago.
  * [14] Knutsen, A.L. (2001). On kth-order embeddings of K3 surfaces and Enriques surfaces. Manuscripta Math. 104(2):211–237.
  * [15] Lee, K. (2023). Automorphisms of some K3 surfaces and Beauville involutions of their Hilbert schemes. Rend. Circ. Mat. Palermo, II. Ser 72(4):2443-2457.
  * [16] Liebeck, M.W., Shalev, A. (2004). Fuchsian groups, coverings of Riemann surfaces, subgroup growth, random quotients and random walks. J.Algebra 276:552-601.
  * [17] Lyndon, R.C., Ullman, J.L. (1968). Pairs of real 2-by-2 matrices that generate free products. Michigan Math. J. 15(2):161-166.
  * [18] Oguiso, K. (2015). Free Automorphisms of positive entropy on smooth Kähler surfaces. In: Algebraic geometry in east Asia-Taipei 2011. Adv. Stud. Pure Math. 65:187-199.
  * [19] Oguiso, K. (2014). Some aspects of explicit birational geometry inspired by complex dynamics. In: Proceedings of the International Congress of Mathematicians. Seoul 2014 Vol.II:695-721.