$\underline{\operatorname{End}}(\\{f_{2}^{*}p_{j}^{*}z_{1}\cdot
U_{j+1},f_{2}^{*}s_{1}(j)\\})\simeq{\mathcal{O}}/(f_{2}^{*}p_{j}^{*}z_{1}\cdot
U_{j+1},f_{2}^{*}s_{1}(j))$;

  * •

$\underline{\operatorname{End}}(\\{f_{2}^{*}p_{j}^{*}z_{0}\cdot
U_{j},f_{2}^{*}s_{2}(j)\\})\simeq{\mathcal{O}}/(f_{2}^{*}p_{j}^{*}z_{0}\cdot
U_{j},f_{2}^{*}s_{2}(j))$;

  * •

$\underline{\operatorname{Hom}}(\\{C_{1,i+1}^{-1}\cdot
U_{i+1},C_{1,i}\\},\\{C_{1,i}\cdot
U_{i+1},C_{1,i+1}^{-1}\\})\simeq{\mathcal{O}}/(C_{1,i+1}^{-1},C_{1,i})[-1]$;

(we can omit the twists by line bundles in the definition of $P_{ij}$ and
$P_{i+1,j}$ since we are working near
$\overline{R}_{ij}\cap\overline{R}_{i+1,j}$). Using trivializations of the
relevant line bundles over
$\overline{R}_{ij}\cap\overline{R}_{i+1,j}\times{\mathbb{A}}^{k}$, we deduce
the result in this case.

Similarly we consider other $\operatorname{Hom}$-spaces in (i) and (ii). Let
us show how a similar computation goes in (iii). For example, for $i-j>1$ we
compute $\underline{\operatorname{Hom}}(P_{i+1,j+1},P_{i,j})$ in the
neighborhood of $(R_{i,j}\cap R_{i+1,j+1})\times{\mathbb{A}}^{k}$ as the
tensor product of

  * •

for $r\neq i+1,j+1$,
$\underline{\operatorname{End}}(\\{U_{r},V_{r}\\})\simeq{\mathcal{O}}/(U_{r},V_{r})$;

  * •

$\underline{\operatorname{Hom}}(\\{C_{1,i+1}^{-1}\cdot
U_{i+1},C_{1,i}\\},\\{C_{1,i}\cdot
U_{i+1},C_{1,i+1}^{-1}\\})\simeq{\mathcal{O}}/(C_{1,i+1}^{-1},C_{1,i})[-1]$;

  * •

$\underline{\operatorname{Hom}}(\\{C_{2,j+1}^{-1}\cdot
U_{j+1},C_{2,j}\\},\\{C_{2,j}\cdot
U_{j+1},C_{2,j+1}^{-1}\\})\simeq{\mathcal{O}}/(C_{2,j+1}^{-1},C_{2,j})[-1]$.

Recall that $R_{ij}\cap R_{i+1,j+1}$ is a single point $p_{i+1,j+1}$. Thus, we
get an isomorphism on $p_{i+1,j+1})\times{\mathbb{A}}^{k}$,

$\underline{\operatorname{Hom}}(P_{i+1,j+1},P_{i,j})\simeq{\mathcal{O}}_{p_{i+1,j+1}\times{\mathbb{A}}^{2}_{U_{i+1},U_{j+1}}}[-2],$

so

$\operatorname{Hom}(P_{i+1,j+1},P_{ij})\simeq\mathbf{k}[U_{i+1},U_{j+1}][-2]\simeq\operatorname{End}(P_{ij},P_{i+1,j+1})[-2]$

and we can chase the generator in the above calculation. ∎

####  5.3.3. Morphisms between $Q_{i}$’s

We have the following analog of Lemma 5.3.5. The proof is analogous (but
simpler), so we omit it.

######  Lemma 5.3.6.

(i) For $j\geq 0$, we have generators

$\alpha_{j}\in\operatorname{Hom}^{-1-2(k-j)}(Q_{j+1},Q_{j}),\
\beta_{j}\in\operatorname{Hom}^{1+2(k-j)}(Q_{j},Q_{j+1})$

such that $\alpha_{j}\beta_{j}=\beta_{j}\alpha_{j}=U_{j+1}$, and such that

$\operatorname{Hom}^{*}(Q_{j+1},Q_{j})=\operatorname{End}(Q_{j},Q_{j+1})\cdot\alpha_{j},\
\
\operatorname{Hom}^{*}(Q_{j},Q_{j+1})=\operatorname{End}(Q_{j},Q_{j+1})\cdot\beta_{j}$

in a way compatible with the
$\operatorname{End}(Q_{j})-\operatorname{End}(Q_{j+1})$-bimodule structure.
The generator $\alpha_{j}$ (resp., $\beta_{j}$) comes from the exact triangle

${\mathcal{O}}_{R_{*j}}(-2-2(k-j))\rTo{c_{2,j}}{\mathcal{O}}_{R_{*j}\cup
R_{*j+1}}\to{\mathcal{O}}_{R_{*j+1}}\rTo{\alpha_{j}}{\mathcal{O}}_{R_{*j+1}}[1](-2-2(k-j))$

(resp., from the triangle

${\mathcal{O}}_{R_{*j+1}}(2(k-j))\rTo{c_{2,j+1}^{-1}}{\mathcal{O}}_{R_{*j}\cup
R_{*j+1}}\to{\mathcal{O}}_{R_{*j}}\rTo{\beta_{j}}{\mathcal{O}}_{R_{*j+1}}[1](2(k-j))\
),$

well defined near $R_{*j}\cap R_{*j+1}$.

####  5.3.4. Morphisms between $Q_{j}$ and $P_{ij}$

######  Lemma 5.3.7.

For $k>i>j$ and any $r$, one has
$\operatorname{Hom}(P_{ij},Q_{r})=\operatorname{Hom}(Q_{r},P_{ij})=0$.

###### Proof.

We only have to consider the cases $r=j-1$, $j$ or $j+1$. In addition, we can
use the vanishing criterion for the proper morphism
$\overline{{\mathcal{X}}}_{2,k}\to{\mathbb{A}}^{2k+2}_{U_{1},\ldots,U_{k},V_{1},\ldots,V_{k},X_{1},X_{2}}$
as in Lemma 2.3.4(ii). Namely, by this Lemma, assuming that either $j>0$ or
$r>0$, we obtain that the vanishing of $\operatorname{Hom}(Q_{r},P_{ij})$
implies the vanishing of $\operatorname{Hom}(P_{ij},Q_{r})$. Indeed, the local
computation shows that $\underline{\operatorname{Hom}}(P_{ij},Q_{r})$ is a
push-forward of a coherent sheaf from $R_{ij}\cap R_{*r}$, so we can use the
fact that for $j>0$ (resp., $r>0$), the relative canonical bundle has trivial
restriction to $R_{ij}$ (resp., $R_{*j}$) (which follows from Corollary
2.4.3).

However, in the case $r=j=0$, we have to deal with both
$\operatorname{Hom}$-spaces.

Case $r=j$, $\operatorname{Hom}(Q_{j},P_{ij})$. As before, we use the
equations of $\overline{R}_{*j}$ and Lemma 2.3.6. For $i-1>j>0$, we use the
splitting (5.3.1) of $W$, and consider the tensor product of sheaf
homomorphisms between the following Koszul matrix factorizations, twisted by
$f_{1}^{*}p_{i}^{*}{\mathcal{O}}(-1)$ (due to the definition of $P_{ij}$),

  * •

for $r\neq i,i+1,j,j+1$,
$\underline{\operatorname{End}}(\\{U_{r},V_{r}\\})\simeq{\mathcal{O}}/(U_{r},V_{r})$;

  * •

$\underline{\operatorname{Hom}}(\\{U_{i+1},V_{i+1}\\},\\{f_{1}^{*}p_{i}^{*}z_{1}\cdot
U_{i+1},f_{1}^{*}s_{1}(i)\\})\simeq{\mathcal{O}}/(U_{i+1},f_{1}^{*}s_{1}(i))$;

  * •

$\underline{\operatorname{Hom}}(\\{U_{i},V_{i}\\},\\{f_{1}^{*}p_{i}^{*}z_{0}\cdot
U_{i},f_{1}^{*}s_{2}(i)\\})\simeq{\mathcal{O}}/(U_{i},f_{1}^{*}s_{2}(i))$;

  * •

$\underline{\operatorname{End}}(\\{f_{2}^{*}p_{j}^{*}z_{1}\cdot
U_{j+1},f_{2}^{*}s_{1}(j)\\})\simeq{\mathcal{O}}/(f_{2}^{*}p_{j}^{*}z_{1}\cdot
U_{j+1},f_{2}^{*}s_{1}(j))$;

  * •

$\underline{\operatorname{End}}(\\{f_{2}^{*}p_{j}^{*}z_{0}\cdot
U_{j},f_{2}^{*}s_{2}(j)\\})\simeq{\mathcal{O}}/(f_{2}^{*}p_{j}^{*}z_{0}\cdot
U_{j},f_{2}^{*}s_{2}(j))$.

Thus, we get an isomorphism on $\overline{R}_{ij}\times{\mathbb{A}}^{k}$,

$\displaystyle\underline{\operatorname{Hom}}(Q_{j},P_{ij})\simeq{\mathcal{O}}/((U_{r}\
|\ r\neq j,j+1),U_{j+1}\cdot
f_{2}^{*}p_{j}^{*}z_{1},f_{2}^{*}p_{j}^{*}z_{0}\cdot U_{j})\otimes
f_{1}^{*}p_{i}^{*}{\mathcal{O}}(-1)=$ $\displaystyle{\mathcal{O}}_{Z}(-1),$

where $Z\simeq{\mathbb{P}}^{1}\times C$, with ${\mathcal{O}}(-1)$ coming from
the factor ${\mathbb{P}}^{1}$. Here, for $j>0$, $C$ is the nodal curve
$C_{2}(U_{j},U_{j+1})$, as in the proof of Lemma 5.3.3. For $j=0$, we have
$C=C_{2}(U_{0})$. Calculating the cohomology of ${\mathcal{O}}(-1)$ on this
product we get zero.

Case $r=j=0$, $\operatorname{Hom}(P_{i0},Q_{0})$. The sheaf
$\underline{\operatorname{Hom}}(P_{i0},Q_{0})\otimes
f_{1}^{*}p_{i}^{*}{\mathcal{O}}(-1)$ is given by the tensor product of

  * •

$\underline{\operatorname{End}}(\\{U_{r},V_{r}\\})\simeq{\mathcal{O}}/(U_{r},V_{r})$,
for $r\neq 0,i,i+1$;

  * •

$\underline{\operatorname{Hom}}(\\{f_{1}^{*}p_{i}^{*}z_{1}\cdot
U_{i+1},f_{1}^{*}s_{1}(i)\\},\\{U_{i+1},V_{i+1}\\})\simeq{\mathcal{O}}/(f_{1}^{*}s_{1}(i),U_{i+1})\otimes
f_{1}^{*}p_{i}^{*}{\mathcal{O}}(-1)$;

  * •

$\underline{\operatorname{Hom}}(\\{f_{1}^{*}p_{i}^{*}z_{0}\cdot
U_{i},f_{1}^{*}s_{2}(i)\\},\\{(U_{i},V_{i}\\})\simeq{\mathcal{O}}/(f_{1}^{*}s_{2}(i),U_{i})\otimes
f_{1}^{*}p_{i}^{*}{\mathcal{O}}(-1)$;

  * •

$\underline{\operatorname{End}}(\\{f_{2}^{*}p_{0}^{*}z_{1}\cdot
U_{1},f_{2}^{*}s_{1}(0)\\})\simeq{\mathcal{O}}/(f_{2}^{*}p_{0}^{*}z_{1}\cdot
U_{1},f_{2}^{*}s_{1}(0))$,

where for $i=1$ we replace the last two factors with

$\displaystyle\underline{\operatorname{Hom}}(\\{f_{1}^{*}p_{1}^{*}z_{0}\cdot
f_{2}^{*}p_{0}^{*}z_{1}\cdot U_{1},s_{1}\\},\\{f_{2}^{*}p_{0}^{*}z_{1}\cdot
U_{1},f_{2}^{*}s_{1}(0)\\})\simeq$
$\displaystyle{\mathcal{O}}/(f_{2}^{*}p_{0}^{*}z_{1}\cdot U_{1},s_{1})\otimes
f_{1}^{*}p_{1}^{*}{\mathcal{O}}(-1).$

This leads to an isomorphism over $\overline{R}_{i0}\times{\mathbb{A}}^{k}$,

$\underline{\operatorname{Hom}}(P_{i0},Q_{0})\simeq{\mathcal{O}}/((U_{r}\ |\
r\neq 0),f_{2}^{*}p_{0}^{*}z_{1}\cdot U_{1})\otimes
f_{1}^{*}p_{i}^{*}{\mathcal{O}}(-1)\simeq{\mathcal{O}}_{Z}(-1),$

where $Z={\mathbb{P}}^{1}\times C_{2}(U_{0})$, with ${\mathcal{O}}(-1)$ coming
from the factor ${\mathbb{P}}^{1}$. Hence, the cohomology of this sheaf
vanish.

Case $r=j+1$, $\operatorname{Hom}(Q_{j+1},P_{ij})$. Assume first that $i>j+1$.
We can work locally near $(R_{ij}\cap R_{*,j+1})\times{\mathbb{A}}^{k}$, so we
can replace $f_{2}^{*}s_{1}(j)$ with $C_{2,j+1}^{-1}$ and $f_{2}^{*}s_{2}(j)$
with $V_{j}$ (resp., (resp., $f_{2}^{*}s_{1}(j+1)$ with $V_{j+2}$ and
$f_{2}^{*}s_{2}(j+1)$ with $C_{2,j}$) in the equations of $R_{ij}$ (resp.,
$R_{*,j+1}$). Thus, $\underline{\operatorname{Hom}}(Q_{j+1},P_{ij})\otimes
f_{1}^{*}p_{i}{\mathcal{O}}(1)$ is the tensor product of

  * •

$\underline{\operatorname{End}}(\\{U_{r},V_{r}\\})\simeq{\mathcal{O}}/(U_{r},V_{r})$,
for $r\neq i,i+1,j+1$;

  * •

$\underline{\operatorname{Hom}}(\\{U_{i+1},V_{i+1}\\},\\{f_{1}^{*}p_{i}^{*}z_{1}\cdot
U_{i+1},f_{1}^{*}s_{1}(i)\\})\simeq{\mathcal{O}}/(U_{i+1},f_{1}^{*}s_{1}(i));$

  * •

$\underline{\operatorname{Hom}}(\\{U_{i},V_{i}\\},\\{f_{1}^{*}p_{i}^{*}z_{0}\cdot
U_{i},f_{1}^{*}s_{2}(i)\\})\simeq{\mathcal{O}}/(U_{i},f_{1}^{*}s_{2}(i));$

  * •

$\underline{\operatorname{Hom}}(\\{C_{2,j+1}^{-1}\cdot
U_{j+1},C_{2,j}\\},\\{C_{2,j}\cdot
U_{j+1},C_{2,j+1}^{-1}\\})\simeq{\mathcal{O}}/(C_{2,j+1}^{-1},C_{2,j})[-1].$

Hence, we get an isomorphism on $(R_{ij}\cap
R_{*,j+1})\times{\mathbb{A}}^{k}$,

$\underline{\operatorname{Hom}}(Q_{j+1},P_{ij})\simeq{\mathcal{O}}/(U_{r}\ |\
r\neq j+1)\otimes
f_{1}^{*}p_{i}^{*}{\mathcal{O}}(-1)[-1]={\mathcal{O}}_{(R_{ij}\cap
R_{*,j+1})\times{\mathbb{A}}^{1}_{U_{j+1}}}\otimes
f_{1}^{*}p_{i}^{*}{\mathcal{O}}(-1)[1],$ (5.3)

which has vanishing cohomology.

In the case $i=j+1$ we can use

$\ldots,V_{j+3},f_{1}^{*}s_{1}(j+1),s_{j+1},V_{j},\ldots$

as equations of $R_{j+1,j}$ and

$\ldots,V_{j+3},V_{j+2},f_{2}^{*}p_{j}^{*}z_{1},V_{j},\ldots$

as equations of $Q_{*,j+1}$, so
$\underline{\operatorname{Hom}}(Q_{j+1},P_{j+1,j})\otimes
f_{1}^{*}p_{i}{\mathcal{O}}(1)$ is the tensor product of

  * •

$\underline{\operatorname{End}}(\\{U_{r},V_{r}\\})\simeq{\mathcal{O}}/(U_{r},V_{r})$,
for $r\neq j+1,j+2$;

  * •

$\underline{\operatorname{Hom}}(\\{U_{j+2},V_{j+2}\\},\\{f_{1}^{*}p_{j+1}^{*}z_{1}\cdot
U_{j+2},f_{1}^{*}s_{1}(j+1)\\})\simeq{\mathcal{O}}/(U_{j+2},f_{1}^{*}s_{1}(j+1));$

  * •

$\underline{\operatorname{Hom}}(\\{s_{j+1}f_{1}^{*}p_{j+1}^{*}z_{0}\cdot
U_{j+1},f_{2}^{*}p_{j}^{*}z_{1}\\},\\{f_{1}^{*}p_{j+1}^{*}z_{0}f_{2}^{*}p_{j}^{*}z_{1}\cdot
U_{j+1},s_{j+1}\\})\simeq{\mathcal{O}}/(f_{2}^{*}p_{j}^{*}z_{1},s_{j+1})\otimes
f_{2}^{*}p_{j}^{*}{\mathcal{O}}(1)[-1].$

Taking into account the trivialization of
$f_{2}^{*}p_{j}^{*}{\mathcal{O}}(1)$, we see that isomorphism (5.3) still
holds, and so the $\operatorname{Hom}$-space vanishes.

Case $r=j-1$, $\operatorname{Hom}(Q_{j-1},P_{ij})$. We work locally near
$(R_{ij}\cap R_{*,j-1})\times{\mathbb{A}}^{k}$, so we can use

$\ldots,V_{i+2},f_{1}^{*}s_{1}(i),f_{1}^{*}s_{2}(i),V_{i-1},\ldots,V_{j+1},C_{2,j-1},V_{j-1},\ldots$

as equations of $R_{ij}$ (even in the case $i=j+1$), and

$\ldots,V_{j+1},C_{2,j}^{-1},V_{j-1},\ldots$

as equations of $R_{*,j-1}$. Thus, we can compute
$\underline{\operatorname{Hom}}(Q_{j-1},P_{ij})\otimes
f_{1}^{*}p_{i}{\mathcal{O}}(1)$ as the tensor product of

  * •

$\underline{\operatorname{End}}(\\{U_{r},V_{r}\\})\simeq{\mathcal{O}}/(U_{r},V_{r})$,
for $r\neq i,i+1,j$;

  * •

$\underline{\operatorname{Hom}}(\\{U_{i+1},V_{i+1}\\},\\{f_{1}^{*}p_{i}^{*}z_{1}\cdot
U_{i+1},f_{1}^{*}s_{1}(i)\\})\simeq{\mathcal{O}}/(U_{i+1},f_{1}^{*}s_{1}(i))$;

  * •

$\underline{\operatorname{Hom}}(\\{U_{i},V_{i}\\},\\{f_{1}^{*}p_{i}^{*}z_{0}\cdot
U_{i},f_{1}^{*}s_{2}(i)\\})\simeq{\mathcal{O}}/(U_{i},f_{1}^{*}s_{2}(i));$

  * •

$\underline{\operatorname{Hom}}(\\{C_{2,j-1}\cdot
U_{j},C_{2,j}^{-1}\\},\\{C_{2,j}^{-1}\cdot
U_{j},C_{2,j-1}\\})\simeq{\mathcal{O}}/(C_{2,j}^{-1},C_{2,j-1})[-1].$

This leads to

$\underline{\operatorname{Hom}}(Q_{j-1},P_{ij})\simeq{\mathcal{O}}_{(R_{ij}\cap
R_{*,j-1})\times{\mathbb{A}}^{1}_{U_{j}}}\otimes
f_{1}^{*}p_{i}^{*}{\mathcal{O}}(-1)[1],$

which has vanishing cohomology. ∎

######  Lemma 5.3.8.

(i) For $j<k$, the space $\operatorname{Hom}(Q_{j},P_{kj})$ is freely
generated over $\operatorname{End}(Q_{j},P_{kj})$ by the generator
$\delta_{j}$ of degree $0$ that corresponds to the morphism

${\mathcal{O}}_{R_{*j}}\to{\mathcal{O}}_{R_{kj}}.$

On the other hand, the space $\operatorname{Hom}(P_{kj},Q_{j})$ is freely
generated over $\operatorname{End}(Q_{j},P_{kj})$ by the generator
$\gamma_{j}$ of degree $0$ the corresponds to the morphism

${\mathcal{O}}_{R_{kj}}\rTo{X_{1}\cdot?}{\mathcal{O}}_{R_{*j}}.$

We have $\gamma_{j}\delta_{j}=\delta_{j}\gamma_{j}=X_{1}$.

(ii) For $j\leq k$, the space $\operatorname{Hom}(Q_{j},P_{k,j-1})$ (resp.,
$\operatorname{Hom}(P_{k,j-1},Q_{j})$) is freely generated over
$\operatorname{End}(Q_{j},P_{k,j-1})$ by $\delta_{j-1}\alpha_{j-1}$ (resp.,
$\beta_{j-1}\gamma_{j-1}$), where we use generators from Lemma 5.3.6 and from
part (i). For $j<k-1$ the space $\operatorname{Hom}(Q_{j},P_{k,j+1})$ (resp.,
$\operatorname{Hom}(P_{k,j+1},Q_{j})$) is freely generated over
$\operatorname{End}(Q_{j},P_{0,j+1}$) by $\delta_{j+1}\beta_{j}$ (resp.,
$\alpha_{j}\gamma_{j+1}$).

(iii) For $j<k$, we have $\delta_{j-1}\alpha_{j-1}=\alpha_{k,j-1}\delta_{j}$
in $\operatorname{Hom}(Q_{j},P_{k,j-1})$ (resp.,
$\beta_{j-1}\gamma_{j-1}=\gamma_{j}\beta_{k,j-1}$ in
$\operatorname{Hom}(P_{k,j-1},Q_{j})$). We also have
$\delta_{j}\beta_{j-1}=\beta_{k,j-1}\delta_{j-1}$ in
$\operatorname{Hom}(Q_{j-1},P_{kj})$ (resp.,
$\alpha_{j-1}\gamma_{j}=\gamma_{j-1}\alpha_{k,j-1}$ in
$\operatorname{Hom}(P_{kj},Q_{j-1})$).

###### Proof.

(i) As in Lemma 5.3.7, we get an isomorphism on
$\overline{R}_{kj}\times{\mathbb{A}}^{k}$,

$\underline{\operatorname{Hom}}(Q_{j},P_{kj})\simeq{\mathcal{O}}_{p^{-1}\overline{R}_{kj}}/(U_{k}).$

Now the locus $U_{k}=0$ in $p^{-1}\overline{R}_{kj}$ is the product
${\mathbb{A}}^{1}_{X_{1}}\times C_{2}(U_{j},U_{j+1})$ for $j>0$, and
${\mathbb{A}}^{1}_{X_{1}}\times C_{2}(U_{1})$ for $j=0$. The space of global
functions on this locus coincides with $\operatorname{End}(Q_{j},P_{kj})$.

Assume now that $j<k-1$. Then as before, we get an isomorphism on
$\overline{R}_{*j}\times{\mathbb{A}}^{k}$,

$\displaystyle\underline{\operatorname{Hom}}(P_{kj},Q_{j})\simeq{\mathcal{O}}/(U_{r}\
|\ r\neq k,j,j+1)\otimes{\mathcal{O}}/(U_{k},C_{1,k-1})\otimes$
$\displaystyle{\mathcal{O}}/(f_{2}^{*}p_{j}^{*}z_{1}\cdot
U_{j+1},f_{2}^{*}p_{j}^{*}z_{0}\cdot
U_{j})\simeq{\mathcal{O}}_{p^{-1}\overline{R}_{kj}}/(U_{k}),$

which is the same sheaf as before.

In the case $j=k-1$, the sheaf
$\underline{\operatorname{Hom}}(P_{k,k-1},Q_{k-1})$ is the tensor product of

  * •

$\underline{\operatorname{End}}(\\{U_{r},V_{r}\\})\simeq{\mathcal{O}}/(U_{r},V_{r})$,
for $r<k-1$;

  * •

$\underline{\operatorname{Hom}}(\\{X_{1}f_{2}^{*}p_{k-1}^{*}z_{1}\cdot
U_{k},s_{k}\\},\\{f_{2}^{*}p_{k-1}^{*}z_{1}\cdot
U_{k},f_{2}^{*}s_{k-1}(1)\\})\simeq{\mathcal{O}}/(f_{2}^{*}p_{k-1}^{*}z_{1}\cdot
U_{k},s_{1})$;

  * •

$\underline{\operatorname{End}}(\\{f_{2}^{*}p_{k-1}^{*}z_{0}\cdot
U_{k-1},f_{2}^{*}s_{2}(k-1)\\})\simeq{\mathcal{O}}/(f_{2}^{*}p_{k-1}^{*}z_{0}\cdot
U_{k-1},f_{2}^{*}s_{2}(k-1))$,

so we get an isomorphism on $R_{k,k-1}\times{\mathbb{A}}^{k}$,

$\underline{\operatorname{Hom}}(P_{k,k-1},Q_{k-1})\simeq{\mathcal{O}}/((U_{r}\
|\ r<k-1),f_{2}^{*}p_{k-1}^{*}z_{1}\cdot U_{k},f_{2}^{*}p_{k-1}^{*}z_{0}\cdot
U_{k-1})={\mathcal{O}}_{Z},$

where $Z\simeq{\mathbb{A}}^{1}_{X_{1}}\times C_{2}(U_{k-1},U_{k})$. Hence,

$\operatorname{Hom}(P_{k,k-1},Q_{k-1})\simeq\mathbf{k}[X_{1},U_{k-1},U_{k}]/(U_{k-1}U_{k})=\operatorname{End}(P_{k,k-1},Q_{k-1}).$

(ii) Similarly to Lemma 5.3.7 we get the following local isomorphisms

$\underline{\operatorname{Hom}}(Q_{j},P_{k,j-1})\simeq\underline{\operatorname{Hom}}(P_{k,j-1},Q_{j})\simeq{\mathcal{O}}_{(R_{k,j-1}\cap
R_{*j})\times{\mathbb{A}}^{1}_{U_{j}}}[-1],$
$\underline{\operatorname{Hom}}(Q_{j},P_{k,j+1})\simeq\underline{\operatorname{Hom}}(P_{k,j+1},Q_{j})\simeq{\mathcal{O}}_{(R_{k,j+1}\cap
R_{*j})\times{\mathbb{A}}^{1}_{U_{j+1}}}[-1].$

Chasing the generators from Lemma 2.3.6 we get the assertion.

(iii) The relation $\delta_{j-1}\alpha_{j-1}=\alpha_{k,j-1}\delta_{j}$ comes
from the morphism of exact triangles The other relations are proved similarly.
∎

####  5.3.5. Recovering the multigrading; shifted generators

Recall that we are interested in the ${\mathbb{L}}_{B}$-graded category of
$\widetilde{T}$-equivariant matrix factorizations

${\mathcal{B}}_{2,k}:=\operatorname{MF}_{\widetilde{T}}(\overline{{\mathcal{X}}}_{2,k},{\mathbf{w}}_{2,k}),$

and our generating matrix factorizations $P_{ij}$ and $Q_{j}$ have natural
lifts to this category (recall that we equip
$f_{1}^{*}p_{i}^{*}{\mathcal{O}}(1)$ with the $T$-equivariant structure, so
that $f_{1}^{*}p_{i}^{*}z_{0}$ has weight $0$).

The group $\widetilde{H}$ of characters of the torus $\widetilde{T}$ is a free
abelian group with generators $x_{1},x_{2},u_{1},\ldots,u_{k},l_{0}$. Hence,
the grading group ${\mathbb{L}}_{B}=2\widetilde{H}+\mathbb{Z}l_{0}$ is free on
generators $2x_{1},2x_{2},2u_{1},\ldots,2u_{k},l_{0}$. So far, we computed
morphisms in the $\mathbb{Z}$-graded category obtained from
${\mathcal{B}}_{2,k}$ by collapsing the ${\mathbb{L}}_{B}$-grading via the
homomorphism $f:{\mathbb{L}}_{B}\to\mathbb{Z}$ given by

$f(2x_{1})=f(2x_{2})=f(2u_{i})=0,f(l_{0})=1.$

for which we have $f(2v_{i})=2$, $f(2c_{1i})=f(2c_{2i})=2(k-i)$, and
$f(2x_{3})=2k$.

Now for all morphisms between the generators $(P_{ij},Q_{j})$ in this
$\mathbb{Z}$-graded category we can identify morphisms in
${\mathcal{B}}_{2,k}$ that they come from, and in this way we will determine
completely morphisms between our generators in ${\mathcal{B}}_{2,k}$.

We start with endomorphisms of $P_{ij}$ and $Q_{j}$. Recall that all of them
are generated by the multiplication with functions $X_{1},X_{2}$ and $U_{i}$.
Hence, we lift them to the same endomorphisms in ${\mathcal{B}}_{2,k}$, and
they acquire the natural (doubled) grading, so $X_{i}$ will have the grading
$2x_{i}$ and $U_{i}$ will have the grading $2u_{i}$.

The exact triangles in Lemmas 5.3.5, 5.3.8 and 5.3.6 show that the generators
of morphisms between $(P_{ij})$ have the following lifts:

$\alpha_{ij}\in\operatorname{Hom}^{l_{0}-2c_{2,j}}(P_{i,j+1},P_{i,j}),\
\alpha_{j}\in\operatorname{Hom}^{l_{0}-2c_{2,j}}(Q_{j+1},Q_{j}),$
$\gamma_{ij}\in\operatorname{Hom}^{l_{0}-2c_{1,i}+2c_{1,i+1}}(P_{i+1,j},P_{i,j})\
\text{ for }i<k,\
\gamma_{kj}\in\operatorname{Hom}^{l_{0}-2c_{1,k-1}}(P_{k,j},P_{k-1,j})$
$\delta_{j}\in\operatorname{Hom}^{0}(Q_{j},P_{kj}).$

The gradings of the remaining generators are determined by the relations and
by the gradings of the endomorphisms.

Now let us define the following shifted generators in ${\mathcal{B}}_{2,k}$:

$\widetilde{Q}_{j}:=Q_{j}[k-j]\otimes(-2c_{2,j}-2c_{2,j+1}-\ldots-2c_{2,k-1}),$
$\widetilde{P}_{ij}:=P_{ij}[2k-i-j]\otimes(-2c_{1,i}-2c_{2,j}-2c_{2,j+1}-\ldots-2c_{2,k-1}).$

Then the morphisms $\alpha_{ij}$, $\alpha_{j}$, $\gamma_{ij}$ and $\delta_{j}$
induce morphisms of degree $0$ between these shifted generators.

######  Theorem 5.3.9.

The ${\mathbb{L}}_{B}$-graded $A_{\infty}$-algebra of endomorphisms of
$\bigoplus\widetilde{Q}_{j}\oplus\bigoplus\widetilde{P}_{ij}$ is formal. The
${\mathbb{L}}_{B}$-grading on its cohomology is uniquely determined by the
condition that the morphisms $\alpha_{ij}$, $\alpha_{j}$, $\gamma_{ij}$ and
$\delta_{j}$ have degree $0$, while $X_{r}$ (resp., $U_{r}$), viewed as an
endomorphism of $\widetilde{Q}_{j}$ or $\widetilde{P}_{ij}$, has degree
$2x_{r}\in{\mathbb{L}}_{B}$ (resp., $2u_{r}\in{\mathbb{L}}_{B}$).

###### Proof.

Let us denote by $S$ a multiplicatively closed subset of arrows in
$\operatorname{End}(\bigoplus\widetilde{Q}_{j}\oplus\bigoplus\widetilde{P}_{ij})$
(viewed as a quiver with relations), which includes all arrows $X_{r}$ and
$U_{r}$ (forming a loop at some vertex), as well as the arrows $\alpha_{ij}$,
$\alpha_{j}$, $\gamma_{ij}$ and $\delta_{j}$. The computations in Sec. 5.3.1,
5.3.2, 5.3.4, 5.3.3 show that for every other generator $t$ of the
$\operatorname{End}$-algebra, there exists an element $s\in S$ such that $st$
is a nonzero element of $S$. Hence, the grading of the entire
$\operatorname{End}$-algebra is uniquely determined by the grading of the
elements of $S$.

The formality statement follows from Lemma 2.1.3, due to the fact that with
the respect to the $\mathbb{Z}$-grading induced by
$f:{\mathbb{L}}_{B}\to\mathbb{Z}$, the subset $S$ has degree $0$. ∎

###  5.4. Localization for $n=2$

As in Section 4.2, we denote by ${\mathcal{B}}_{2,k}^{\mathbb{Z}}$ (resp.,
$\overline{{\mathcal{B}}}_{2,k}^{\mathbb{Z}}$) the category of
$\mathbb{Z}$-graded matrix factorizations on ${\mathbf{w}}_{2,k}$ on
${\mathcal{X}}_{2,k}$ (resp., $\overline{{\mathcal{X}}}_{2,k}$), corresponding
to the grading

$|U_{i}|=2,\ |X_{i}|=|C_{ij}|=|V_{i}|=0.$

We have equivalences

${\mathcal{B}}_{2,k}^{\mathbb{Z}}\simeq
D^{b}\operatorname{Coh}({\mathcal{Z}}_{2,k}),\ \
\overline{{\mathcal{B}}}_{2,k}^{\mathbb{Z}}\simeq
D^{b}\operatorname{Coh}(\overline{{\mathcal{Z}}}_{2,k}),$

and we can identify $D^{b}\operatorname{Coh}({\mathcal{Z}}_{2,k})$ with the
quotient of $\overline{{\mathcal{B}}}_{2,j}^{\mathbb{Z}}$ by the subcategory
of matrix factorizations supported on the boundary of
$\overline{{\mathcal{Z}}}_{2,k}$,

$D:=f_{2}^{-1}(p)=\operatorname{crit}(w)\cap(\overline{{\mathcal{Y}}}_{2,k}\setminus{\mathcal{Y}}_{2,k}),$

where $p$ is the infinite point on
$\overline{R}_{0}={\mathbb{P}}^{1}\subset\overline{{\mathcal{Y}}}_{1,k}$.

Recall that $D$ is a nodal chain,

$D=D^{\prime}_{0}\cup D_{0}\cup D_{1}\cup\ldots\cup D_{k},$

where $D_{i}=\overline{R}_{i0}\setminus R_{i0}$ for $i>0$ and
$D^{\prime}_{0}\cup D_{0}=\overline{R}_{00}\setminus R_{00}$. Thus, by Lemma
4.1.4, our subcategory is generated by the matrix factorizations corresponding
to the structure sheaves of the components $D_{0},\ldots,D_{k}$, as well as
the structure sheaf of $D$. Actually, the same proof shows that the objects

${\mathcal{O}}_{D},{\mathcal{O}}_{D_{0}}(-1),\ldots,{\mathcal{O}}_{D_{k}}(-1)$

generate $D^{b}\operatorname{Coh}(D)$.

Similarly to Lemma 4.2.1 we can find the resolutions for these objects. Let us
set for brevity
${\mathcal{O}}(H_{i}^{v}):=f_{1}^{*}p_{i}^{*}{\mathcal{O}}(1)$,
${\mathcal{O}}(H_{i}^{h}):=f_{2}^{*}p_{i}^{*}{\mathcal{O}}(1)$ (here $v$
stands for “vertical”, $h$ stands for “horizontal”). Let us set
$\overline{R}_{i,\leq j}:=\cup_{j^{\prime}\leq j}\overline{R}_{ij^{\prime}}$,
$\overline{R}_{i*}:=\cup_{j}\overline{R}_{ij}$ $\overline{R}_{\leq
i,j}:=\cup_{i^{\prime}\leq i}\overline{R}_{i^{\prime}j}$, and consider the
objects, for $i<j$,

$P_{i,\leq j}:={\mathcal{O}}_{\overline{R}_{i,\leq
j}}(H_{j-1}^{h}+\ldots+H_{0}^{h}-H_{i}^{v}),$ $P_{i,\leq
i}:={\mathcal{O}}_{\overline{R}_{i*}}(H_{i}^{h}+\ldots+H_{0}^{h}-H_{i}^{v}),$
$P^{\prime}_{i,\leq j}:={\mathcal{O}}_{\overline{R}_{i,\leq
j}}(H_{j}^{h}+\ldots+H_{1}^{h}-H_{i}^{v}),$ $P^{\prime}_{i,\leq
i}:={\mathcal{O}}_{\overline{R}_{i*}}(H_{i}^{h}+\ldots+H_{1}^{h}-H_{i}^{v}),$
$P_{\leq i,j}:={\mathcal{O}}_{\overline{R}_{\leq i,j}}(-H_{i}^{v}),\ \
P_{ii}=P_{\leq i,i}:={\mathcal{O}}_{\overline{R}_{ii}}(H_{i}^{h}-H_{i}^{v}),$
$P^{\prime}_{\leq i,j}:={\mathcal{O}}_{\overline{R}_{\leq i,j}},\ \
P^{\prime}_{ii}=P^{\prime}_{\leq i,i}:={\mathcal{O}}_{\overline{R}_{ii}}.$

Note that for $i>0$,

$P_{i,\leq 0}=P^{\prime}_{i,\leq 0}=P_{i0},\ \ P_{\leq k,j}=P^{\prime}_{\leq
k,j}=Q_{j}.$

It is also convenient to set $P^{\prime}_{ij}=P_{ij}$ for $i>j$.

######  Lemma 5.4.1.

(i) We have the following exact triangles, for $i>j$,

$P_{ij}[-1]\to P_{\leq i-1,j}\to P_{\leq i,j}\to\ldots,$ $P^{\prime}_{\leq
i,j}\to P^{\prime}_{\leq i-1,j}\to P_{ij}[1]\to\ldots,$

and for $i\geq j$,

$P_{i,\leq j-1}\to P_{i,\leq j}\to P_{ij}\to\ldots,$ $P^{\prime}_{ij}\to
P^{\prime}_{i,\leq j}\to P^{\prime}_{i,\leq j-1}\to\ldots,$

which recursively express the objects $(P_{\leq i,j})$, $(P_{i,\leq j})$ in
terms of the generators. Finally, we have the exact triangles, for $i>0$,

$P^{\prime}_{i,\leq i}\to P_{i,\leq i}\to{\mathcal{O}}_{D_{i}}(-1),$

and

$P^{\prime}_{00}\to P_{00}\to{\mathcal{O}}_{D_{0}}(-1).$

(ii) One has ${\mathcal{O}}_{D}\simeq f_{2}^{*}{\mathcal{O}}_{p}$,
$Q_{j}=f_{2}^{*}{\mathcal{O}}_{R_{j}}$, so the pull-back of the exact
triangles of Lemma 4.2.1 under $f_{2}$ give a resolution for
${\mathcal{O}}_{D}$.

###### Proof.

These are pretty straightforward. The only nontrivial point is to use the
following relations between divisors on $R_{ii}$ (and $\overline{R}_{00}$),
which is isomorphic to the blow up of a point on
${\mathbb{A}}^{1}\times{\mathbb{P}}^{1}$. Let us set $E_{i}^{v}:=R_{ii}\cap
R_{i+1,i}$, $E_{i}^{h}:=R_{ii}\cap R_{i,i-1}$. Then one has

${\mathcal{O}}_{R_{ii}}(E_{i}^{v})\simeq{\mathcal{O}}_{R_{ii}}(-E_{i}^{h})\simeq{\mathcal{O}}_{R_{ii}}(H_{i}^{v}-H_{i}^{h}).$

∎

##  6\. Matching the A-side with the B-side

###  6.1. Equivalence of the Fukaya category with one stop with the
compactified LG-model

Here we prove Theorem 1.4.2.

First, we observe that we can match the grading groups. Namely, for this we
use canonical decompositions

${\mathbb{L}}=\mathbb{Z}\oplus H,$ ${\mathbb{L}}_{B}=\mathbb{Z}\cdot
l_{0}\oplus 2H,$

where $H=H_{1}(\Sigma)$ (see Sec. 3.1). The isomorphism
${\mathbb{L}}\to{\mathbb{L}}_{B}$ sends $(n,h)$ to $nl_{0}+2h$. Recall that by
definition the homomorphism $\sigma:{\mathbb{L}}\to\mathbb{Z}/2$ sends $H$ to
$0$, hence our isomorphism is compatible with the homomorphisms to
$\mathbb{Z}/2$.

In the case $n=1$, we consider the correspondence between the generating
objects

$P_{i}\leftrightarrow L_{i},\ i=0,\ldots,k.$

It is easy to see that it extends to the isomorphism between the corresponding
(cohomology) endomorphism algebras, so that endomorphisms $U_{i}$ and $X_{1}$
on the B-side correspond to $u_{i}$ and $x_{1}$ on the $A$-side, and
$a_{i}:P_{i}\to P_{i-1}$ (resp., $b_{i}:P_{i-1}\to P_{i}$) corresponds to
$r_{i}:L_{i}\to L_{i-1}$ (resp. $\ell_{i}:L_{i-1}\to L_{i}$). It is easy to
see that there is a unique (up to common shift by an element of
${\mathbb{L}}$) choice of gradings of $L_{i}$, so that the gradings match.
Since both endomorphism algebras are formal, the equivalence follows.

In the case $n=2$, we consider the following correspondence between the
generating objects

$\begin{array}[]{l}P_{ij}\leftrightarrow L_{i}\times L_{j},\ \text{ for }0\leq
j<i\leq k,\\\ Q_{j}\leftrightarrow L_{j}\times L_{k+1},\ \text{ for }0\leq
j\leq k.\end{array}$

A straightforward check shows that the morphisms on both sides match, where
the generators $u_{i}$ (resp., $x_{1},x_{2}$) on the A-side correspond to
$U_{i}$ (resp., $X_{1},X_{2}$) on the B-side.

By [14, Thm. 3.2.5(iii)], up to shifts of the generators, the grading on the
A-side is uniquely determined by the degrees of $u_{r}$ and $x_{r}$. Hence, by
choosing appropriate shifts, we can make it match with the grading on the
B-side. Again both endomorphism algebras are formal, so the equivalence
follows.

###  6.2. Matching the localizations

Now we can deduce Theorem 1.2.2 by localization.

####  6.2.1. Case $n=1$

We have to match the resolution (3.2) for the object $T$ in the Fukaya
category with the resolution for ${\mathcal{O}}_{p}$ obtained in Lemma 4.2.1.
The exact triangles of that Lemma show recursively the following
correspondence between the A-side and the B-side:

${\mathcal{O}}_{R_{[0,i]}}(1,\ldots,1,0)\leftrightarrow[L_{i}\rTo{r_{i}}\ldots\rTo{r_{1}}L_{0}],\
i<k,$
${\mathcal{O}}_{R_{[0,k]}}(1,1,\ldots,1)\leftrightarrow[L_{k}\rTo{r_{k}}\ldots\rTo{r_{1}}L_{0}],$
${\mathcal{O}}_{R_{[0,i]}}(0,1,\ldots,1)\leftrightarrow[L_{0}\rTo{\ell_{1}}\ldots\rTo{\ell_{i}}L_{i}],\
i\leq k.$

Finally, we can identify the morphism

${\mathcal{O}}_{R_{0,k}}(0,1,\ldots,1)\to{\mathcal{O}}_{R_{[0,k]}}(1,\ldots,1)$

in the resolution for ${\mathcal{O}}_{p}$ with the degree $0$ morphism of
twisted complexes

$[L_{0}\to\ldots\to L_{k}]\to[L_{k}\to\ldots\to L_{0}]$

given by $x_{1}:L_{k}\to L_{k}$. The cone of this morphism is exactly the
resolution of the arc $T$, which therefore corresponds to ${\mathcal{O}}_{p}$.

####  6.2.2. Case $n=2$

We will match the resolutions of $L_{i}\times T$ obtained from (3.3) with the
resolutions in Lemma 5.4.1. The exact triangles of that Lemma show recursively
the following correspondences for $i\leq k$:

  * •

for $j=0,\ldots,i-1$, $P_{i,\leq j}\leftrightarrow[L_{i,j}\to\ldots\to
L_{i,0}]$,

  * •

for $r=k-1,\ldots,i$, $P_{\leq r,i}\leftrightarrow[L_{k+1,i}\to
L_{k,i}[-1]\to\ldots\to L_{r+1,i}[-1]]$,

  * •

for $j=0,\ldots,i-1$, $P^{\prime}_{i,\leq
j}\leftrightarrow[L_{i,0}\to\ldots\to L_{i,j}]$,

  * •

for $r=k-1,\ldots,i$, $P^{\prime}_{\leq
r,i}\leftrightarrow[L_{r+1,i}[1]\to\ldots\to L_{k,i}[1]\to L_{k+1,i}]$.

Next, using the exact triangles for $P_{i,\leq i}$ and $P^{\prime}_{i,\leq i}$
we sew together the above complexes and get the correspondences

$P_{i,\leq i}\leftrightarrow[L_{k+1,i}\to L_{k,i}[-1]\to\ldots\to
L_{i+1,i}[-1]\to L_{i,i-1}\to\ldots\to L_{0}],$ $P^{\prime}_{i,\leq
i}\leftrightarrow[L_{i,0}\to\ldots\to L_{i,i-1}\to L_{i+1,i}[1]\to\ldots\to
L_{k,i}[1]\to L_{k+1,i}].$

Finally, using the exact triangle for ${\mathcal{O}}_{D_{i}}(-1)$, we see that
it corresponds to the complex (3.3) representing $T\times L_{i}$.

For $i=k+1$, similarly to the case $n=1$, we check that the complex (3.4)
corresponds to the resolution for ${\mathcal{O}}_{D}$ in Lemma 5.4.1(ii).

## References

  * [1] Abouzaid, M., Auroux, D., Katzarkov, L., Lagrangian fibrations on blowups of toric varieties and mirror symmetry for hypersurfaces, Publ. Math. Inst. Hautes Études Sci. 123 (2016), 199–282.
  * [2] Auroux, D., Fukaya categories of symmetric products and bordered Heegaard-Floer homology, J. Gökova Geom. Topol. GGT 4 (2010), 1–54.
  * [3] Auroux, D., Speculations on homological mirror symmetry for hypersurfaces in $(\mathbb{C}^{*})^{n}$, Surveys in Differential Geometry, Vol. 22, Intl. Press, 2018, 1–47.
  * [4] Fukaya, K., Oh, Y.G., Ohta, H., Ono, K. Anchored Lagrangian submanifolds and their Floer theory, In: Mirror symmetry and tropical geometry,Contemp. Math., vol. 527, pp. 15–54. Amer. Math. Soc.,Providence, RI (2010). DOI 10.1090/conm/527/10399
  * [5] Gelfand, I. M., Kapranov, M. M., Zelevinsky, A. V., Discriminants, resultants, and multidimensional determinants, Birkhäuser, Boston MA, 1994.
  * [6] Gross, M., Hacking, P., and Keel, S., Mirror symmetry for log Calabi-Yau surfaces I, Publ. Math. Inst. Hautes Études Sci.122(2015), 65–168.
  * [7] Isik, M. U. Equivalence of the derived category of a variety with a singularity category. Int. Math. Res. Not. IMRN 2013, no. 12, 2787–2808.
  * [8] Keating, A. Homological mirror symmetry for hypersurface cusp singularities Selecta Mathematica 24, 1411–1452 (2018).
  * [9] Hacking, P., Keating, A. Homological mirror symmetry for log Calabi–Yau surfaces, arXiv:2005.05010.
  * [10] Lipshitz, R., Ozsváth P., Thurston D., Bordered Heegaard-Floer homology. Mem. Amer. Math. Soc. 254 (2018), no. 1216, viii+279 pp.
  * [11] Lekili Y., Perutz T., Arithmetic mirror symmetry for the 2-torus arXiv:1211.4632, unpublished.
  * [12] Lekili, Y., Polishchuk, A., Auslander orders over nodal stacky curves and partially wrapped Fukaya categories, Journal of Topology 11 (2018), no. 3, 615–644.
  * [13] Lekili Y., Polishchuk A., Derived equivalences of gentle algebras via Fukaya categories, Math. Ann. 376 (2020), no. 1-2, 187–225.
  * [14] Lekili Y., Polishchuk A., Homological mirror symmetry for higher dimensional pairs of pants, Compos. Math. 156 (2020), no. 7, 1310–1347.
  * [15] Lin K. H., Pomerleano D., Global matrix factorizations, Math. Res. Lett. 20 (2013), no. 1, 91–106.
  * [16] Perutz, T., Hamiltonian handleslides for Heegaard Floer homology Proceedings of Gökova Geometry-Topology Conference 2007, 15–35, Gökova Geometry/Topology Conference (GGT), Gökova, 2008.
  * [17] Polishchuk, A., Vaintrob, A. Matrix factorizations and singularity categories for stacks, Ann. Inst. Fourier (Grenoble) 61 (2011), no. 7, 2609–2642.
  * [18] Polishchuk, A., Vaintrob, A. Matrix factorizations and Cohomological Field Theories, J. Reine Angew. Math. 714 (2016), 1–122.
  * [19] Segal, E. Equivalence between GIT quotients of Landau-Ginzburg B-models, Comm. Math. Phys. 304 (2011), no. 2, 411–432.
  * [20] Seidel P., Graded Lagrangian submanifolds, Bull. Soc. Math. France 128 (2000), no. 1, 103–149.
  * [21] Seidel P. Fukaya categories and deformations. Proceedings of the International Congress of Mathematicians, Vol. II (Beijing, 2002), 351–360, Higher Ed. Press, Beijing, 2002.
  * [22] Seidel, P.,  Fukaya categories and Picard-Lefschetz theory.  Zurich Lectures in Advanced Mathematics. European Mathematical Society (EMS), Zürich, 2008.
  * [23] Seidel, P., Homological Mirror Symmetry for the Genus Two Curve. Journal of Algebraic Geometry 20.4 (2011)
  * [24] Sheridan, N. Homological mirror symmetry for Calabi-Yau hypersurfaces in projective space, Invent. Math. 199 (2015), no. 1, 1–186.
  * [25] Sheridan, N. Versality of the relative Fukaya category Geom. Topol. 24 (2020), no. 2, 747–884.


# Flexible Networks for Learning Physical Dynamics of Deformable Objects

Jinhyung Park, 1 Dohae Lee, 1 In-Kwon Lee 1

###### Abstract

Learning the physical dynamics of deformable objects with particle-based
representation has been the objective of many computational models in machine
learning. While several state-of-the-art models have achieved this objective
in simulated environments, most existing models impose a precondition, such
that the input is a sequence of ordered point sets. That is, the order of the
points in each point set must be the same across the entire input sequence.
This precondition restrains the model from generalizing to real-world data,
which is considered to be a sequence of unordered point sets. In this paper,
we propose a model named _time-wise PointNet_ (TP-Net) that solves this
problem by directly consuming a sequence of unordered point sets to infer the
future state of a deformable object with particle-based representation. Our
model consists of a shared feature extractor that extracts global features
from each input point set in parallel and a prediction network that aggregates
and reasons on these features for future prediction. The key concept of our
approach is that we use global features rather than local features to achieve
invariance to input permutations and ensure the stability and scalability of
our model. Experiments demonstrate that our model achieves state-of-the-art
performance with real-time prediction speed in both synthetic dataset and
real-world dataset. In addition, we provide quantitative and qualitative
analysis on why our approach is more effective and efficient than existing
approaches.

##  1 Introduction

Humans have the ability to reason on historical trajectories of various
objects and easily make inferences on future states based on their intuitions.
In order to mimic such ability, developing a computational model such as
differentiable physics engine that can learn the physical dynamics of objects
has been a core domain in computer vision, robotics, and in machine learning
(Degrave et al. 2019; Collins et al. 2021; de Avila Belbute-Peres et al. 2018;
Freeman et al. 2021). To implement such computational models, physical
reasoning on objects with particle-based representations has garnered
attention in recent years (Macklin et al. 2014). As particle-based
representations provide flexibility in representing objects with complex
shapes and high degree of freedom, several studies have provided frameworks
that can learn to simulate the physical dynamics of deformable objects with
particle-based representation (Mrowca et al. 2018; Li et al. 2019b, 2020).

(a)

(b)

Figure 1: (a) We can easily configure a sequence of ordered point sets in
simulated environments. (b) The previous condition is seldom met in many real-
world scenarios. Figure 2: TP-Net Architecture. The network takes a sequence
of $m$ unordered point sets as input, and outputs the predicted unordered
point set in the next time step. “MLP”, “FC” and “B-LSTM” stands for multi-
layer perceptron, fully-connected layer and bidirectional-LSTM layer,
respectively. The input transform net and feature transform net is directly
adopted from the existing classification network – PointNet.

While these models have achieved state-of-the-art performances in predicting
the future states of deformable objects, these models are optimized from
learning _a sequence of ordered point sets_. In other words, the order of the
points in each point set must be the same across the entire input sequence
when training or testing the model. One of the reasons for imposing this
precondition in the data is to use a weight-shared LSTM (Hochreiter and
Schmidhuber 1997) layer in the model architecture or to easily reason on the
historical trajectories of individual points. However, this approach is only
feasible in simulated environments where we can map each point in the current
frame to its corresponding point in the previous frame (Fig. 1 (a)). In many
real-world scenarios, individual point correspondences between different
frames are not observable, thus making it challenging to configure a sequence
of ordered point sets from the given input (Fig. 1 (b)). Although sorting the
points in each point set into a canonical order might seem to resolve this
problem, PointNet (Qi et al. 2017) demonstrates that in high-dimensional
space, there exists no ordering that is stable with respect to point
perturbations. Therefore, sorting-based approaches also result in lower
performance which we will show this empirically in Section IV.

In order to build a flexible neural network for constructing a differentiable
physics engine that is independent of this concern, we propose our model named
_time-wise PointNet_ (TP-Net) that can directly consume a sequence of
unordered point sets to infer the future dynamics of a deformable object. TP-
Net extends a key concept of an existing classification network called
PointNet (Qi et al. 2017), and achieves invariance to $(N!)^{m}$ input
permutations, where $N$ denotes the number of points in each input point set
and $m$ denotes the length of the input sequence, i.e., the number of point
sets in the sequence that the model takes as input. In Section 3, we
demonstrate how our model achieves this property and explain why extending a
key idea from a classification network to design a time-series prediction
model for physical reasoning is an effective approach. The results from our
experiments indicate that TP-Net outperforms existing models, including state-
of-the-art, in predicting the future states of both simulated and real-world
deformable objects, with $O(N)$ time and space complexity.

The key contributions of our work are as follows:

  * •

We design a deep neural network that can directly consume a sequence of
unordered point sets to predict the future particle positions of simulated and
real-world deformable objects;

  * •

Based on qualitative and quantitative analysis, we show how the key idea from
an existing classification network can be extended to develop an effective
time-series prediction model for physical reasoning;

  * •

We demonstrate the efficiency of our approach compared to existing approaches
for consuming a sequence of unordered point sets, based on theoretical and
empirical analysis.

##  2 Related Work

#### Physical Reasoning on N-Body System

An early attempt to develop a learnable and differentiable physics engine that
can reason on interacting objects was achieved by the interaction network
(Battaglia et al. 2016). The interaction network showed the effectiveness of
utilizing graph neural networks (GNN) (Scarselli et al. 2008) for physical
reasoning on systems with $N$ bodies, which inspired several other studies to
propose learnable simulators based on GNN (Cranmer et al. 2020; Kipf et al.
2018; Graber and Schwing 2020; Webb et al. 2019; Li et al. 2019a; Martinkus,
Lucchi, and Perraudin 2021; Lee, Oh, and Lee 2021; Asif et al. 2021). These
approaches have in common that for each time step, the system is represented
as a graph where each node and edge represents an object and the relationship
between objects, respectively. By updating the state of each node over time
with message-passing (Duvenaud et al. 2015), the network can learn to infer
the future states of each node, based on relational reasoning (Santoro et al.
2017) and relational inductive bias (Battaglia et al. 2018).

Recently, dNRI (Graber and Schwing 2020) and NRI (Kipf et al. 2018) have
demonstrated its effectiveness in using GNN to predict the interactions
between particles purely from observational data. These approaches are also
applicable in predicting the future state of a deformable object that is
modeled with mass-spring system. However, these models can only learn to
predict the future from a sequence of ordered point sets. Although a graph for
each time step can be constructed from a _single_ unordered point set since a
graph is invariant to input permutations of the nodes, the order of the points
matters when reasoning on two or more different point sets. For example, if
the input point set at time step $t$ and $t+1$ are given as
$\\{(x_{1}^{t},y_{1}^{t}),(x_{2}^{t},y_{2}^{t}),(x_{3}^{t},y_{3}^{t})\\}$ and
$\\{(x_{2}^{t+1},y_{2}^{t+1}),(x_{1}^{t+1},y_{1}^{t+1}),(x_{3}^{t+1},y_{3}^{t+1})\\}$,
respectively, graph based networks learn a message-passing function based on
the assumption that $(x_{1},y_{1})$, $(x_{2},y_{2})$, and $(x_{3},y_{3})$
changes to $(x_{2}^{t+1},y_{2}^{t+1})$, $(x_{1}^{t+1},y_{1}^{t+1})$, and
$(x_{3}^{t+1},y_{3}^{t+1})$, respectively. Therefore, training or testing the
network with misaligned point set sequences results in poor performances, thus
restraining the model to generalize to real-world data where a sequence of
ordered point sets is not configurable. We empirically prove such behavior in
Section 4.2.

#### Physical Reasoning on Deformable Objects

Similar approaches of utilizing GNN were adopted to develop a learnable
simulator for deformable objects (Mrowca et al. 2018; Li et al. 2019b, 2020;
Sanchez-Gonzalez et al. 2020). In these studies, deformable objects are
modeled with particle-based representation to make future predictions on
individual particles, based on the learned dynamics of the system. These
approaches showed excellent performances in inferring the future dynamics of
simulated deformable objects, yet they assume a few conditions that are seldom
met in practice. In addition to the assumption that these approaches assume a
sequence of _ordered_ point sets as their input, the hierarchical relation
network (Mrowca et al. 2018) and dynamics prior (Li et al. 2019b, 2020) also
require additional attributes as input, such as relational attributes that
describes the relationship between individual particles in the system. Since
relational attributes are not observable in many real-world scenarios, the
model lacks generalization ability and shows poor performance without those
attributes, which we show this behavior in Section 4.

#### Consuming Sequence of Unordered Point Sets

While our method of consuming a sequence of unordered point sets is
fundamentally different from existing approaches, a few studies have also
proposed networks that can process a sequence of unordered point sets for
various purposes (Min et al. 2020; Fan and Yang 2019; Gomes, Rossi, and Toni
2021; Liu, Yan, and Bohg 2019). The key idea common in these approaches is
that for each point in the current frame, the network searches for k-nearest-
neighbors or relevant points in the previous frame. By combining state
information of neighboring points in the past, the model achieves permutation
invariance while preserving the spatial structure of the input. While this
approach showed state-of-the-art performance in gesture recognition, such
method shows poor performance when applied in future prediction (see results
of GraphRNN in Section 4.1). As the network significantly depends on local
features and neighbor points of each point, it suffers from serious
performance degradation when outliers or unseen points exist in the input,
which is a common scenario in long-term prediction. In addition, finding
spatio-temporal neighbor points for every point in the point set is a
$O(N^{2})$ operation – quadratic to the number of input points – that is
computationally expensive and lacks scalability to systems with large number
of particles.

##  3 Method

We define an unordered point set with $N$ points at time $t$ as
$P_{t}=\\{{p_{1,t}},{p_{2,t}},...{p_{N,t}}\\}$ with $p_{i,t}\in\mathbb{R}^{2}$
being the Euclidean coordinates of point $i$ in $P_{t}$. Then, we formulate
our problem by learning a continuous function $f$ that infers the future point
set $P_{t+1}$ of a deformable object modeled with mass-spring system, when a
sequence of $m$ unordered point sets is given as input:

$P_{t+1}=f(P_{t-(m-1)},\ P_{t-(m-2)},\ ...\ ,P_{t-1},\ P_{t}).$ (1)

###  3.1 Model Overview

To develop a framework that can process a sequence of unordered point sets for
physical reasoning, we propose our model called TP-Net, which has two key
modules: shared feature extractor and prediction network (Fig. 2). The key
idea of our model is that we aggregate global features of each point set to
predict individual particle positions in the next time step. In addition to
making TP-Net invariant to $(N!)^{m}$ input permutations, this approach is
also robust to input perturbations and outliers, thereby facilitating stable
long-term predictions, especially in predicting the shape deformation of the
object. We explain how TP-Net achieves these objectives in the following
section and demonstrate its effectiveness in Section 4.

###  3.2 Shared Feature Extractor

We first design our shared feature extractor that learns a continuous set
function $f_{G}$, which takes a single unordered point set $P_{t}$ as input
and outputs global features $G_{t}$ as described in (4). Then, we apply this
to each one of $m$ input point sets in parallel. This process can be
represented by the following equations:

$G_{t-(m-1)}=f_{G}(P_{t-(m-1)}),$ (2) $...\\\ $ $G_{t-1}=f_{G}(P_{t-1}),\\\ $
(3) $G_{t}=f_{G}(P_{t}).$ (4)

Since the global features extracted from each input point set are invariant to
$N!$ input permutations (i.e. invariant to the order of the points in the
input point set), we achieve invariance to $(N!)^{m}$ input permutations in
total since we extract global features from $m$ input point sets
independently. While global features $G_{t}$ are invariant to the order of the
points in the input point _set_ , we define global features $G_{t}$ as a
_sequence_ of features:

$G_{t}=(g_{1}^{t},\ g_{2}^{t},...\ ,g^{t}_{k}),$ (5)

where $g_{i}^{t}$ is the $i$-th feature of the point set $P_{t}$, and $k$ is
the number of features we encode from a point set. The purpose of defining
global features as a _sequence_ of features rather than a _set_ of features,
is to align the same type of features of each point set when we aggregate
multiple global features $G_{t-(m-1)},\ ...\ G_{t-1},\ G_{t}$ in the
prediction network. This allows the prediction network to apply a weight-
shared LSTM layer on these aligned global features.

#### Implementation Details

Our shared feature extractor adopts a submodule of the classification network
from PointNet (Qi et al. 2017). The classification network from PointNet takes
a single unordered point set as input and outputs the predicted score of each
class for object classification. While we use the same structure of this
network to design our shared feature extractor, we modify the last part of the
network to output global features. We also adjust the input vector shape and
layer sizes to process 2D data, instead of 3D data, since we use 2D datasets
(Fig. 2).

The network first applies a series of shared MLP layers on each point
independently for input transformations, feature transformations, and local
features extraction. Then, the network aggregates per point features by max
pooling to extract global features of the point set. Although the original
PointNet applies another series of MLP layers and a softmax function on these
global features to compute the scores for object classification, our network
applies a single MLP layer and a ReLU function on these global features
instead, to refine and extract the final global features as output.

#### Extending PointNet for Future Prediction

Although PointNet is originally designed to capture relevant features for
classification or semantic segmentation tasks, PointNet is also an excellent
source for future frame prediction of deformable objects with particle-based
representation. PointNet learns a continuous set function that summarizes the
shape of an object into a sparse set of key points. Since the global features
extracted from PointNet are bottleneck features of these key points, these
features are robust to small perturbation or outliers in the input. Therefore,
we extract global features from each point set in parallel and predict the
future particle positions by reasoning on these global features sequentially,
with the expectation that our model will be robust to outliers during
prediction rollout. In Section 4, we show this behavior of TP-Net and prove
its effectiveness.

###  3.3 Prediction Network

The prediction network learns a continuous function $f_{P}$, that takes global
features of each input point set as input and predicts the future particle
positions $P_{t+1}$:

$P_{t+1}=f_{P}(G_{t-(m-1)},\ ...\ ,\ G_{t-1},G_{t}).\\\ \\\ $ (6)

#### Implementation Details

We implement our prediction network by first aggregating the global features
$G_{t-(m-1)}$, $...$, $G_{t-1}$, $G_{t}$ into a sequence of global features
$\mathcal{G}$:

$\mathcal{G}=(G_{t-(m-1)},\ ...\ ,\ G_{t-1},\ G_{t}),\\\ $ (7)

to preserve the temporal structure of the original input. Note that global
features per every time step are also a _sequence_ of features as illustrated
in Equation (5). Since we apply the same shared feature extractor to all input
point set in parallel, the order of the features in every global features is
the same – i.e., $G_{i}=(g_{1}^{i},\ g_{2}^{i},...\ ,g_{k}^{i})$ for all
$i=t-(m-1),\ ...\ ,t-1,\ t$. Since $\mathcal{G}$ maintains the same order of
features for every $G_{i}$ across its entire sequence, we then apply a
bidirectional LSTM layer to $\mathcal{G}$ with the assumption that each
feature $g_{j}^{t}$ has an independent hidden state $h_{j}^{t}$ and a cell
state $c_{j}^{t}$:

$h_{j}^{t},\ c_{j}^{t}=\,\textrm{LSTM}\ (g_{j}^{t},\ h_{j}^{t},\
c_{j}^{t}).\\\ $ (8)

After we apply three bidirectional LSTM layers (with layer normalization (Ba,
Kiros, and Hinton 2016) in between bidirectional LSTM layers), we apply a
fully-connected layer and reshape its output to finally predict the 2D point
set $P_{t+1}$ (Fig. 2).

###  3.4 Model Training

#### Dataset

We first train and test our model on a synthetic dataset of moving deformable
object with particle-based representation. Then, we directly test our model on
real-world dataset that is similar to the synthetic dataset in terms of object
radius, velocity and rigidness, but essentially different in that the order of
the particles in each point set is unordered.

For synthetic dataset, we use the Box2D physics engine (Catto 2020) to
generate 7125 simulated trajectories of a deformable object. We model our
deformable object with a mass-spring system where 30 particles comprises the
boundary of the object and each particle is jointly connected to a number of
other particles by a restorable spring (Fig. 6). When generating the simulated
trajectories, we randomize the initial position, initial force and initial
direction of the object.

For real-world dataset, we capture 40 trajectories of a real-world deformable
object by using a high speed camera. Then, we preprocess the video data to
construct a sequence of unordered point sets that represents the trajectory of
the object. For details on how we prepare and preprocess our synthetic and
real-world dataset, please refer to Section A in the appendix.

#### Loss Function and Hyperparameters

We define our loss function $L_{CD}$ by using the Chamfer distance (Barrow et
al. 1977) between the ground truth point set $P$ and the predicted point set
$\hat{P}$:

$L_{CD}(P,\hat{P})=\sum_{x\in P}{\min_{y\in\hat{P}}\ \lVert
x-y\rVert^{2}_{2}}+\sum_{y\in\hat{P}}{\min_{x\in P}\ \lVert
x-y\rVert^{2}_{2}}.$ (9)

In order to empower our model to achieve long-term prediction ability with
robustness to input perturbations, we make our model to recursively predict
eight future point sets during training. Thus, we define our final loss $L$ as
the sum of the losses of each predicted point set:

$L=\sum^{8}_{i=1}{w_{i}L_{CD}(P_{t+i},\hat{P}_{t+i})},$ (10)

where $w_{1}=9$ and $w_{i}=1$ for $i=2,\ 3,\ ...\ ,8$ to balance the weight of
predicting future point sets between when a clean input is given and when
input with noise is given.

When training our model, we use a learning rate of $10^{-3}$ with the Adam
optimizer (Kingma and Ba 2015) and train it for 100 epochs with a batch size
of 128 on Intel(R) Core(TM) i5-8400 CPU @ 2.80Hz and GeForce GTX1080 Ti for 18
to 24 hours.

##  4 Experiment

Figure 3: Future prediction on moving deformable object (synthetic dataset). We visualize the prediction results after 15 to 22 time steps and after 145 time steps when five input frames are given initially. For the eight output frames from $t+15$ to $t+22$, we zoom-in the left-bottom quadrant of the frame. For the output frame at $t+145$, we draw a yellow bounding box to locate the deformable object in the frame. Table 1: Average rollout error of future prediction on moving deformable object (synthetic dataset). CD indicates the position error (Chamfer distance) and CD* indicates the shape error (variation of the vanilla Chamfer distance) of the prediction made from each model. The asteroid(*) next to DynamicsPrior denotes that the model is trained and tested without relation attributes. | # input frames = 3 | # input frames = 4 | # input frames = 5
---|---|---|---
Prediction Steps | 40 | 80 | 40 | 80 | 40 | 80
Rollout Error | CD | CD* | CD | CD* | CD | CD* | CD | CD* | CD | CD* | CD | CD*
TP-Net (ours) | 0.094 | 0.007 | 0.246 | 0.007 | 0.071 | 0.007 | 0.164 | 0.007 | 0.101 | 0.008 | 0.225 | 0.007
DynamicsPrior* | 0.110 | 0.044 | 0.252 | 0.050 | 0.074 | 0.040 | 0.186 | 0.046 | 0.116 | 0.041 | 0.231 | 0.064
dNRI | 0.291 | 0.031 | 0.867 | 0.055 | 0.092 | 0.024 | 0.882 | 0.046 | 0.106 | 0.022 | 0.613 | 0.035
DynamicNRI | 0.788 | 0.113 | 0.760 | 0.292 | 0.313 | 0.036 | 0.946 | 0.047 | 0.254 | 0.031 | 0.882 | 0.044
StaticNRI | 0.668 | 0.143 | 0.714 | 0.286 | 0.312 | 0.036 | 0.946 | 0.047 | 0.254 | 0.031 | 0.882 | 0.044
GraphRNN | 1.235 | 0.055 | 2.398 | 0.069 | 0.370 | 0.030 | 0.699 | 0.054 | 0.342 | 0.024 | 1.154 | 0.092

We study our framework using two datasets: synthetic dataset and real-world
dataset. For each dataset, we quantiatively and qualitatively compare the
performance of TP-Net with other benchmarks – dNRI (Graber and Schwing 2020),
StaticNRI (Kipf et al. 2018), DynamicNRI (Kipf et al. 2018), DynamicsPrior (Li
et al. 2019b, 2020), and GraphRNN (Gomes, Rossi, and Toni 2021) \- that are
capable of predicting the future states of a deformable object. Then, we
analyze and compare the time and space complexity of our model with GraphRNN.

#### Evaluation Metrics

For all experiments, we quantitatively measure the rollout error in two
aspects: position error and shape error. We first define the position error
$E_{p}$ as the average Chamfer distance between the ground truth point set $P$
and the predicted point set $\hat{P}$:

$E_{p}(P,\hat{P})=\frac{1}{|P|}\left(\sum_{x\in P}{\min_{y\in\hat{P}}\ \lVert
x-y\rVert^{2}_{2}}+\sum_{y\in\hat{P}}{\min_{x\in P}\ \lVert
x-y\rVert^{2}_{2}}\right).$ (11)

However, the average Chamfer distance cannot fully reflect the shape
difference between two point sets, as it simply measures the positional
difference between individual particles (refer to Section B in the appendix
for details). Therefore, we define the shape error $E_{s}$ by using a
variation of the vanilla Chamfer distance. We first apply a different
translation matrix to each point set $P$ and $\hat{P}$, such that the center
of each point set can be translated to the origin. Then, we compute the
average Chamfer distance on these translated point sets. The equation for
computing the shape error is as follows:

$E_{s}(P,\hat{P})=E_{p}(M_{c}P,\ M_{\hat{c}}\hat{P}),\\\ $ (12)

where $c$ is the center of mass of $P$, $\hat{c}$ is the center of mass of
$\hat{P}$, $M_{c}$ is a translation matrix that translates $x$ to $x-c_{x}$
and $y$ to $y-c_{y}$, and $M_{\hat{c}}$ is a translation matrix that
translates $x$ to $x-\hat{c}_{x}$ and $y$ to $y-\hat{c}$. In this way, we
directly compare the geometric shape of two point sets regardless of their
relative positions in the Euclidean space.

#### Environments

We train each model on synthetic dataset three times by giving different input
conditions: giving three, four, and five input frames. Although our model TP-
Net and GraphRNN can process both sequences of unordered and ordered point
sets, we train and evaluate each model by solely using sequences of ordered
point sets from the synthetic dataset to respect the precondition of the input
for dNRI, StaticNRI, DynamicNRI, and DynamicsPrior. Moreover, we only use the
position data for the input when training and evaluating each model, since
other attributes are difficult to obtain from real-world data. Therefore, we
zero-pad the relation attribute in the input when training and testing
DynamicsPrior. Also, we use the positional difference along the time steps as
the velocity attribute when training dNRI, StaticNRI, and DynamicNRI.

###  4.1 Synthetic Dataset

#### Qualitative Comparison

We test each model and visualize the predicted points along with the ground
truth for qualitative comparison. As illustrated in Fig. 3, TP-Net is the only
model that well predicts both the future trajectory and shape deformation of
the object. Generally, all other models except TP-Net show significant
performance degradation after the first or second collision of the object.
While DynamicsPrior can predict the future trajectory of the object with high
accuracy even in long term prediction, TP-Net significantly outperforms
DynamicsPrior in predicting the shape deformation. This proves the
effectiveness of our approach – using max-pooled global features to predict
the future state of each particles. Although TP-Net intermittently generates
an outlier during the rollout, the model selects informative features from the
point set and corrects those outliers in the next time step, thus maintaining
reasonable shape of the object in long term. Conversely, model that uses
neighboring points or local structures to predict the future state of each
points, e.g., GraphRNN, shows significantly poor performance in long-term
prediction.

#### Quantitative Comparison

We also evaluate and compare the performance by using quantitative metrics. We
compute the average rollout error on 60 simulated trajectories of a deformable
object, and compare its value after 40 and 80 time steps, respectively. As
presented in Table 1, our model TP-Net achieves state-of-the-art performance
in terms of both metrics – position error and shape error. Although
DynamicsPrior shows as high performance as TP-Net in terms of position error,
the metric needs additional interpretation. As illustrated in Fig. 3,
DynamicsPrior is simply good at predicting the general trajectory of
individual particles without reasoning on its geometric shape, which could
still result in low position error. In fact, TP-Net not only outperforms
DynamicsPrior by a large margin (6-9 times) in terms of shape error, but also
maintains a relatively constant shape error even in long term prediction.

Moreover, note that the analysis above is based on the results when each model
is trained and tested solely on sequences of ordered point sets. It is clear
that training and testing each model on sequences of unordered point sets will
result in poor performances for dNRI, StaticNRI, DynamicNRI, and
DynamicsPrior, since they highly depend on the order of the points in each
input point set. We will also show this behavior in the next section.

Figure 4: Future predictions on real deformable object. The images in the
first row are the original frames from the camera. The images in the bottom
row are generated by plotting the predicted positions of each particle in the
background image. The images in the middle row are generated by painting the
interior of the contour drawn by connecting the predicted particles.

(a) Average position error of future prediction on real-world data

(b) Average shape error of future prediction on real-world data

Figure 5: For each model, we measure the rollout error after 40 time steps of
each scenario with different input ordering method.

###  4.2 Real world Dataset

#### Generalization to Real World Data

To demonstrate the generalization ability of TP-Net to real-world scenarios,
we test our model that was only trained on synthetic dataset, directly on
real-world dataset. Since we cannot configure a sequence of ordered point sets
for input from real-world data, we arbitrarily configure the order of the
points in each input frame and validate our model’s ability in processing a
sequence of unordered point sets. We give five steps of particle positions as
input and make our model predict the future particle positions until the
object disappears from the scene.

The results illustrated in Fig. 4 shows that the learned policy of our model
shows great potential to generalize to real world scenarios. Without knowing
any information of the point correspondences between different input frames,
TP-Net can reason on the underlying physical dynamics and infer the future
states of a real world deformable object with reasonable accuracy.

#### Comparison to Other Works

We also compare the generalization ability of TP-Net to other benchmarks. To
test the robustness of each model to different input ordering methods, we test
each model on each test case with three different kinds of input. The three
different input ordering method we use to preprocess the input are as follows:
(1) Sorting the points in each input frame in ascending $x$ positions, (2)
sorting the points in each input frame in descending $y$ positions, and (3)
randomly shuffling the order of the points in each input frame.

We evaluate the rollout error on 40 trajectories of real-world deformable
objects when three input frames are given. Fig. 5 shows that TP-Net
outperforms all other benchmarks in terms of both metrics and for all kinds of
input ordering methods. It is not surprising that TP-Net and GraphRNN shows
constant performance regardless of the input ordering method as they are
invariant to input permutations. However, TP-Net outperforms GraphRNN by a
large margin in both metrics, which demonstrates the effectiveness of our
approach in processing a sequence of unordered point sets. For other models,
the performance fluctuates depending on the input ordering method, but there
exists no ordering that results in the best performance for all models, or
results in higher performance than TP-Net. Although we have only tried three
methods of input ordering, these results imply that there exists no ordering
of a point set that ensures stable performance in the general sense, thus
demonstrating the importance of achieving permutation invariance.

###  4.3 Time and Space Complexity Analysis

In this section, we compare the time and space complexity of TP-Net and
GraphRNN (Gomes, Rossi, and Toni 2021) which are both capable of processing a
sequence of unordered point sets for future prediction.

Table 2 shows the time complexity – number of floating-point operations per
sample (FLOPs/sample) – and space complexity – number of parameters in the
network (#params) – of TP-Net and GraphRNN. While GraphRNN is more space
efficient than TP-Net in terms of #params in the network ($18.75$ times less
parameters), TP-Net is significantly more efficient in computational cost as
it has $354.4$ times less floating point operations per sample. As described
in Section 2, the time complexity of GraphRNN is $O(N^{2})$ – quadratic in the
number of points – because it searches for spatio-temporal neighbor points for
every point in the current frame. However, the time and space complexity of
TP-Net is $O(N)$ since it simply extracts global features from each point set
by max pooling per point features, and reasons on those global features
sequentially. This makes our approach more scalable than existing methods.
Empirically, TP-Net can predict more than 120 future point sets per second
when tested on our real-world dataset using a GTX 1080Ti GPU on Tensorflow.
Based on these results, we believe our framework can be widely used in real-
time applications.

Table 2: Time and space complexity of deep architectures for processing a sequence of three unordered point sets with 30 particles. The “M” stands for million. |  FLOPs/sample |  #params
---|---|---
TP-Net (ours) |  21.4M |  7.5M
GraphRNN |  7584.6M |  0.4M

##  5 Conclusion

We presented a deep neural network named TP-Net that can directly consume a
sequence of unordered point sets to infer the future dynamics of a deformable
object. By using global features of each point set, our model achieves
invariance to $(N!)^{m}$ input permutations and shows state-of-the-art
performance, even in long term prediction. We also demonstrated that our model
well generalizes to real-world scenarios and is much more efficient than
existing methods of consuming a sequence of unordered point sets.

Although we have only used a 2D dataset in this study, in principle, our
framework easily generalizes to datasets with higher dimensions. Moreover, we
focused on consuming a sequence of static point sets in this study whereas
many situations involve reasoning on dynamic point sets, where the number of
points change over time. We believe that further works that address these
concerns will empower and extend our framework with much more flexibility.

## Acknowledgements

This research was supported by the MSIT(Ministry of Science and ICT), Korea,
under the ITRC(Information Technology Research Center) support
program(IITP-2021-2018-0-01419) supervised by the IITP(Institute for
Information and Communications Technology Planning and Evaluation) and the
National Research Foundation of Korea(NRF) grant funded by the Korea
government(MSIT). (No. NRF-2020R1A2C2014622)

## References

  * Asif et al. (2021) Asif, N. A.; Sarker, Y.; Chakrabortty, R. K.; Ryan, M. J.; Ahamed, M. H.; Saha, D. K.; Badal, F. R.; Das, S. K.; Ali, M. F.; Moyeen, S. I.; et al. 2021.  Graph Neural Network: A Comprehensive Review on Non-Euclidean Space.  _IEEE Access_.
  * Ba, Kiros, and Hinton (2016) Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016.  Layer normalization.  _arXiv preprint arXiv:1607.06450_.
  * Barrow et al. (1977) Barrow, H. G.; Tenenbaum, J. M.; Bolles, R. C.; and Wolf, H. C. 1977.  Parametric correspondence and chamfer matching: Two new techniques for image matching.  Technical report, SRI Int. Menlo Park CA Artificial Intelligence Center.
  * Battaglia et al. (2018) Battaglia, P. W.; Hamrick, J. B.; Bapst, V.; Sanchez-Gonzalez, A.; Zambaldi, V.; Malinowski, M.; Tacchetti, A.; Raposo, D.; Santoro, A.; Faulkner, R.; et al. 2018.  Relational inductive biases, deep learning, and graph networks.  _arXiv preprint arXiv:1806.01261_.
  * Battaglia et al. (2016) Battaglia, P. W.; Pascanu, R.; Lai, M.; Rezende, D.; and Kavukcuoglu, K. 2016.  Interaction networks for learning about objects, relations and physics.  _Advances in Neural Information Processing Systems_.
  * Catto (2020) Catto, E. 2020.  Box2D: A 2D Physics Engine for Games.
  * Collins et al. (2021) Collins, J.; Chand, S.; Vanderkop, A.; and Howard, D. 2021.  A Review of Physics Simulators for Robotic Applications.  _IEEE Access_ , 9: 51416–51431.
  * Cranmer et al. (2020) Cranmer, M.; Sanchez-Gonzalez, A.; Battaglia, P.; Xu, R.; Cranmer, K.; Spergel, D.; and Ho, S. 2020.  Discovering symbolic models from deep learning with inductive biases.
  * de Avila Belbute-Peres et al. (2018) de Avila Belbute-Peres, F.; Smith, K.; Allen, K.; Tenenbaum, J.; and Kolter, J. Z. 2018.  End-to-End Differentiable Physics for Learning and Control.  In _Advances in Neural Information Processing Systems_.
  * Degrave et al. (2019) Degrave, J.; Hermans, M.; Dambre, J.; and wyffels, F. 2019.  A Differentiable Physics Engine for Deep Learning in Robotics.  _Frontiers in Neurorobotics_ , 13: 6.
  * Duvenaud et al. (2015) Duvenaud, D.; Maclaurin, D.; Aguilera-Iparraguirre, J.; Gómez-Bombarelli, R.; Hirzel, T.; Aspuru-Guzik, A.; and Adams, R. P. 2015.  Convolutional networks on graphs for learning molecular fingerprints.  2224–2232.
  * Fan and Yang (2019) Fan, H.; and Yang, Y. 2019.  PointRNN: Point Recurrent Neural Network for Moving Point Cloud Processing.  _arXiv_ , 1910.08287.
  * Freeman et al. (2021) Freeman, C. D.; Frey, E.; Raichuk, A.; Girgin, S.; Mordatch, I.; and Bachem, O. 2021\.  Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation.  _CoRR_ , abs/2106.13281.
  * Gomes, Rossi, and Toni (2021) Gomes, P.; Rossi, S.; and Toni, L. 2021.  Spatio-temporal Graph-RNN for Point Cloud Prediction.
  * Graber and Schwing (2020) Graber, C.; and Schwing, A. 2020.  Dynamic Neural Relational Inference.  In _Computer Vision and Pattern Recognition_.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S.; and Schmidhuber, J. 1997.  Long short-term memory.  _Neural Computation_ , 9(8): 1735–1780.
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015.  Adam: A Method for Stochastic Optimization.  In _Int. Conf. Learning Representations_.
  * Kipf et al. (2018) Kipf, T.; Fetaya, E.; Wang, K.-C.; Welling, M.; and Zemel, R. 2018.  Neural Relational Inference for Interacting Systems.  _Int. Conf. Machine Learning_.
  * Lee, Oh, and Lee (2021) Lee, D.; Oh, Y. J.; and Lee, I.-K. 2021.  Discovering Latent Representations of Relations for Interacting Systems.  _IEEE Access_ , 9: 149089–149099.
  * Li et al. (2020) Li, Y.; Lin, T.; Yi, K.; Bear, D.; Yamins, D. L.; Wu, J.; Tenenbaum, J. B.; and Torralba, A. 2020.  Visual Grounding of Learned Physical Models.  In _Int. Conf. Machine Learning_.
  * Li et al. (2019a) Li, Y.; Meng, C.; Shahabi, C.; and Liu, Y. 2019a.  Structure-informed graph auto-encoder for relational inference and simulation.  In _Int. Conf. Machine Learning Workshop on Learning and Reasoning with Graph-Structured Representations_.
  * Li et al. (2019b) Li, Y.; Wu, J.; Tedrake, R.; Tenenbaum, J. B.; and Torralba, A. 2019b.  Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids.  In _Int. Conf. Learning Representations_.
  * Liu, Yan, and Bohg (2019) Liu, X.; Yan, M.; and Bohg, J. 2019.  MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequences.  In _Int. Conf. Computer Vision_.
  * Macklin et al. (2014) Macklin, M.; Müller, M.; Chentanez, N.; and Kim, T.-Y. 2014.  Unified particle physics for real-time applications.  _ACM Transactions on Graphics (TOG)_ , 33(4): 1–12.
  * Martinkus, Lucchi, and Perraudin (2021) Martinkus, K.; Lucchi, A.; and Perraudin, N. 2021.  Scalable Graph Networks for Particle Simulations.  _Proceedings of the AAAI Conf. on Artificial Intelligence_ , 35(10): 8912–8920.
  * Min et al. (2020) Min, Y.; Zhang, Y.; Chai, X.; and Chen, X. 2020.  An Efficient PointLSTM for Point Clouds Based Gesture Recognition.  In _Computer Vision and Pattern Recognition_ , 5761–5770.
  * Mrowca et al. (2018) Mrowca, D.; Zhuang, C.; Wang, E.; Haber, N.; Fei-Fei, L.; Tenenbaum, J. B.; and Yamins, D. L. 2018.  Flexible Neural Representation for Physics Prediction.  In _Advances in Neural Information Processing Systems_.
  * Qi et al. (2017) Qi, C. R.; Su, H.; Mo, K.; and Guibas, L. J. 2017.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _Computer Vision and Pattern Recognition_ , 652–660.
  * Sanchez-Gonzalez et al. (2020) Sanchez-Gonzalez, A.; Godwin, J.; Pfaff, T.; Ying, R.; Leskovec, J.; and Battaglia, P. 2020.  Learning to simulate complex physics with graph networks.  In _Int. Conf. Machine Learning_ , 8459–8468. PMLR.
  * Santoro et al. (2017) Santoro, A.; Raposo, D.; Barrett, D. G.; Malinowski, M.; Pascanu, R.; Battaglia, P.; and Lillicrap, T. 2017.  A simple neural network module for relational reasoning.  In _Advances in Neural Information Processing Systems_ , volume 30, 4967–4976. Curran Associates, Inc.
  * Scarselli et al. (2008) Scarselli, F.; Gori, M.; Tsoi, A. C.; Hagenbuchner, M.; and Monfardini, G. 2008\.  The graph neural network model.  _IEEE Transactions on Neural Networks_ , 20(1): 61–80.
  * Webb et al. (2019) Webb, E.; Day, B.; Andres-Terre, H.; and Lió, P. 2019.  Factorised neural relational inference for multi-interaction systems.  In _Int. Conf. Machine Learning Workshop on Learning and Reasoning with Graph-Structured Representations_.

##  Appendix A Data Generation

###  A.1 Synthetic Data

#### Formulating a Deformable Object

(a)

(b)

Figure 6: A simplified diagram describing the formulation of a deformable
object using the Box2D physics engine. (a) At rest state, the distance between
the center $C$ of the object and each point $p_{i}$ is set to two – i.e.,
$r_{i}=2$ for all $i=1,\ ...\ ,n$. We also connect $p_{l}$ and $p_{k}$ with a
restorable spring $s_{l,k}$ for all $|l-k|<\frac{2}{5}n$, although most of the
spring connections that exist in the deformable object are omitted in this
figure for simplicity. We also set $\theta_{i,j}=\frac{2\pi}{n}$ for all $i,j$
$\in\\{1,\ ...\ ,n\\}$, which denotes the angle between neighboring points
with respect to the center of the object. (b) Spring connections between
particles enable deformation of the object. Figure 7: The process of obtaining
point set data from raw video frames. $F_{t}$ denotes a frame captured from
the original video at time $t$. $F_{t}^{{}^{\prime}}$ denotes a frame with
background subtraction at time $t$. $C_{t}$ denotes the contour of the object
extracted from the frame at time $t$. Note that $P_{t}$ denotes a point set
obtained from the original video frame at time $2t-1$ due to time step
adjustment.

In this section, we describe how we formulate a deformable object when
generating the synthetic dataset with Box2D physics engine (Catto 2020). As
illustrated in Fig. 6, we model a deformable object with mass-spring system,
that has $n$ particles $P=\\{p_{1},p_{2},...,p_{n}\\}$ comprising its
boundary. For every particle pair $(p_{i},p_{j})$ that is
$|i-j|<\frac{2}{5}n$, we connect the particles with a restorable spring that
shares the same spring constant across all springs within the object. In order
to mimic the movement of a real-world deformable object that we use to
generate our real-world dataset, we configure the Box2D environment as
follows:

  * •

n = 30,

  * •

gravity = (0, -0.5),

  * •

friction of each particle = 1,

  * •

friction of the wall = 1,

  * •

frequency of the spring = 1,

  * •

damping ratio of the spring = 0,

  * •

restitution of each particle = 0,

  * •

restitution of the wall = 1.

#### Data preparation

After we model a deformable object in a 2D Euclidean space, we generate 7125
simulated trajectories of a deformable object such that each trajectory is a
sequence of 600 point sets. When generating simulated trajectories, we
randomize the initial position, initial force (magnitude), and initial
direction of the object. The scope of randomizing each property is as follows:

  * •

initial center position = $\\{(x,y)\ |\ 2.9\leq x,y\leq 42.1\\}$.

  * •

initial force (magnitude) = $\\{1.0,1.15,1.3,1.45,1.6\\}$,

  * •

initial direction = $\\{$$,$$,$$,\ ...\ ,$$\\}$,

After we generate 7125 simulated trajectories, we construct our training
dataset and validation dataset by sampling subsequences from each trajectory.
Each sub-sequence is a sequence of 11 or 12 or 13 point sets, depending on the
number of input frames we use, which varies from three to five point sets. For
240 trajectories, we set the entire trajectory as the training scope, and for
6635 trajectories, we only sample subsequences from certain parts to construct
our training dataset. The rest of the trajectories are used to generate the
validation dataset and test dataset. In order to increase the prediction
accuracy of our model especially in terms of shape deformation, for each
trajectory, we sample 15 subsequences that involves the interval in which the
object collides with the wall, and sample 5 subsequences from a normal
interval within the trajectory. We also normalize the position data of each
points to be between 0 and 1 when training, and denormalize the data when
visualizing the results.

When generating our test dataset, we generate 60 simulated trajectories with a
unique pair of initial position, initial force, and initial direction that the
model was never trained on, in order to test the generalization ability of our
model.

###  A.2 Real-World Data

Figure 8: Limitations of using the vanilla Chamfer distance (position error)
as a single evaluation metric when comparing the performance of each model in
our study. Figure 9: The process of computing shape error $E_{s}$.

#### Generating Real-World Trajectories

Due to the absence of public dataset for moving real-world deformable objects,
we collect and construct a novel real-world dataset from scratch in our study.
By using a high-speed camera that can capture up to 480 frames per second, we
film a fast moving real-world deformable object. We throw a deformable object
from various positions with varying forces and directions to be similar but
not the same with the synthetic dataset. Since we convert our video data into
a sequence of 2D point sets, we control the depth value of the object by
throwing the object in parallel with the camera at a constant distance.

#### Data preprocessing

We preprocess the video data to obtain the particle positions data that our
model can consume as input. As illustrated in Fig. 7, we apply a series of
operations to the original video frames to obtain the point set $P$ for each
time step. First, we sample video frames at odd time steps to adjust the time
step of real-world deformable object to be similar to that of the simulated
deformable object which our model was trained on. Then, we subtract the
background of each frame to identify the contour $C$ of the deformable object.
After we obtain the contour of the object, we uniformly divide the contour
into $n$ points as described in Fig. 6 to obtain the particle positions data
$P$. Although the obtained point sets $P_{1}$ and $P_{2}$ in Fig. 7 represent
the states that are originally two time steps away from each other, we define
them as point sets with consecutive time steps when composing our final real-
world dataset. Note that we also normalize the position data to be between 0
and 1 by using the relative pixel coordinates with respect to the wall and
floor. Conversely, when we visualize the predicted point sets, we plot the
points on the background image by computing its relative pixel coordinate.

##  Appendix B Evaluation

#### Adoption of shape error

As described in Section 4, we use two metrics to evaluate our model – position
error($E_{p}$) and shape error($E_{s}$). The purpose of using two metrics for
evaluation is to evaluate the prediction accuracy of our model in two
different perspectives. As shown in (11), position error is the Chamfer
distance between the ground truth point set and predicted point set, which
measures the distance between individual particles. In Fig. 8, $P_{A}$ is a
better prediction of the ground truth than $P_{B}$ in terms of position error
since the distances between individual particles in each point set are much
closer than that of $P_{B}$. However, it is clear that $P_{B}$ is a much
better prediction of the ground truth in terms of shape. Therefore, using the
vanilla Chamfer distance as a single evaluation metric can mislead to false
interpretation of the performance of the model in these cases. In order to
prevent such misunderstanding, we define our shape error as described in (12).

#### Computing shape error

In this section, we provide a detailed explanation of (12) which is used for
computing the shape error. As illustrated in Fig. 9, we first compute the
center of mass $c$ and $\hat{c}$ of each point set $P$ and $\hat{P}$,
respectively. Then, we apply a translation matrix $M_{c}$ and $M_{\hat{c}}$ to
$P$ and $\hat{P}$, respectively, to shift the center of each point set to the
origin. After translating each point set, we then compute the Chamfer distance
between these translated point sets to compute the shape error.