# Tracking monocular camera pose and deformation for
SLAM inside the human body

Juan J. Gómez Rodríguez, J.M.M. Montiel, and Juan D. Tardós This work was
supported by EU-H2020 grant 863146: ENDOMAPPER, Spanish government grant
PGC2018-096367-B-I00 and by Aragón government grant DGA_T45-17R and PhD
scholarship of J. J. Gómez-Rodrígez.The authors are with the Instituto de
Investigación en Ingeniería de Aragón (I3A), Universidad de Zaragoza, María de
Luna 1, 50018 Zaragoza, Spain. E-mail: {jjgomez, josemari<EMAIL_ADDRESS>
###### Abstract

Monocular SLAM in deformable scenes will open the way to multiple medical
applications like computer-assisted navigation in endoscopy, automatic drug
delivery or autonomous robotic surgery. In this paper we propose a novel
method to simultaneously track the camera pose and the 3D scene deformation,
without any assumption about environment topology or shape. The method uses an
illumination-invariant photometric method to track image features and
estimates camera motion and deformation combining reprojection error with
spatial and temporal regularization of deformations. Our results in simulated
colonoscopies show the method’s accuracy and robustness in complex scenes
under increasing levels of deformation. Our qualitative results in human
colonoscopies from Endomapper dataset show that the method is able to
successfully cope with the challenges of real endoscopies: deformations, low
texture and strong illumination changes. We also compare with previous
tracking methods in simpler scenarios from Hamlyn dataset where we obtain
competitive performance, without needing any topological assumption.

##  I INTRODUCTION

Visual Simultaneous Localization and Mapping (SLAM) and Visual Odometry (VO)
in static environments have been hot research topics in the last decades and
many methods have raised to solve them with outstanding accuracy and
robustness using features [1], direct methods [2], or hybrid techniques [3].
The increasing popularity of these techniques has raised expectations to solve
SLAM in more complex scenarios. For example, one can think of many useful
applications of SLAM in Minimal Invasive Surgery (MIS) like guiding surgeons
through augmented reality annotations towards the place where a polyp was
detected in a previous exploration, and automatic polyp measurement to analyze
its evolution. Moreover, surgical robots would greatly benefit from SLAM
inside the human body as they will be more secure, robust and accurate, and
they will be able to combine information coming from previous explorations or
from other sensors like Computerized Tomography (CT).

However, visual SLAM inside the human body poses tremendous challenges like
weak texture, changing illumination, specular reflections, and lack of
rigidity (Fig. 1). Weak texture and specular reflections hinder data
association algorithms based on feature matching, preventing methods like ORB-
SLAM3 [4] from working in these sequences. On the other hand, changing
illumination puts direct methods like DSO [2] or DSM [5] and hybrid methods
like SVO [3] in serious trouble, as they assume constant illumination of the
environment. In contrast, we solve data association with a modified Lucas-
Kanade algorithm, first presented in [6], that is able to cope with local
illumination changes.

Figure 1: Tracking the camera pose and scene deformation in a human
colonoscopy from Endomapper dataset. Top: images from the sequence. Bottom:
camera trajectory in blue, undeformed map points in black and map point
deformation trajectories in red.

But the major challenge to be addressed for SLAM inside the human body is
deformable scenes, as breaking up with the rigidity assumption impairs both
environment reconstruction and camera tracking. The recent DefSLAM [7] is the
first monocular deformable SLAM system able to perform tracking and mapping,
but it strongly relies on the assumption of a smooth continuous shape with
planar topology, which does not hold in colonoscopies (see Fig. 1).

In this paper we present the first pure monocular method able to initialize a
map and track camera pose and scene deformation in general scenes inside the
human body (Fig. 1), without any topological or shape assumption. Our main
contribution is a simple formulation that combines photometric feature
tracking and an optimization based on reprojection error with spatial and
temporal regularizers that encode local assumptions over the environment
deformation, endowing our algorithm with enough expressivity to model complex
scenes and track their deformations in real-life endoscopies. We provide
quantitative evaluation on realistic colonoscopy simulations [8] and
qualitative results on real human colonoscopies from the Endomapper project
[9], that were out of reach for previous techniques. We present quantitative
comparisons in almost-planar scenes from Hamlyn dataset [10] where we obtain
competitive performance, despite not using any assumption on the scene
topology or shape.

##  II RELATED WORK

The computer vision and robotics communities have developed excellent rigid
visual SLAM systems [2] [3] [4] in the last years. While all these algorithms
use quite different techniques they all rely in a vital, yet simple,
assumption: that the environment is static. In contrast, deformable SLAM,
which completely breaks up with the rigidity assumption, is still a
challenging research topic.

Many works have tried to solve deformable SLAM by using sensors that provide
complete 3D information of the environment like stereo or RGB-D cameras. This
is the case of the seminal DynamicFusion [11] which uses RGB-D images to
reconstruct highly deforming environments with an Iterative Closest Point
(ICP) algorithm and a spatial regularizer to constrain deformations of points
that are close to each other, that we adopt in our work. Several extensions to
DynamicFusion have been developed since then, being the most notably
VolumeDeform [12] which combines the use of SIFT features and reprojection
error with a dense ICP data term, to reduce drift and increase robustness. In
[13] they formulate a variational method that takes RGB-D images to
reconstruct a deforming environment. This work is later extended in [14]
introducing camera pose computation.

There is increasing interest in SLAM in Minimally Invasive Surgery (MIS),
where RGB-D sensors are not available. For this reason, works like [15] [16]
use depth coming form stereo images and an error function that combines
reprojection errors and regularizers to perform deformable SLAM. As before,
the reprojection error is augmented using other 3D terms like ICP errors or
Point-to-Plane errors. Regarding the regularizers used, they are similar to
the one introduced in DynamicFusion to represent that deformations occur
locally, using pair-wise deformation terms between close points, which
prevents divergence of individual points in the reconstruction.

Nevertheless stereo cameras are not appropriate for certain applications like
colonoscopies where two cameras with enough baseline may not fit in the body
cavities. In this kind of scenarios, only monocular deformable SLAM can be
performed. This is even a harder problem since no real 3D information is
available from a single view, scale is unobservable, and combining multiple
views of a deforming scene is an open issue. The first monocular deformable
SLAM system is DefSLAM [7] which splits the deformable SLAM problem in 2
threads, one for tracking and one for mapping. They use ORB features and
minimize a reprojection error term with a deformation energy term that
penalizes stretching and bending of the imaged surface. However, as ORB
features are quite unstable in intracorporeal sequences, SD-DefSLAM [6]
extends DefSLAM to a semi-direct method by integrating an illumination-
invariant Lucas-Kanade tracker to perform data association, achieving better
robustness and accuracy. Crucially, both methods assume that the surface has
planar topology, and model the surface with a triangle mesh which impose a
strong global condition over the environment: the imaged surfaces have to be
continuous with no holes. This is quite a strong assumption that seriously
limits the kind of scenes that can be handled by both algorithms excluding,
for example, colonoscopies (Fig. 1).

To tackle this limitation, [17] proposes a fully photometric algorithm to
track camera pose and deformation using sparse 3D surfels (surface elements)
under the assumption of local isometry. The use of surfels that have no
constrains between them allows to model any kind of topologies. While
obtaining very good results in medical scenes, the method still requires 3D
information coming from a stereo camera to initialize the surfels. Also, the
use of large surfels (in practice, square patches of $23\times 23$ pixels in
the image) can easily violate the local isometry assumption and is inefficient
as using too many close pixels provide redundant information with little to no
improvement in accuracy [2]. Furthermore, the regularizers used impose small
deformations with respect to a pose at rest, which can be inappropriate in
many applications.

In contrast, in this work we propose a pure monocular method for tracking
camera pose and deformation. Under the assumption of slow deformations, we
perform fully automatic monocular map initialization to obtain a first seed of
the environment structure. Following previous works [3, 6], we use photometric
feature tracking for robustness and accuracy, and reprojection error for
convergence and efficiency during optimization. In addition, we integrate two
regularizers that encode our assumptions of smooth and slow deformation in
order to constrain the reconstruction problem.

##  III DEFORMABLE TRACKING

This section is devoted to presenting our tracking algorithm. We first present
the assumptions that governing our system. Afterwards we introduce our data
association for tracking. Then we explore our algorithm for monocular map
initialization and finally we present our optimization backbone and its
formulation encoding each one of our assumptions.

###  III-A Assumptions

The biggest difficulty when dealing with deformable scenarios is that the
rigid assumption is violated. This makes camera pose and deformation
prediction a non-separable problem for which infinite solutions arise, i.e.
not all degrees of freedom (DoF) are observable.

This is even drastically worse when using a single monocular camera as the
scale is also unknown. For all this, one must incorporate some a-priori
knowledge into the problem in order to confine the possible solutions into a
reduced set of solutions that correctly represents the real nature of the
environment. In this paper, we propose the following assumptions to constrain
our reconstruction problem:

  1. 1.

Local isometry: we assume that the vicinity of a surface point follows an
isometric model, that is local distances are preserved.

  2. 2.

Smooth deformation: we consider that points that are close in space must
undergo similar deformations.

  3. 3.

Slow deformation: deformations are assumed to happen slowly over time.

  4. 4.

Camera motion is faster than deformation: finally, we assume that camera can
move faster than deformations, so we attribute rigid motions to the camera,
computing deformations as small as possible.

Assumption (1) enables us to use a phtometric feature tracker defining a small
neighbourhood around each tracked point that is assumed to be locally rigid.
This allows us to take an approach similar to [18] to perform short term data
association. Assumption (2) introduces local constrains in the deformations
observed without imposing a global deformation or surface model. This
effectively makes our system general enough to model any environment.
Assumption (3) allow us to imposes temporal continuity in the position of
surface points, reducing the effect of image and data association noise.
Finally, Assumption (4) is the one that allows us to separate camera motion
and environment deformation. In all SLAM systems, the sensor provides relative
information, and as a result, the absolute pose of camera and environment is
not observable. In rigid SLAM this is simply addressed by choosing an
arbitrary global pose, for example the first camera pose is chosen to be zero.
In deformable SLAM this is not enough as a camera motion is indistinguishable
from a hypothetical case where all the environment moves rigidly, what is
called the floating map ambiguity in [17]. This assumption allow us to use
regularizers that penalize deformation over camera motion. In that way, the
rigid part of the relative motion between environment and camera will be
attributed to camera motion, obtaining deformations as small as possible.

It is important to note that none of the above assumptions impose global
constrains over the surface topology, smoothness, or deformation, allowing us
to model generic deformations and environments.

###  III-B Data Association

Our previous experience in monocular deformable SLAM [6] has proven that an
accurate data association is crucial in order to reach good accuracy and
robustness. Indeed other works have shown the potential of direct methods in
this task like [2] in which the photometric term allows to get as a byproduct
of the tracking the feature associations. This is done by imposing a global
rigid transformation to all points as it is assumed that the environment is
stationary. However this is far to be true in deformable SLAM. Indeed one can
not impose any global constrain to the data association step as it is easily
violated by deformations.

For that, we propose to perform photometric data association with Shi-Tomasi
features [19] prior the camera pose and deformation estimation using the
modified multi-scale Lucas-Kanade algorithm proposed in [6]:

$\operatorname*{arg\,min}_{\mathbf{d},\alpha,\beta}\sum_{\mathbf{v}\in
P(\mathbf{u})}(I^{0}(\mathbf{v})-\alpha
I^{t}\left(\mathbf{v}+\mathbf{d})-\beta\right)^{2}$ (1)

where $P(\mathbf{u})$ is a small pixel patch centered at the keypoint
$\mathbf{u}$. $I^{0}$ is the firs frame, where the points are intialize, and
$I^{t}$ is the current frame in time $t$. These patches are updated every 5
images to account for big scale changes or rotations. This algorithm has been
proven to achieve excellent results when tracking image features in short time
steps even in the presence of deformations or local illumination changes (Fig.
2). The key of its performance lies in using no global model: each point can
move freely with respect the others. Also a local illumination invariance is
achieved by computing local gain $\alpha$ and bias $\beta$ terms for each
point.

In order to remove any possible outlier track, we compute the Structural
Similarity Index (SSIM) [20] between the reference $x$ and tracked $y$ pixel
patches to identify any outlier track:

$SSIM(x,y)=\frac{(2\mu_{x}\mu_{y}+C_{1})(2\sigma_{xy}+C_{2})}{(\mu^{2}_{x}+\mu^{2}_{y}+C_{1})(\sigma^{2}_{x}+\sigma^{2}_{y}+C_{2})}$
(2)

where $\mu_{x}$ and $\sigma_{x}$ are the mean and covariance of the pixel
patch, $\sigma_{xy}$ is the crossed covariance between both patches and
$C_{1}$ and $C_{2}$ are constant values to avoid inestability when means and
covariances approaches to zero. This has been proven to be a good similarity
metric for small pixel windows as it combines in a same metric a luminance,
contrast and structure comparison.

Figure 2: Top: Two images separated by 3 frames from our EndoMapper dataset
with tracked features. Bottom: map initialized from those tracks

###  III-C Monocular map initialization

Initializing a map from monocular images in rigid environments is well known
in Structure from Motion (SfM). In deforming environments, Non-Rigid Structure
from Motion (NRSfM) techniques can be used [7]. However, they require, for
example in the map initialization, assumptions such as a smooth scene surface
with planar topology, which are not met in real colonoscopies (see Fig. 1 and
2).

We propose to exploit assumption (4) using two close frames in which the
environment can be considered quasi-rigid, and most image innovation can be
attributed to the camera motion. This allows to apply SfM to obtain a first
estimation of the map as if it is rigid and treat any deformation as small
noise.

Ideally, the method should to be independent of the camera model either
pinhole or fish-eye. We propose to initialize the monocular map by computing
the Essential matrix between 2 close frames using as input normalized
projective rays from features in the images. Our proposed initialization
algorithm goes through the following steps:

  1. 1.

Extract Shi-Tomasi features evenly distributed in the reference frame $I^{0}$
and track them in the current frame $I^{t}$ using the Lucas-Kanade optical
flow algorithm. Unproject the matched features into normalized rays
$\mathbf{x}_{i}^{0}$ and ${\mathbf{x}^{t}_{i}}$ and using the camera model
unprojection function.

  2. 2.

Compute an Essential matrix that relates poses of the two frames:

${{\mathbf{x}}^{t}_{i}}^{T}\mathbf{E}\mathbf{x}_{i}^{0}=0$ (3)

This is done inside a RANSAC scheme to reduce the influence of outliers coming
from the data association.

  3. 3.

Recover the relative camera motion $\mathbf{T}_{C^{t}C^{0}}$ from
$\mathbf{E}$. This will yield to 4 motion hypothesis (2 rotations and 2
translations). We are using close frames to initialize, hence the camera
rotation should be small so we can safely select the smallest rotation to
solve the rotation ambiguity. Finally we disambiguate the translation
component by selecting the one that yields to the highest number of points in
front of both cameras.

  4. 4.

Reconstruct environment using the camera motion recovered. For that, we use
the Inverse Depth Weighted Midpoint [21] to triangulate tracked features as it
provides low 3D-2D errors in low parallax scenarios.

###  III-D Camera pose prediction

To encode assumption (4), we first estimate a preliminar camera pose
$\mathbf{T}_{C^{t}W}$ for time $t$ prior estimating any deformation. We assume
that the camera follows a physical model of constant velocity. This provides
us with an initial guess of the camera pose that will then be refined using
Non Linear Least Squares (NLLS) using a reprojection error with the
environment geometry observed in the previous frame $t-1$. This can be seen as
a way to attribute to a camera motion most of the image innovation seen. Note
that this is not the final pose we compute but just a seed for our global
optimization for the deformations and camera pose detailed in the next
section.

###  III-E Tracking camera pose and deformation

Our goal is, given some feature matches in the current frame
$\mathbf{u}_{i}^{t}$ and the 3D reconstruction in the previous temporal
instant $\mathbf{X}_{i}^{t-1}$, to estimate the current camera pose
$\mathbf{T}_{C^{t}W}$ and the deformation $\boldsymbol{\delta}_{i}^{t}$ of
each point such as the current scene can be estimated as
$\mathbf{X}_{i}^{t}=\mathbf{X}_{i}^{t-1}+{\boldsymbol{\delta}}_{i}^{t}$.

For that we introduce a reprojection data term $E^{t}_{i,rep}$ along with 2
regularizers $E^{t}_{i,spa}$ and $E^{t}_{i,tmp}$ to constrain the deformations
in our total global cost function, $\mathcal{E}^{t}$ for time $t$, defined by:

$\mathcal{E}^{t}=\sum_{i\in\mathcal{P}}E^{t}_{i,rep}+\lambda_{spa}E^{t}_{i,spa}+\lambda_{tmp}E^{t}_{t,tmp}$
(4)

where $\mathcal{P}$ represents the set of points being observed in the current
frame. Our global problem can be solved using Non-Linear Squares optimization
such us:

$\mathbf{T}_{C^{t}W},{\boldsymbol{\delta}}_{i}^{t}=\underset{\mathbf{T}_{C^{t}W},{\boldsymbol{\delta}}_{i}^{t}}{\operatorname*{arg\,min}}\
{\mathcal{E}^{t}}$ (5)

Next we define the terms of our cost function $\mathcal{E}^{t}$.

####  III-E1 Reprojection term

we obtain feature matches $\mathbf{u_{i}^{t}}$ in the current frame with the
modified Lucas-Kanade algorithm presented in [6], computing on this way the
reprojection error as it follows:

$E^{t}_{i,rep}=\rho\big{(}\|\mathbf{u}_{i}^{t}-\mathbf{\hat{u}}_{i}^{t}\|^{2}_{\Sigma{rep}}\big{)}$
(6)

where $\rho$ is the Hubber robust cost, $\mathbf{\hat{u}}_{i}^{t}$ and
$\mathbf{u}_{i}^{t}$ are respectively the match of feature $i$ in the current
image $I_{t}$ and its projection given by:

$\mathbf{\hat{u}}_{i}^{t}=\Pi(\mathbf{T}_{C^{t}C^{0}}(\Pi^{-1}(\mathbf{u}_{i}^{0},d_{i})+{\mathbf{X}_{i}^{t-1}+\boldsymbol{\delta}}_{i}^{t})$
(7)

The accuracy of indirect methods is limited by the feature detector resolution
(typically no less than 1 pixel). However matches obtained with photometric
methods have subpixel accuracy boosting on this way the accuracy of our
reprojection term while keeping its nice convergence basin.

####  III-E2 Spatial regularizer

Following [11] we encode assumption (2) with a regularizer that constrains
deformations locally so they are spatially smooth:

$E^{t}_{i,spa}=\sum_{j\in\mathcal{G}(i)}\rho\big{(}\|w^{t}_{ij}({\boldsymbol{\delta}}_{i}^{t}-{\boldsymbol{\delta}}_{j}^{t})\|^{2}_{\Sigma_{spa}}\big{)}$
(8)

Here $\mathcal{G}$ represents a weighted graph that encodes related points
whose deformations should be regularized together. The weight in $\mathcal{G}$
of two connected points $i$ and $j$ is $w^{t}_{ij}$ which depends on the
Euclidean distance between both points at the immediately previous time
instant $t-1$ and is computed according to the following formula:

$w^{t}_{ij}=\exp\left({\frac{-\left\|\mathbf{X}_{i}^{t-1}-\mathbf{X}_{j}^{t-1}\right\|^{2}}{2\sigma^{2}}}\right)$
(9)

where $\sigma$ is a radial basis weight that controls the influence radius of
each point. This regularizer is crucial as it enforces as rigid-as-possible
deformations and contributes towards a global consistency of the deformations.

####  III-E3 Temporal regularizer

Finally we add a temporal regularizer on the deformations to represent that
they occur slowly over time (assumption (3)):

$E^{t}_{i,tmp}=\rho\big{(}\|{\boldsymbol{\delta}}_{i}^{t}\|^{2}_{\Sigma_{tmp}}\big{)}$
(10)

This regularizer also interacts with assumption (4) as it penalizes big
deformations that could be explained with a camera motion.

##  IV EXPERIMENTS

We evaluate our method in a Minimal Invasive Surgery sequences, more
specifically in colonoscopies. This kind of sequences pose big challenges as
they exhibit continuous deformations, poor textures and harsh illumination
conditions. We provide quantitative results in photorealistic synthetic data
and qualitative experiments with in-vivo human colonoscopies. For comparison
purposes we also test our method in the Hamlyn dataset using its stereo setup
to evaluate our reconstructions against other state-of-the-art methods. A
summary of our main results can be seen in Fig 3.

(a)

(b)

(c)

(d)

Figure 3: Results of our algorithm for different sequences. Per each sequence,
in columns results after the initial middle and final frame. Two rows per
sequence. The first row displays the 3D reconstruction: black points are the
undeformed map, red lines are the map point deformation trajectories, in blue
the camera trajectory. The second row the RGB frames with the tracked features
in green. From top to bottom: (a) EndoMapper real in-vivo sequence, (b)
Simulated sequence, (c) Hamlyn 20 sequence and (d) Hamlyn 21 sequence. All
datasets have been processed using only monocular images.

###  IV-A Implementation details

We implement the monocular map initialization, camera pose and deformation
estimation in C++. For non Linear Squares optimization we use the Levenberg-
Marquart algorithm implemented in the g2o library [22]. For feature extraction
and matching we implement our own Shi-Tomasi feature extractor and Lucas-
Kanade tracker (Eq. 1). We set a threshold of 0.8 for the SSIM score (Eq. 2)
to detect and reject spurious feature tracks. Regarding the optimization, we
set $\Sigma_{rep}$ to 1 pixel, $\Sigma_{spa}$ and $\Sigma_{tmp}$ to 10 mm.
Since $\Sigma_{spa}$ and $\Sigma_{tmp}$ correctly scales the $E_{spa}$ and
$E_{tmp}$ terms, we set their respective $\lambda$ to 1. Finally, for the
Hubber cost threshold we use the 95 percentile of $\chi^{2}$, with 2 DoF for
$E_{rep}$ and with 3 DoF for $E_{spa}$ and $E_{tmp}$.

Regarding the regularization graph $\mathcal{G}$, for each point we only add
regularization terms with its $K=20$ closest points in 3D with $\sigma$ set to
15 mm when initializing the map with the monocular camera and 55 mm when using
the stereo to get the first map reconstruction. This is done to ensure that a
points is always regularized and at the same time ignoring points that have
little influence with the current point to reduce the computational burden.

###  IV-B Simulated Colon dataset

We use the VR-Caps [8] to generate photerealistic synthetic image sequences of
a 3D colon model obtained from a Computed Tomography. Since this is a
simulation, we have access full to camera pose and 3D scene ground truth.
Indeed, we can generate sequences with different camera trajectories and
degrees of deformation enabling us to test each one of the components of our
system individually.

For evaluation purposes, we simulate an insertion maneuver (Fig. 3(b)) with
different degrees of deformation. We model the deformations via a sine wave
propagating along the simulated colon. We apply this deformations to the $y$
coordinate of the point surfaces simulating perilstatic movements according to
the following formula:

$V_{y}^{t}=V_{y}^{0}+A\sin(\omega t+V_{x}^{0}+V_{y}^{0}+V_{z}^{0})$ (11)

where $V_{x}^{0}$, $V_{y}^{0}$ and $V_{z}^{0}$ are the coordinates of the
surface point at rest. We can control the magnitude and velocity of the
deformations according to the parameters $A$ and $\omega$ respectively. Table
I shows the reconstruction accuracy of our system in the simulated sequence
with different deformation velocities and amplitudes. The error shown is the
Root Mean Square Error (RMSE) of the reconstructed points for all frames
according to:

$e_{\text{rms}}=\sqrt{\frac{\sum_{i}\|s^{t}\mathbf{\hat{X}}_{i}^{t}-\mathbf{X}^{t,{gt}}_{i}\|^{2}}{n}}$
(12)

Since this is a full monocular formulation, we find, for each frame, an
optimal scale factor, $s^{t}$, to align our reconstructions with the ground
truth.

TABLE I: Reconstruction RMSE (mm) in simulated colonoscopies [23] for different deformation types $A$ (mm) $\omega$ (rad/s) | 0 | 2.5 | 5
---|---|---|---
0 | 1.15 | - | -
2.5 | - | 1.77 | 1.70
5 | - | 1.84 | 3.65
10 | - | 2.27 | 4.57

Results show that our formulation can reach nice reconstruction error around
2-3 mm even though in presence of deformations. One interesting result is that
our system is more sensitive to deformation velocities than the magnitude
itself being aligned with assumption (3).

TABLE II: Comparison with previous methods in sequences 20 and 21 from Hamlyn
Dataset as shown in [17].
We report reconstruction RMSE (mm) and number of frames processed. |  | Stereo Initialization | Monocular Init.
---|---|---|---
|  | ORB-SLAM3 [4] | SD-DefSLAM [6] | DSDT [17] |  | Ours
---
| Ours
---
20 | RMSE | 1.37 | 4.68 | 2.9 | 1.48 | 2.79
# Fr. | 220 | 252 | 500 | 350 | 350
21 | RMSE | - | 6.19 | 1.3 | 1.55 | 3.31
# Fr. | - | 323 | 300 | 300 | 300

###  IV-C Hamlyn dataset

We also test our formulation in real endoscopic sequences. For that purpose,
we use sequences 20 (Fig. 3(c)) and 21 (Fig. 3(d)) of the Hamlyn dataset [10].
Sequence 20 (from frame #750) corresponds to abdominal exploration with slow
deformation. Sequence 21 (also from frame #750) images a liver with 2 lobes
each of them moving on its own. This can be considered as an articualted
motion. In both sequences, surface texture is poor and illumination conditions
are unfavorable. This dataset is recorded with a stereo endoscope, allowing us
to estimate environment groundtruth from the disparity observed by the stereo
sensor.

We evaluate our formulation in 2 setups (Table II) for comparison purposes. In
the first setup, we initialize our system with the first stereo images and
perform monocular tracking, in order to allow comparison with previous methods
ORB-SLAM [1], SD-DefSLAM [6] and Direct and Sparse Deformable Tracking (DSDT)
[17]. Since we are initializing from the stereo images, we do not perform any
scale alignment when computing the RMSE. We achieve competitive results
regarding reconstruction error, obtaining a consistent error around 1.5 mm.
Since we do not impose any restriction on the surface topology or in the
deformations, we achieve a significantly smaller error compared with SD-
DefSLAM. This is specially clear in sequence 21 with the 2 lobes moving
independently what limits the accuracy of SD-DefSLAM. The comparison with DSDT
suggests that our regularizers are versatile, we are able to code better the
spatial smoothness of sequence 20, achieving a lower error, while still being
competitive in hard discontinuity of sequence 21. DSDT is able to keep the
track longer because, in contrast with DSDT, our method still does not
implement any policy to recover points lost during tracking.

The second setup uses the full monocular pipeline including our monocular
initalization (Sec. III-C) computing the RMSE after a per frame scale
correction (Eq. 12). In this scenario, our system reaches errors around
2.8-3.3 mm which is aligned with the errors obtained in the simulation dataset
under significant deformations. The increase in error compared with the stereo
setup is due to the quality of the map initialization that no longer relies on
a perfect stereo initialization.

Also it is important to note that in these sequences the surfaces shape and
deformations observed are completely different from the ones seen in the
simulation dataset proving that we can model general surface shapes and
deformations.

###  IV-D Real endoscopy sequences

We provide qualitative results in real in-vivo human colonoscopy sequences
from the EndoMapper project [9]. These sequences display the big challenges
real colonoscopies pose, such as deformation, little to no texture in the
images, lighting conditions varying from frame to frame, reflections and fish-
eye optics (fig. 1.)

In this case, there is no ground truth to compare with, because the dataset
just records standard monocular endoscope procedures. For this reason, we only
provide qualitative results. Fig. 2 displays how we are able to initialize
maps with high density of points form quite close (3 frames apart) frames
capturing the tubular topology of the colon. In Figures 1 and 3(a), it can be
seem how our algorithm is able to capture the scene deformation and the
endoscope trajectory, being able to track map points for more than 30 frames
in two examples of real colonoscopies of two different patients.

##  V CONCLUSIONS

In this work we have presented an approach for monocular camera tracking and
deformation estimation without assumptions on the environment shape or
topology. Instead, we successfully encode with simple regularizers the
assumptions about the type of deformations that are common in endoscopy.
Compared with the state of the art, and our method, including map
initialization, is applicable in a much wider range of shape and topologies,
like colonoscopies, while having similar accuracy in more standard almost-
planar scenarios.

The presented monocular initialization and tracking contributes to make real a
fully deformable SLAM system. The deformable mapping to expand the map as the
camera explores new regions is closer after our contribution, being a
promising venue of future work in the short term. In the mid-term, a multi-map
deformable SLAM offers a profitable for future work because will be able to
cope with occlusions and tracking losses prevalent in real colonoscopies.

## References

  * [1] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardós, “ORB-SLAM: a versatile and accurate monocular SLAM system,” _IEEE Transactions on Robotics_ , vol. 31, no. 5, pp. 1147–1163, 2015.
  * [2] J. Engel, V. Koltun, and D. Cremers, “Direct sparse odometry,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 3, pp. 611–625, 2017.
  * [3] C. Forster, Z. Zhang, M. Gassner, M. Werlberger, and D. Scaramuzza, “SVO: Semidirect visual odometry for monocular and multicamera systems,” _IEEE Transactions on Robotics_ , vol. 33, no. 2, pp. 249–265, 2016.
  * [4] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós, “ORB-SLAM3: An accurate open-source library for visual, visual–inertial, and multimap SLAM,” _IEEE Transactions on Robotics_ , vol. 37, no. 6, pp. 1874–1890, 2021.
  * [5] J. Zubizarreta, I. Aguinaga, and J. M. M. Montiel, “Direct sparse mapping,” _IEEE Transactions on Robotics_ , vol. 36, no. 4, pp. 1363–1370, 2020.
  * [6] J. J. Gómez-Rodríguez, J. Lamarca, J. Morlana, J. D. Tardós, and J. M. Montiel, “SD-DefSLAM: Semi-direct monocular SLAM for deformable and intracorporeal scenes,” in _IEEE Int. Conf. on Robotics and Automation (ICRA)_ , 2021, pp. 5170–5177.
  * [7] J. Lamarca, S. Parashar, A. Bartoli, and J. Montiel, “DefSLAM: Tracking and mapping of deforming scenes from monocular sequences,” _IEEE Transactions on robotics_ , vol. 37, no. 1, pp. 291–303, 2020.
  * [8] K. İncetan, I. O. Celik, A. Obeid, G. I. Gokceler, K. B. Ozyoruk, Y. Almalioglu, R. J. Chen, F. Mahmood, H. Gilbert, N. J. Durr _et al._ , “Vr-caps: a virtual environment for capsule endoscopy,” _Medical image analysis_ , vol. 70, p. 101990, 2021.
  * [9] “Endomapper project,” https://sites.google.com/unizar.es/
endomapper/home, accessed: 2022-02-28.

  * [10] P. Mountney, D. Stoyanov, and G.-Z. Yang, “Three-dimensional tissue deformation recovery and tracking,” _IEEE Signal Processing Magazine_ , vol. 27, no. 4, pp. 14–24, 2010.
  * [11] R. A. Newcombe, D. Fox, and S. M. Seitz, “DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 343–352.
  * [12] M. Innmann, M. Zollhöfer, M. Nießner, C. Theobalt, and M. Stamminger, “VolumeDeform: Real-time volumetric non-rigid reconstruction,” in _European Conference on Computer Vision_. Springer, 2016, pp. 362–379.
  * [13] M. Slavcheva, M. Baust, and S. Ilic, “SobolevFusion: 3D reconstruction of scenes undergoing free non-rigid motion,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 2646–2655.
  * [14] ——, “Variational level set evolution for non-rigid 3D reconstruction from a single depth camera,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 8, pp. 2838–2850, 2020.
  * [15] J. Song, J. Wang, L. Zhao, S. Huang, and G. Dissanayake, “Dynamic reconstruction of deformable soft-tissue with stereo scope in minimal invasive surgery,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 1, pp. 155–162, 2017.
  * [16] H. Zhou and J. Jayender, “EMDQ-SLAM: Real-time high-resolution reconstruction of soft tissue surface from stereo laparoscopy videos,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2021, pp. 331–340.
  * [17] J. Lamarca, J. J. G. Rodriguez, J. D. Tardos, and J. Montiel, “Direct and sparse deformable tracking,” _arXiv preprint arXiv:2109.07370_ , 2021.
  * [18] B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in _Proceedings of the 7th International Joint conference on Artificial Intelligence_ , vol. 2, 1981, p. 674–679.
  * [19] J. Shi and C. Tomasi, “Good features to track,” in _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition_ , 1994, pp. 593–600.
  * [20] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [21] S. H. Lee and J. Civera, “Triangulation: why optimize?” _arXiv preprint arXiv:1907.11917_ , 2019.
  * [22] G. Grisetti, R. Kümmerle, H. Strasdat, and K. Konolige, “g2o: A general framework for (hyper) graph optimization,” in _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2011, pp. 9–13.
  * [23] K. B. Ozyoruk, G. I. Gokceler, G. Coskun, K. Incetan, Y. Almalioglu, F. Mahmood, E. Curto, L. Perdigoto, M. Oliveira, H. Sahin, H. Araujo, H. Alexandrino, N. J. Durr, H. B. Gilbert, and M. Turan, “Endoslam dataset and an unsupervised monocular visual odometry and depth estimation approach for endoscopic videos: Endo-sfmlearner,” 2020.


# Automatic Anonymization of Swiss Federal Supreme Court Rulings

Joel Niklaus${}^{1,2,3\;}\thanks{\hskip 5.69054ptEqual contribution.}$ Robin
Mamié${}^{4\;*}$ Matthias Stürmer1,2 Daniel Brunner4 Marcel Gygli2
1University of Bern 2Bern University of Applied Sciences
3Stanford University 4Swiss Federal Supreme Court
Equal contribution.

###### Abstract

Releasing court decisions to the public relies on proper anonymization to
protect all involved parties, where necessary. The Swiss Federal Supreme Court
relies on an existing system that combines different traditional computational
methods with human experts. In this work, we enhance the existing
anonymization software using a large dataset annotated with entities to be
anonymized. We compared BERT-based models with models pre-trained on in-domain
data. Our results show that using in-domain data to pre-train the models
further improves the F1-score by more than 5% compared to existing models. Our
work demonstrates that combining existing anonymization methods, such as
regular expressions, with machine learning can further reduce manual labor and
enhance automatic suggestions.

Automatic Anonymization of Swiss Federal Supreme Court Rulings

Joel Niklaus${}^{1,2,3\;}\lx@make@thanks{\hskip 5.69054ptEqualcontribution.}$
Robin Mamié${}^{4\;*}$

Matthias Stürmer1,2 Daniel Brunner4 Marcel Gygli2

1University of Bern 2Bern University of Applied Sciences 3Stanford University
4Swiss Federal Supreme Court

Figure 1: Main window of Anom2. Anonymizations are configured on the left, and
the anonymized court ruling appears on the right. The system highlights
completed anonymizations in gold and the current setting in yellow.

##  1 Introduction

The Swiss Federal Supreme Court (SFSC) is the highest judicial authority in
Switzerland. It is the final arbiter in legal disputes and ensures the uniform
application of federal law throughout the country. It consists of several
divisions specialized in different areas of law, including civil, criminal,
administrative, and social security matters Glaser et al. (2021). In a year,
the SFSC roughly handles 7K cases and publishes its rulings. In this process,
personal information must be anonymized from the rulings in order to protect
involved parties. In the traditional setting, court rulings are anonymized by
skilled experts. This task is highly complex, as the removal/anonymization of
a word is dependent on the context it is written in. For example, Zuerich
needs to be removed if it is part of the name of the legal entity "Zurich
Insurance Group", but not if it is a reference to the city. At the SFSC,
experts are already supported in their work through an application called
Anom2 (see Figure 1). Anom2 provides access to various methods and algorithms
for finding and replacing text entities (e.g., with regular expressions). The
aim of this work is to enhance the capabilities of Anom2 with Machine Learning
capabilities that provide the user with more suggestions that need to be
anonymized. Our results show that this approach allows users to find more
elements that require anonymization.

##  2 Related Work

For identifying elements that might require anonymization, a process called
Named Entity Recognition (NER) is employed. Traditionally, NER recognizes and
categorizes text parts according to a set of semantic categories like Location
(LOC), Organization (ORG), or Person (PER) Benikova et al. (2014). As these
classes are not enough for the anonymization of court cases Leitner et al.
(2020) suggested expanding this list to seven coarse and 19 fine-grained
classes, including entities such as Judge (RR), or Lawyer (AN). Using this
dataset, Darji et al. (2023) fine-tuned GermanBERT Chan et al. (2020), clearly
outperforming a BiLSTM-CRF+ model. Similar approaches have been applied and
tested in other languages, such as Romanian Pais et al. (2021), Greek
Angelidis et al. (2018), Portuguese Luz de Araujo et al. (2018), and
multilingually de Gibert et al. (2022); Niklaus et al. (2023a).

Domain-specific pretraining has flourished in the legal domain recently.
Chalkidis et al. (2020) pretrained LegalBERT on EU and UK legislation, ECHR
and US cases, and US contracts. Zheng et al. (2021) pretrained CaseHoldBERT on
US case law, while Henderson et al. (2022) trained PoL-BERT on the 256 GB Pile
of Law corpus. Niklaus and Giofré (2022) pretrained Longformer Beltagy et al.
(2020) models using the Replaced Token Detection (RTD) task on the Pile of
Law. Hua et al. (2022) used RTD to pretrain Reformer Kitaev et al. (2020)
models on 6 GB of US case law. Finally, Niklaus et al. (2023b) released a
large multilingual legal corpus and trained various legal models. We continue
pretraining the German, French, and Italian models for 800K and 300K steps
more for base and large models, respectively. Rasiah et al. (2023) pretrain
models on Swiss legal data, termed Legal-Swiss-RoBERTa.

Document anonymization has a long tradition in the medical domain, where
personal data need to be removed from documents. Initially, this task was
handled using methods such as semantic lexicons Ruch et al. (2000) or regular
expressions to replace text occurrences. Recently, this has been expanded to
include BERT-style models as well Mao and Liu (2019). In the legal domain,
Glaser et al. (2021) worked on 1400 anonymized German rulings. Using already
anonymized rulings, they trained different Recurrent Neural Networks (RNN)
using BERT embeddings. Using this approach, they achieved a maximum of 68.9%
precision and 79.1% recall rates. Garat and Wonsever (2022) performed similar
work on 80K documents from Uruguayan courts. Our work specifically tackles
court decisions by the SFSC. We compare the generic cased mBERT model Devlin
et al. (2019) with models pre-trained on in-domain data (such as Legal-Swiss-
RoBERTa-base Rasiah et al. (2023)). We also investigate monolingual model
performance in the three languages of the SFSC rulings: German, French, and
Italian.

Much prior work used SFSC cases as data for their research because of wide
availability in three languages, giving good coverage of the most important
Swiss case law. Niklaus et al. (2021, 2022) introduced and studied judgment
prediction on SFSC rulings. Brugger et al. (2023) investigated and improved
multilingual sentence boundary detection in the legal domain using SFSC
decisions. Christen et al. (2023) studied negation scope resolution and
Nyffenegger et al. (2023) investigated how easily LLMs can re-identify persons
occurring in anonymized SFSC decisions. Rasiah et al. (2023) created a large
benchmark of ten text classification tasks, two text generation tasks, an
information retrieval, and a citation extraction task.

(a) Token Distribution

(b) Anonymized Token Distribution

(c) Entity Distribution

(d) Anonymized Entity Distribution

Figure 2: Histograms illustrating the distribution of (anonymized) tokens and
entities across the three languages.

##  3 Dataset

We used 119156 rulings (77262 German, 40099 French, 6795 Italian) Supreme
court decisions and split them into sentences using Spacy Honnibal et al.
(2020). We prepared the decisions for NER based on the manual labels from the
paralegals who performed manual anonymizations. The histograms in Figure 2
illustrate the distribution of four key measures, namely, Number of Tokens,
Anonymized Tokens, Entities, and Anonymized Entities, in three languages:
German (de), French (fr), and Italian (it). Different color schemes for each
language enhance the visual interpretability of the plots. Measures concerning
tokens and entities exhibit a long-tailed distribution, signifying a
concentration of instances at the lower end of the value spectrum.
Specifically, the distribution of Number of Tokens and Number of Entities is
examined within a 10 to 100,000 range, capturing their broad spread. In
contrast, anonymized tokens and entities are evaluated within a 1 to 10,000
range, reflecting their constrained distribution.

Table 1: Evaluation Results. Best results per setup are in bold.

Model | Normal | Uniformizing
---|---|---
Precision | Recall | F1-Score | Precision | Recall | F1-Score
Multilingual Models |  |  |  |  |  |
bert-base-multilingual-cased | 90.72 | 83.76 | 87.10 | 85.85 | 94.95 | 90.17
Legal-XLM-RoBERTa-base | 94.84 | 81.98 | 87.94 | 89.93 | 86.85 | 88.36
Legal-Swiss-RoBERTa-base | 92.26 | 92.57 | 92.42 | 83.13 | 94.85 | 88.60
Monolingual Models |  |  |  |  |  |
bert-base-german-cased | 95.14 | 80.00 | 86.92 | 91.49 | 85.86 | 88.58
Legal-German-RoBERTa-base | 95.40 | 80.09 | 87.07 | 89.20 | 84.97 | 87.03
dbmdz/bert-base-french-europeana-cased | 95.86 | 81.84 | 88.30 | 88.92 | 89.14 | 89.03
Legal-French-RoBERTa-base | 95.45 | 83.48 | 89.06 | 88.77 | 89.17 | 88.97
dbmdz/bert-base-italian-cased | 93.49 | 80.21 | 86.35 | 76.71 | 83.85 | 80.12
Legal-Italian-RoBERTa-base | 94.16 | 80.59 | 86.85 | 84.03 | 84.06 | 84.05

##  4 Legal Pretraining

To improve the SFSC anonymization system, we pretrained legal-specific models
on diverse legal text in German, French, and Italian.

(a) We warm-start (initialize) our models from the original XLM-R checkpoints
(base or large) of Conneau and Lample (2019). Model recycling is a standard
process followed by many Wei et al. (2021); Ouyang et al. (2022) to benefit
from starting from an available “well-trained” PLM, rather from scratch
(random). XLM-R was trained on 2.5 TB of cleaned CommonCrawl data in 100
languages.

(b) We train a new tokenizer of 32K BPEs on the training subsets to better
cover legal language. However, we reuse the original XLM-R embeddings for all
lexically overlapping tokens Pfeiffer et al. (2021), i.e., we warm-start word
embeddings for tokens that already exist in the original XLM-R vocabulary, and
use random ones for the rest.

(c) We continue pretraining our monolingual models on our pretraining corpus
with batches of 512 samples for an additional 1M/500K steps for the base/large
model. We do initial warm-up steps for the first 5% of the total training
steps with a linearly increasing learning rate up to $1e\\!-\\!4$, and then
follow a cosine decay scheduling, following recent trends. For half of the
warm-up phase (2.5%), the Transformer encoder is frozen, and only the
embeddings, shared between input and output (MLM), are updated. We also use an
increased 20/30% masking rate for base/large models respectively, where also
100% of the predictions are based on masked tokens, compared to Devlin et al.
(2019)111Devlin et al. (2019) – and much follow-up work – used a 15% masking
ratio, and a recipe of 80/10/10% of predictions made across masked/randomly-
replaced/original tokens., based on the findings of Wettig et al. (2023).

(d) We consider mixed cased models, i.e., both upper- and lowercase letters
covered, similar to recently developed large PLMs Conneau and Lample (2019);
Raffel et al. (2020); Brown et al. (2020).

(e) This leaves us with two models for each language (base and large).
Additionally, we consider the multilingual legal models pretrained by Niklaus
et al. (2023b) and the Swiss legal models pretrained by Rasiah et al. (2023).

##  5 Anonymization System

The SFSC employs an anonymization system, Anom2, to assist paralegals in
anonymizing rulings for public access. The main UI is shown in Figure 1. Upon
loading a ruling, the application auto-identifies terms requiring
anonymization and lists them on the left, along with replacement text. The
search function allows direct term marking for anonymization. Anom2 uses
different algorithms for the search for text that needs to be anonymized:
Conventional is based on a statistical analysis of the loaded ruling. Using
polyglot222See: https://polyglot.readthedocs.io an initial set of named
entities is detected. Using the specific knowledge of the format, the rubrum
is dynamically detected, allowing for the labelling of important names and
addresses.
BERT performs the recognition of entities to be anonymized using a BERT Devlin
et al. (2019) model fine-tuned for NER. Entity recognition is performed on the
sentence level, as the rulings are often too long for the model. This approach
could lead to inconsistencies in recognition, as a term identified in one
sentence might not be identified in another. This is solved in post-
processing, where any identified term is automatically anonymized in the whole
document.
Legal-Swiss-RoBERTa-base works analogously to the BERT method, but uses a
fine-tuned Legal-Swiss-RoBERTa-base Rasiah et al. (2023) model.

##  6 Experimental Setup

We used the following hyperparameters for all evaluated models: batch size of
64, learning rate of 5e-5, and weight decay of 0.01. We employed the seqeval
metric for evaluation. We set the maximum sequence length to 192 tokens, which
we determined to be the optimal trade-off between average sentence size and
training time for computational efficiency. We used early stopping based on
the F1-score of the validation set, which constitutes 10% of the entire
dataset, following an 80-10-10 split for the training, validation, and test
sets, respectively. Training ceases once the F1-score on the validation set
starts to decline. Due to resource constraints (we only had two Tesla T4 GPUs)
we could only run one random seed per model.

We define and configure two special parameters: 1) _TruncationStrideRatio_ :
We set this parameter to 0.5. When a sentence exceeds 192 tokens, we truncate
it using a specific overlap strategy. The overlap consists of half of the
previous snippet and half of the next snippet.
2) _NonAnonymizedSentencesRatioToAnonymizedSentences_ : We set the ratio at
1.5, including only 150% of sentences without anonymization examples compared
to those with examples. This minimizes data redundancy and maximizes utility.

##  7 Results

Table 1 presents a comprehensive evaluation of various BERT and RoBERTa-based
models on two different conditions: Normal and Uniformizing. For the Normal
condition, in the multilingual setting, Legal-XLM-RoBERTa-base exhibits the
highest Precision at 94.84%, while Legal-Swiss-RoBERTa-base demonstrates
superior Recall and F1-Score values, achieving 92.57% and 92.42% respectively.
With Uniformizing, we describe the process of forcing the model to replace all
occurrences of a detected term across the whole document. This approach leads
to better Recall, but reduces Precision. In the Uniformized case, again Legal-
XLM-RoBERTa-base shows highest Precision, while mBERT achieves highest Recall
and F1-Score. The improved Recall and F1-Score in the Normal condition show
that pre-training on legal data can improve the performance of models. We
observe similar behavior for the monolingual models. All models pre-trained on
legal data achieve a higher F1-Score than generic monolingual models.

##  8 Discussion

We pretrained models on Swiss legal data and performed a detailed comparison
of legal and generic models, both multilingually and monolingually in the
ruling anonymization task. Our experiments indicate that pretraining on legal
data improves the performance of models significantly compared to generic
multi- or monolingual models.

To reduce errors in sentence splitting, we suggest future work to use legal
specific sentence splitters Brugger et al. (2023). Due to computational
constraints we only experimented with base size encoder models. Future work
may expand this by also testing larger models.

## Acknowledgements

We greatly appreciate Google’s generous support of TPUs v3-8 machines for
pretraining the models. This work has been partially supported by the Swiss
National Research Programme “Digital Transformation” (NRP-77) grant number
187477.

## References

  * Angelidis et al. (2018) I. Angelidis, Ilias Chalkidis, and M. Koubarakis. 2018.  Named Entity Recognition, Linking and Generation for Greek Legislation.  In _JURIX_.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Benikova et al. (2014) Darina Benikova, Chris Biemann, and Marc Reznicek. 2014.  NoSta-D Named Entity Annotation for German: Guidelines and Dataset.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 2524–2531, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Brugger et al. (2023) Tobias Brugger, Matthias Stürmer, and Joel Niklaus. 2023.  MultiLegalSBD: A Multilingual Legal Sentence Boundary Detection Dataset.  ArXiv:2305.01211 [cs].
  * Chalkidis et al. (2020) Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020.  LEGAL-BERT: The Muppets straight out of Law School.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2898–2904.
  * Chan et al. (2020) Branden Chan, Stefan Schweter, and Timo Möller. 2020.  German’s Next Language Model.  _arXiv:2010.10906 [cs]_.  ArXiv: 2010.10906.
  * Christen et al. (2023) Ramona Christen, Anastassia Shaitarova, Matthias Stürmer, and Joel Niklaus. 2023\.  Resolving Legalese: A Multilingual Exploration of Negation Scope Resolution in Legal Documents.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual Language Model Pretraining.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Darji et al. (2023) Harshil Darji, Jelena Mitrović, and Michael Granitzer. 2023.  German BERT Model for Legal Named Entity Recognition:.  In _Proceedings of the 15th International Conference on Agents and Artificial Intelligence_ , pages 723–728, Lisbon, Portugal. SCITEPRESS - Science and Technology Publications.
  * de Gibert et al. (2022) Ona de Gibert, A García-Pablos, Montse Cuadros, and Maite Melero. 2022.  Spanish datasets for sensitive entity detection in the legal domain.  In _Proceedings of the Thirteenth International Conference on Language Resources and Evaluation (LREC’22), Marseille, France, june. European Language Resource Association (ELRA)_.  Dataset URL: https://tinyurl.com/mv65cp66.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Garat and Wonsever (2022) Diego Garat and Dina Wonsever. 2022.  Automatic Curation of Court Documents: Anonymizing Personal Data.  _Information_ , 13(1):27.
  * Glaser et al. (2021) Ingo Glaser, Tom Schamberger, and Florian Matthes. 2021.  Anonymization of german legal court rulings.  In _Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law_ , pages 205–209, São Paulo Brazil. ACM.
  * Henderson et al. (2022) Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. 2022.  Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset.  ArXiv:2207.00220 [cs].
  * Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.  spaCy: Industrial-strength Natural Language Processing in Python.
  * Hua et al. (2022) Wenyue Hua, Yuchen Zhang, Zhe Chen, Josie Li, and Melanie Weber. 2022.  LegalRelectra: Mixed-domain Language Modeling for Long-range Legal Text Comprehension.  ArXiv:2212.08204 [cs].
  * Kitaev et al. (2020) Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020.  Reformer: The Efficient Transformer.  _arXiv:2001.04451 [cs, stat]_.  ArXiv: 2001.04451.
  * Leitner et al. (2020) Elena Leitner, Georg Rehm, and Julián Moreno-Schneider. 2020.  A Dataset of German Legal Documents for Named Entity Recognition.  _arXiv:2003.13016 [cs]_.  ArXiv: 2003.13016.
  * Luz de Araujo et al. (2018) Pedro Henrique Luz de Araujo, Teófilo E. de Campos, Renato R. R. de Oliveira, Matheus Stauffer, Samuel Couto, and Paulo Bermejo. 2018.  LeNER-Br: A Dataset for Named Entity Recognition in Brazilian Legal Text.  In _Computational Processing of the Portuguese Language_ , Lecture Notes in Computer Science, pages 313–323, Cham. Springer International Publishing.
  * Mao and Liu (2019) Jihang Mao and Wanli Liu. 2019.  Hadoken: a BERT-CRF Model for Medical Document Anonymization.  In _Proceedings of the Iberian Languages Evaluation Forum co-located with 35th Conference of the Spanish Society for Natural Language Processing, IberLEF@SEPLN 2019, Bilbao, Spain, September 24th, 2019_ , volume 2421 of _CEUR Workshop Proceedings_ , pages 720–726. CEUR-WS.org.
  * Niklaus et al. (2021) Joel Niklaus, Ilias Chalkidis, and Matthias Stürmer. 2021.  Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark.  In _Proceedings of the Natural Legal Language Processing Workshop 2021_ , pages 19–35, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Niklaus and Giofré (2022) Joel Niklaus and Daniele Giofré. 2022.  BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch? ArXiv:2211.17135 [cs].
  * Niklaus et al. (2023a) Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Ilias Chalkidis. 2023a.  Lextreme: A multi-lingual and multi-task benchmark for the legal domain.
  * Niklaus et al. (2023b) Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, and Daniel E. Ho. 2023b.  MultiLegalPile: A 689GB Multilingual Legal Corpus.  ArXiv:2306.02069 [cs].
  * Niklaus et al. (2022) Joel Niklaus, Matthias Stürmer, and Ilias Chalkidis. 2022.  An Empirical Study on Cross-X Transfer for Legal Judgment Prediction.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 32–46, Online only. Association for Computational Linguistics.
  * Nyffenegger et al. (2023) Alex Nyffenegger, Matthias Stürmer, and Joel Niklaus. 2023.  Anonymity at Risk? Assessing Re-Identification Capabilities of Large Language Models.  ArXiv:2308.11103 [cs].
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Pais et al. (2021) Vasile Pais, Maria Mitrofan, Carol Luca Gasan, Vlad Coneschi, and Alexandru Ianov. 2021.  Named Entity Recognition in the Romanian Legal Domain.  In _Proceedings of the Natural Legal Language Processing Workshop 2021_ , pages 9–18, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Pfeiffer et al. (2021) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2021.  UNKs everywhere: Adapting multilingual language models to new scripts.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10186–10203, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rasiah et al. (2023) Vishvaksenan Rasiah, Ronja Stern, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, Daniel E. Ho, and Joel Niklaus. 2023.  SCALE: Scaling up the Complexity for Advanced Language Model Evaluation.  ArXiv:2306.09237 [cs].
  * Ruch et al. (2000) P. Ruch, R. H. Baud, A. M. Rassinoux, P. Bouillon, and G. Robert. 2000.  Medical document anonymization with a semantic lexicon.  _Proceedings. AMIA Symposium_ , pages 729–733.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021.  Finetuned language models are zero-shot learners.  _CoRR_ , abs/2109.01652.
  * Wettig et al. (2023) Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2023.  Should you mask 15% in masked language modeling? In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 2985–3000, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Zheng et al. (2021) Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. 2021\.  When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset.  _arXiv:2104.08671 [cs]_.  ArXiv: 2104.08671 version: 3.