119–126

# The quest for collapsed/frozen stars in
single-line spectroscopic binary systems

Virginia Trimble  University of California Irvine,
Department of Physics and Astronomy
Irvine, California 92697, USA
email<EMAIL_ADDRESS>

(2014)

###### Abstract

Black holes are now commonplace, among the stars, in Galactic centers, and
perhaps other places. But within living memory, their very existence was
doubted by many, and few chose to look for them. Zeld́ovich and Guseinov were
first, followed by Trimble and Thorne, using a method that would have
identified HDE 226868 as a plausible candidate, if it had been in the 1968
catalogue of spectroscopic binaries. That it was not arose from an unhappy
accident in the observing program of Daniel M. Popper long before the
discovery of X-ray binaries and the identification of Cygnus X-1 with that
hot, massive star and its collapsed companion.

###### keywords:

Black holes, collapsed stars, Cygnus X-1, HDE 226868, spectroscopic binaries

††volume: 308††journal: The Zeld́ovich Universe
Genesis and Growth of the Cosmic Web††editors: A.C. Editor, B.D. Editor & C.E.
Editor, eds.

##  1 A Contextual Introduction

The context in which this presentation was assembled and written up includes a
recent book called Discovery and Classification in Astronomy: Controversy and
Consensus (Dick 2013) in which the author states at least twice that black
holes were not taken seriously by astronomers until the mid 1970’s or later.
Now the two widely-cited pioneering papers on powering of quasars by accretion
on a central supermassive, compact object came from E.E. Salpeter (1964) and
Ya. B. Zeld́ovich & I.D. Novikov (1964b) which had me doubting the claims
immediately. This was reinforced by the sudden memory that one of my own
papers (Trimble & Thorne 1969) took them seriously rather earlier than the
mid-1970’s, and that it had its origins in an idea from Zeld́ovich & Guseinov
(1966). There followed a resolve to look back at this topic and offer an oral
version of it to the June, 2014 Zeld́ovich Centenary conference in Moscow.
Complexities associated with the June 15th end of the school year at UCI kept
me from attending that meeting, and the organizers of IAU Symposium 308 the
next week generously made room for “The Quest for Collapsed Stars” just before
the conference dinner on the second to last day of the meeting.

##  2 Cast of Characters

A Newtonian version of the story can be pushed back to publications in the
late 18th century, when John Michell and Pierre-Simon de Laplace noted the
possibility of astronomical entities with escape velocity equaling or
exceeding the speed of light. Relativity enters the picture with a pair of
much-cited, but rarely read, papers by Karl Schwarzschild (1916a, b,) that
defined what we call the Schwarzschild radius and was already being called the
gravitational radius when Oppenheimer and Snyder (1939) wrote “On continued
gravitational contraction.”

They wrote, and we all now agree, that a sufficiently massive star, having
exhausted its thermonuclear sources of energy, must either reduce its mass to
of order that of the sun or suffer continuous contraction. Reaching the
gravitational radius takes a day or two for an observer riding the star, but
forever as seen from infinity, hence the phrase “frozen star” (which came over
as “cooled star” in some translations). The last photon that reaches outside
observers, however, departs within days, not millennia.

Then there was a war, with very little work on general relativity and its
implications being done anyplace for many years. On the far side of that
abyss, John Wheeler and a few Princeton students and colleagues began
exploring the territory near very massive compact objects, while in the USSR,
two critical early papers predicting inescapable collapse were published by
Zeld́ovich & Novikov (1964, 1965). We interrupt this tale to explain that
those names are in alphabetical order, in Russian, where the letter that
English-speakers transcribe as Z looks sort of like a backwards Greek epsilon
and comes early in the alphabet, while N looks like H and comes later.

The oral presentation in Tallinn included a number of images of people,
publications, and other pieces of paper not readily reproducible here, and
also a Sidney Harris cartoon (caption: It’s black, and it looks like a hole.
I’d say it’s a black hole) still under copyright. Figures 1 and 2 are an
inscription from the flyleaf of a 1985 book and a 1986 post-New Years letter,
and are included to demonstrate that I have a right to tell Zeld́ovich
stories.

Figure 1: Fly-leaf inscription on a copy of a 1985 volume of reprints of many
of Zeld́ovich’s significant papers across many fields, together with a fairly
complete list of all his publications up to that time. Nearly all in Russian,
of course. Figure 2: Copy of a February, 1986 letter recalling the three (and
only three) occasions on which he and Joe and/or I had been together. A
somewhat earlier letter told us of Angelica’s death and instructed Joe to take
good care of “Vergie.” A later one expressed regret that we had not been at
his NAS talk, and I have always hoped, though never been quite sure, that our
response arrived before he died.

The events referred to near the end of the letter are (1) the GRG meeting in
Warsaw in 1963, from which Richard Feynman wrote back to his new wife Gwyneth
expressing a considerable distaste for GR and the work being done in its name
(reproduced in the volume Surely You’re Joking, Mr. Feynman), (2) a later GRG
meeting in Jena, when Zeld́ovich and Joe Weber were both elected members of
the governing body, but only Joe attended its evening meeting, because at that
time Soviet representatives to international organizations could only be those
appointed by the government, so Jascha treated Angelica and me to sizable
quantities of vodka and heavily whip-creamed desserts, and (3) the General
Assembly of the International Astronomical Union in Patras, Greece in 1982,
where Zeld́ovich gave one of the Invited Discourses in a reconditioned ancient
Greek theatre. This was his first time in western Europe, marking his
achievement of what he called “the second cosmic velocity.” The first was
enough for a Soviet scientist to travel in Eastern Europe (hence to the Warsaw
and Jena meetings), and only the “third cosmic velocity” reached to the United
States.

The Patras conference dinner was also the occasion on which Zeld́ovich
expressed to me his puzzlement that Americans, who were allowed to read just
about anything, rarely bothered, while Russians would go to a good deal of
trouble to read things that were not readily available to them. Indeed we were
having a lively discussion about literature (necessarily in English!) when
another conference participant came up and asked Yakov Borisovich why he
wasn’t doing more to help Sakharov. Suddenly his proficiency in the
international language of science (broken English, he said) vanished, and he
professed not to understand the question.

A later letter expressed unhappiness that we had not been present when he
spoke at the US National Academy of Sciences during his first and only visit
to the US. I wrote back as soon as possible (letters between the USA and the
USSR could take a long time in those days, when they were presumably read by
both sides), explaining that merely being on the wrong coast in April would
never have kept us away, but that neither Joe nor I was, or would ever be, a
member of NAS! I heard from Oktay H. Guseinov once after the advent of email.
He died in 2009, having spent the latter years of his career in Azerbaijan and
Turkey.

The last preliminary remark was an explanation that John Archibald Wheeler
might well have invented the phrase “black hole,” but that he could not have
done so in 1967 or 1968, as is frequently written, for ’“Black Holes” in
Space’ is the title of a news item by Anne Ewing, published in Science News
Letters in 1964 (Ewing 1964). The item was a report of the December 1963
meeting of the American Association for the Advancement of Science, held in
Cleveland. There is also a rumor around that the phrase was heard at the very
first “Texas” symposium in Dallas, earlier in December 1963. Use of white
holes to reverse time has never proven practical, so the phrase mush have been
coined before those meetings.

##  3 The Catalogue Searches

Again one could start the story several places, but a good concrete point is
the pair of papers (Zeld́ovich & Novikov 1964, 1965), called “Relativistic
Astrophysics I and II” and presaging much of the information later found in
the two volumes of their similarly-titled book. The papers (at least the
English translations!) talk about neuron stars, continued gravitational
collapse, and collapsed or frozen stars (the latter comes out as cooled stars
in translation) with some calculations of likely accretion rates and energy
release. They cite the Oppenheimer papers, Baade & Zwicky (1934) on neutron
star formation as a supernova energy source, and Landau (1938) on whatever you
think that paper is about. Zeld́ovich (1964) which includes the idea of
accretion on a neutron star as a source of X-ray is also relevant.

Chapter 13 of “Relativistic Astrophysics II” includes a footnote crediting
Guseinov with the idea that detectability of compact stars is increased in
binary systems. If you think that I.S. Shklovsky should be part of this story
well, so apparently did he. And I refer you to Ginzburg (1990), who outlived
the other two and so got the last word, for yet another point of view.

At any rate, armed with the idea that an accreting collapsed star should be an
X-ray source but contribute no significant visible light in comparison with a
companion of normal luminosity, Guseinov & Zeld́ovich (1966, submitted 18
October 1965: Zeld́ovich & Guseinov 1966, submitted 19 November 1965) set out
in search of single-line (meaning spectral features from only one star)
spectroscopic binaries with invisible companions more massive than the visible
stars. They had access only to a 1948 catalogue of SBs and a smattering of
later orbits, and found seven candidates. Of the seven, four were very long
period systems; none was or is a known X-ray source; and several are probably
not even binary systems, but other periodic or quasi-periodic sorts of
variables. The earlier paper mentions only collapsed stars as candidates; the
latter suggests a neutron star for the seventh system, Alpha Her B.

Five things happened over the next couple of years: (1) the number of known
X-ray sources increased, (2) neutron stars as pulsars became part of the
inventory of known objects, (3) Batten (1968) compiled a much more extensive
and critical catalogue of spectroscopic binaries, (4) Kip Thorne picked up the
idea of potential detectability of collapsed stars in SB1s, and (5) Trimble
completed her PhD dissertation at Caltech (on motions and structures of
filamentary envelope of the Crab Nebula) and had several stray weeks while the
thesis was being typed, duplicated, and read by her committee before it could
be defended (on 15 April 1968) and she could take off for England as a
volunteer postdoc.

KS suggested that VT should go through the Batten catalogue and identify all
the SB1 systems with $M_{2}$ larger than $1.4M_{\odot}$ and larger than the
mass of the visible star, $M_{1}$, as estimated from its spectral type.
Barbara A. Zimmerman of the Caltech computing staff provided essential help in
writing and executing the program that took the catalogued properties (period
and velocity amplitude for the optical star, $P$ and $K_{1}$ plus Kepler’s
third law and turned them into

$(M^{3}_{2}sin^{3}i)/(M_{1}+M_{2})^{2}=const\,X\,PK^{3}_{1}$ (1)

where $M_{2}$ is the invisible star; $M_{1}$ is the mass you have to guess
from its spectral type; and $sin=1$ yields the lower limit on $M_{2}$. It is
left as an exercise for the reader to derive this (which applies only to
circular orbits, the norm for short-period systems). The presentation
erroneously had $K_{2}$ in the derivation and expression (3.1) which is
customarily called $f(M)$ or the mass function.

We recovered six of the seven Zeld́ovich & Guseinov (1966) systems and added
six more with $M_{2}>M_{1}$ and 38 with $M_{2}<M_{1}$, but greater than
$1.4M_{\odot}$ (the limit for stable white dwarfs). More than 40 years
downstream, none of the identified binaries is a known X-ray source (i.e.
collapsed star candidate). This was already clear at the time, because the
search found about as many systems with $M_{2}>M_{1}$ (etc.) that showed
eclipses, meaning that $M_{2}$ could not be a compact star.

The paper was written during a two-day stop-over I made in Chicago, where
Thorne was then on sabbatical. It is the only paper I ever hand-wrote (typing
being so very much easier). And there was a difficulty about who was to be
first author. Not I, said I, because it wasn’t my idea. Not me, said Thorne,
because he hadn’t done the necessary bits of arithmetic. Should we, he then
asked, call it Zeld́ovich and Zimmerman? Well perhaps not, and Trimble &
Thorne (1969) it then became, and remained sporadically cited for a number of
years, because quite a few of the stars turned out to be interesting for other
reasons, though some of them weren’t even really binaries and were removed
from the Ninth Catalogue (Batten 2014).

##  4 What Might Have Been (the first true Z&G system and the last wrong
paper)

Three years passed. The Uhuru satellite was launched and, among the strong
X-ray sources, Cygnus X-1 proved to be particularly, rapidly, and erratically
variable (Schreier et al. 1971, Tananbaum et al. 1972). Simultaneous flaring
in X-ray and radio (Hjellming 1973) permitted optical identification with a
previously-known, late O-type supergiant, HDE 226868. Optical astronomers
rushed in droves to their telescopes (Bolton 1972, Webster & Murdin 1972;
droves were smaller in those days) to check for radial velocity variability.
Sure enough, periodic, at $P=5.6$ days with a velocity half-amplitude of about
$75km/sec$, implying a minimum mass of $3.3M_{\odot}$ for a normal O6
supergiant as the visible star. That limit has crept up over the years
(Paczyński 1974 and beyond).

How could the proper response to this tale be anything but wild cheering? Like
so. Some time between 1972 and 1999, I was on a conference tour bus seated
next to Dan (Daniel Magnes) Popper. Among the anecdotes he related was the
fact that he had observed HDE 226868 as part of a radial velocity survey long
ago, but had had the bad luck to observe at an integral multiple of the orbit
period. In preparing for this talk, I hunted out the relevant paper (Popper
1950). Sure enough, there among 253 O-B6 stars with apparent magnitudes
between 8.5 and 11, were color, spectral type, and radial velocity for HD
226868 (between 227704 and 226951). The stellar velocity is tabulated as
$-13km/sec$, and of quality A, meaning that two different determinations
differed by less than $4km/sec$. The dates of the observing runs appear, and
his two spectra were taken 263 days apart, 47 times the orbit period, rather
exactly. By ill chance, any two observations could catch an unknown binary at
nearly the same phase in its orbit, but I think the constancy to $4km/sec$ out
of a full aptitude of $150km/sec$ should happen something less than 10% of the
time. Doing a better job of this calculation is also left as an exercise for
the reader.

The key point, of course, is that if Popper had prepared a good orbit for the
star, it would have been in Batten’s catalogue, and the 1972 application of
the Zeld́ovich and Guseinov’s (1966) method would have identified it as a
black hole candidate.

Meanwhile, as it were, folks interested in the nature of X-ray binaries were
very much aware that the mass limit for Cyg X-1 was heavily dependent on that
assumed for the visible star on the basis of its spectral type. A very massive
star leaving the main sequence is not the only possibility for the combination
of high temperature and low atmospheric pressure. Post-asymptotic-giant-branch
stars pass through the same conditions (though at much lower luminosity) on
their way to becoming white dwarfs. I think I may actually have been the first
to think of this possibility for Cygnus X-1, for the fairly obvious reason
that I had just calculated models for low mass B stars with low surface
gravity (Trimble 1973) meant to describe HZ 22. When I mentioned the idea to a
stellar evolution colleague at the University of Maryland, Bill Rose, he said
he thought it was a good idea and would like to be an author on the paper.
Seeing my fame as author of “Trimble (1973)” fade into the relative obscurity
of “Trimble & Rose (1973)” or even “Rose and Trimble (1973)” alphabetically, I
co-opted my husband, Joseph Weber, as a third author of Trimble, et al.
(1973). This is the only paper I have ever had appear in print before the
postcard acknowledging receipt of the manuscript arrived.

Once again, optical observers flocked to their telescopes in droves (Margon et
al. 1973, Bregman et al. 1973, droves having become slightly larger in the
intervening year) to prove us wrong. This they quickly did, getting distance
estimates for the system closer to the $2kpc$ implied by a $20M_{\odot}$
slightly evolved O supergiant than the $200pc$ required for a post-AGB star.
Both distances are far too large for the parallax determinations of the
period, and the method used was the determination of the amount of absorption
in comparison with that seen for other hot stars in the same direction of the
sky.

Later work has, of course, refined knowledge of HDE 226868 = Cygnus X-1, but
Trimble et al. (1973) was almost certainly the last fundamentally wrong paper
on the subject. There is a moral here for theorists: a sufficiently definite
prediction of something that can be observed will almost certainly attract the
attention of colleagues wanting to prove you wrong. Unfortunately, most of the
topics discussed at IAUS 308 are too complex for such definitive predictions
to be made at present.

Acknowledgements
I am grateful to Alan Batten and Roger Griffin for e-discussions of the status
of some of the Z&G systems, to the late Dan Popper for confiding his unhappy
tale long ago, to the organizers of IAUS 308 for finding time on the program
for what I had to say, and to Alison Lara for her usual expert keyboarding of
my usual untidy typescript.

## References

  * [Baade & Zwicky (1934)] Baade, W. & Zwicky, F. 1934, Proc. USNAS, 20, 254
  * [Batten (1968)] Batten, A.H. 1968, Publ. Dom. Ap. Obs. 8, 119
  * [Batten (2014)] Batten, A.H. 2014, personal e-mail communication
  * [Bolton (1972)] Bolton, C.T. 1972, Nature 235, 271; Nature Phys. Sci. 240, 124
  * [Bregman (1973)] Bregman, J. et al. 1973, ApJL 186, L117
  * [Dick (2013)] Dick, S.J. 2013, Discovery and Classification in Astronomy: Controversy and Consensus, Cambridge Univ. Press
  * [Ewing (1964)] Ewing, A. 1964, Science News Letters 8, 39 (January 18)
  * [Ginzburg (1990)] Ginzburg, V.L. 1990, ARA&A 20, 1
  * [Guseinov & Zeldovich (1966)] Guseinov, O.H. & Zeld́ovich, Ya.B. 1966, A. Zh 43; 303 Sov. Astr. AJ 10, 251
  * [Hjellming (1973)] Hjellming, R.M. 1973, ApJL 182, L29
  * [Landau (1932)] Landau, L. 1932, Phys. Z. Sowjetunion 1, 285
  * [Landau (1938)] Landau, L. 1938, Nature 141, 333
  * [Margon (1973)] Margon, B. 1973, ApJL 185, L143
  * [Oppenheimer & Snyder (1939)] Oppenheimer, J.R. & Snyder, H. 1939, PR 56 455
  * [Oppenheimer & Volkoff (1939)] Oppenheimer, J.R. & Volkoff, G. 1939, PR 55, 374
  * [Paczynski (1974)] Paczyński, B. 1974, A&A 46, 513
  * [Popper (1950)] Popper, C.M. 1950, ApJ 111, 495
  * [Salpeter et al. (1971)] Salpeter, E.E. et al. 1971, ApJ 140, 796
  * [Schreier (1971)] Schreier, E. et al. 1971, ApJ 170, 121
  * [Schwarzschild (1916)] Schwarzschild, K. 1916, Sitzber. Preus. Deut. Akad. Wiss. Berlin, KL Math.Phys. 189-196 & 424-434
  * [Tananbaum et al. (1972)] Tananbaum, H. et al. 1972, ApJL 177, L5
  * [Trimble (1973)] Trimble, V. 1973, A&A 23, 81
  * [Trimble, Rose & Weber (1973)] Trimble, V., Rose, W.K. & Weber, J. 1973, MNRAS 162, 1p
  * [Trimble & Thorne (1969)] Trimble, V.L. & Thorne, K.S. 1969, ApJ 156, 1013
  * [Webster & Murdin (1972)] Webster, V.L. & Murdin, P. 1972, Nature 235, 37
  * [Zeldovich (1964)] Zeld́ovich, Ya.B. 1964, Sov. Phys. Dokl. 9, 165
  * [Zeldovich & Guseinov(1966)] Zeld́ovich, Ya.B. & Guseinov, O.H. 1966, ApJ 144, 840; Sov. Phys. Dokl. 162, 791
  * [Zeldovich, Ya.B. & Novikov, I.D. (1964b)] Zeld́ovich, Ya.B. & Novikov, I.D. 1964b, Sov. Phys. Dokl. 9, 834
  * [Zeldovich & Novikov (1964)] Zeld́ovich, Ya.B. & Novikov, I.D. 1964, Uspekhi Phys. Nauk. 84, 377 (= 7,763 in translation)
  * [Zeldovich & Novikov (1965)] Zeld́ovich, Ya.B. & Novikov, I.D. 1965, Uspekhi Phys. Nauk. 86, 447 (= 8,522, in translation)
  * [Zeldovich & Novikov (1966)] Zeld́ovich, Ya.B. & Novikov, I.D. 1966, Nuovo Cimento. I 4, 840


# Deep Multi-Scale U-Net Architecture and Label-Noise Robust Training
Strategies for Histopathological Image Segmentation
††thanks: *Equal Contribution.

Nikhil Cherian Kurian1∗, Amit Lohan1∗, Gregory Verghese2,3,4, Nimish
Dharamshi1, Swati Meena1,
Mengyuan Li2,3, Fangfang Liu2,5, Cheryl Gillet6, Swapnil Rane7, Anita
Grigoriadis2,3,4, Amit Sethi1

1Department of Electrical Engineering, Indian Institute of Technology Bombay
Mumbai, India
2Cancer Bioinformatics, School of Cancer Pharmaceutical Sciences, Faculty of
Life Sciences
& Medicine, King’s College London London, UK.
3School of Cancer Pharmaceutical Sciences, King’s College London Faculty of
Life Sciences & Medicine, London, UK.
4Breast Cancer Now Unit, School of Cancer & Pharmaceutical Sciences,
King’s College London, London, UK.
5Department of Breast Pathology & Research Laboratory, Key Laboratory of
Breast Cancer
Prevention & Therapy (Ministry of Education), National Clinical Research
Center for Cancer,
Tianjin Medical University Cancer Institute & Hospital, Tianjin, China.
6CRUK King’s Health Partners Centre, King’s College London, Innovation Hub,
Cancer Centre at Guy’s Hospital, Great Maze Pond, London, UK.
7Dept. of Pathology, Tata Memorial Centre-Tata Memorial Hospital, HBNI,
Mumbai, India

###### Abstract

Although the U-Net architecture has been extensively used for segmentation of
medical images, we address two of its shortcomings in this work. Firstly, the
accuracy of vanilla U-Net degrades when the target regions for segmentation
exhibit significant variations in shape and size. Even though the U-Net
already possesses some capability to analyze features at various scales, we
propose to explicitly add multi-scale feature maps in each convolutional
module of the U-Net encoder to improve segmentation of histology images.
Secondly, the accuracy of a U-Net model also suffers when the annotations for
supervised learning are noisy or incomplete. This can happen due to the
inherent difficulty for a human expert to identify and delineate all instances
of specific pathology very precisely and accurately. We address this challenge
by introducing auxiliary confidence maps that emphasize less on the boundaries
of the given target regions. Further, we utilize the bootstrapping properties
of the deep network to address the missing annotation problem intelligently.
In our experiments on a private dataset of breast cancer lymph nodes, where
the primary task was to segment germinal centres and sinus histiocytosis, we
observed substantial improvement over baselines based on the two proposed
schemes.

###### Index Terms:

Segmentation, Multi-Scale, Unet, Noisy-labelled, Histopathology

##  I Introduction

In pathology, hematoxylin and eosin (H&E) stained histopathological whole-
slide images (WSIs) are sources of rich diagnostic information[1, 2]. Various
segmented regions of a WSI can give biomarker information, such as areas in
various histologically distinct regions and their ratios. Deep learning has
become a framework of choice for automating segmentation with high accuracy,
provided a large dataset of annotated images is available for training[3].
Within deep learning, architectures inspired from the U-Net have been widely
adopted due to their unique ability to both gather evidence for semantic
segmentation from somewhat large receptive fields, and yet be able to produce
fine-grained and precise boundaries [4]. However, U-Net has been shown to
perform poorly when the target regions exhibits large variations in their
shape and scale that is common in histology [5, 6]. This challenge is common
in our target problem, where we want to segment germinal centers (GC) or
sinus-histiocytosis in breast cancer lymph node slides. In this work we
present Multi-Scale U-Net (MS U-Net), a lightweight architecture that performs
effective segmentation of histology images by introducing explicit multi-scale
feature extraction in the U-Net encoder. We compare the performance of our
architecture to other U-Net based baselines. To account for edge artifacts and
padding issues arising when the constituent patches of WSI are segmented
independently, we also propose a better loss calculation strategy for each
image patch. Here we calculate the loss only on the centre cropped regions
from each patch, and subsequently the segmentation mask from these regions are
only utilised to stitch back and create a smooth whole slide level
segmentation mask.

Furthermore, the advantage of high accuracy offered by most of the deep
learning networks is realized only when a large and accurately labeled dataset
is available for supervised learning [7]. For tissue region segmentation on
WSIs, the large dimensions of these images and inherent continuums in lesion
grades make it prohibitively difficult for a human annotator to mark exact
boundaries of various tissue regions in addition to avoiding errors of totally
missing some small regions from annotations. Thus, the quality of annotations
provided by a human expert in these settings are often inaccurate and noisy.
Such noisy supervision degrades the performance of the deep learning model,
especially in case of segmentation task where the ground truth involves pixel
level annotations [7]. In an endeavor to address these challenges, we explore
and adapt certain robust deep learning training strategies to further improve
our segmentation results on top of our proposed MS U-Net architecture.
Specifically, we relax the strict imposition of cross-entropy (CE) loss at the
boundary of annotations where the annotator are most uncertain, through
blurred confidence maps that modulate the original CE loss. Also, we attempt
to address the challenging problem of missing annotations through a
bootstrapping framework, where we gather the most confident model predictions
from an early training iteration as an auxiliary masks that can enhance model
predictions in case of missing annotations.

##  II Related Work

###  II-A Multi-Resolution Segmentation

The most widely used architecture for medical segmentation is the U-Net model,
which consists of an encoder and a decoder module each [4]. Although the
pooling and up-convolution operations in a U-Net give it some ability to
process information at multiple scales, in several followup works the U-Net
model has been improved by explicitly incorporating multi-resolution
processing capabilities. These proposed multi-resolution image analysis
architectures can effectively fuse the contextual information of a tissue with
its nuclear level morphologies. Another popular modification of vanilla U-Net
called attention U-net [8] has been extensively reported to improve results on
multi resolution problems compared to the baseline models. In attention
U-nets, attention gates are used in the network and it will handle areas of
high relevance that are multiplied with a larger weight and areas of low
relevance are tagged with smaller weights.

In [9], a multi-scale information extractor was utilised to process patches at
multiple resolution at each layer of the encoder stage after passing through a
convolutional layer. An alternate method was suggested in [10], where they
used multiple separate encoder and decoders for patches taken from different
resolution. They further concatenated these separate decoders at multiple
decoder layers. In [11], the authors designed a multi-scale unet model
tailored for radiology and clinical image segmentation. Here the authors
propose a sequence of convolution layers to effectively extend the receptive
field of the segmentation model. The concept of multi-path training was
explored in [12], where they used three separate paths for encoding features
at three separate resolutions, where each path consists of dense blocks. Also
each path in the model is trained individually and separately, along with an
overall end to end training.

Though these methods effectively fuse multi-resolution information for
segmentation, these architectures are computationally expensive and difficult
to train.

###  II-B Learning from Noisy Annotations

Noisy annotations are almost unavoidable in certain types of medical images,
especially when these have continuum of disease grades, numerous objects of
varied sizes to be segmented, or low contrast or intricate boundaries between
anatomical structures. This problem is more challenging when multiple
annotators work on the same image. Deep learning-based medical image
segmentation in the face of noisy or incomplete ground truth masks is a
challenging problem gaining some attention [13, 14]. In mandible CT images Yu
et.al [15] studied the effect of noisy annotations in a deep learning-based
segmentation. The study empirically highlighted the degradation of deep
learning model performance when trained with noisy ground truth mask.
Similarly, in [14], the authors explicitly show the negative impact of missing
annotations in deep segmentation models for medical imaging. However, more
work is needed to effectively address this problem.

##  III Methodology

###  III-A Multi-Scale U-Net Architecture

The proposed architecture modifies the original encoder block of the U-Net
architecture, where we introduce a downsampling block with multiple kernels
for producing feature maps at different scales. The architecture of the
downsampling blocks in the proposed Multi-scale U-Net (MS U-Net) model is
shown in Figure 1. Though the high-level architecture is similar to that of a
U-Net, but each downsampling block on the encoder side contains the
modification capable of multi-scale processing. In a typical U-Net block,
there are two convolutional layers and the output feature map is passed
through a max-pooling layer. In the proposed multi-scale architecture, each
block contains three sets of kernels in the first convolutional layer, where
each set works at a different scales. That is, one set of kernels works at the
resolution of input to the block, another set has a dilation and stride of two
to reduce the resolution to a half, and a third set has a dilation and stride
of four to reduce the resolution to a fourth of the input.

The outputs of the two sets of kernels that downscale the feature maps scaled
back up to the original resolution using bilinear interpolation and
concatenated with the feature maps of the first set of kernels to make a
single set of feature maps. This set of feature maps is then passed to a max-
pool layer before being sent to the next block where the same sequence of
operations is repeated. In the decoder, blocks that have an up-convolution are
then passed these extracted encoder features from corresponding level
concatenated along with previous layer decoder feature maps. After the
interpolation and concatenation of the two feature maps, they are passed
through single convolutional layer before passing outputs to next decoder
block.

Figure 1: Proposed Multi-Scale U-Net (MS U-Net) model architecture with the
encoder block expanded to show the multi-scale processing of segmentation
features. Figure 2: (a)The original WSI input to the U-Net architecture, (b)
The WSI mask with missing annotations of certain sinus regions as shown in the
green box (c) The mask predicted by our model in its bootstrapping period
indicating the presence of missing region

###  III-B Augmentations

Augmenting the training data by randomly rotating the images and their
segmentation maps have been found to be useful for improving the segmentation
results. For whole slide images, as the training is done at the patch level
and final prediction is required to be done at the whole slide image level, it
is important to feed the patches without any artificial background to the
network. This was done by creating $\sqrt{2}\times\sqrt{2}$ times larger
patches (1444x1444) than the actual size (1024x1024) used for training. With
respect to the center of the larger patch, after randomly deciding a rotation
angle, four coordinates of the rotated patch are calculated and the rotated
patch is then extracted as square patch using affine transformation. This way
the actual patch is created by extracting a randomly rotated patch from the
center of larger patches. Along with this, horizontal and vertical flipping
augmentations were carried out, as well as colour jitter with brightness,
saturation, hue and contrast as 0.15.

###  III-C Robust Training Strategies for Noisy Annotations

The annotation procedure is quite laborious and the provided annotations are
almost always inevitably noisy, especially with a large amount of incorrectly
marked pixels for regions such as sinus and follicle, where even pathologists
may disagree on the correct label for a region of pixels. Thus, we inevitable
give the neural network some wrong labels for some of the regions to learn
from, and that makes it to produce poor results at the test time. Using sinus
as an example, the two major problems of inaccurate boundaries and missing
annotations (where some sinus region might be completely ignored by
annotator), are shown in figure 2(b).

Fuzzy Boundaries helps deal with inaccurate boundaries of annotations. In this
technique, while computing the loss for the prediction of a patch, an
additional mask is computed by blurring the edges in the original segmentation
mask. This blurred mask is then subtracted from the original mask and the
absolute value at each location is taken. This mask is then normalized and
subtracted from a mask of all 1’s. This way we finally obtain a mask
indicating the confidences with value 1’s at the regions away from any kind of
boundaries and smaller values closer to 0, near to and on the boundaries,
depending on the gradient of each location.This mask is then multiplied with
the cross entropy loss values calculated for respective locations. Finally the
overall loss is averaged. This way, the values closer to boundaries contribute
less to the loss calculation and the values away from boundaries contribute
more. That is, we let the model decide for the locations closer to boundaries
and don’t force it to learn the labels marked in annotations. Weights for
fuzzy boundary, $w_{fb}$, are given in equation 1 as,

$w_{fb}=1-abs(t-g(t,k))$ (1)

where, g() is the Gaussian blurring function, abs() is the absolute function,
t is normalized target mask, k is kernel size for blurring. For the blurring,
if the patch is from a normal dataset, a Gaussian blurring is applied using
kernel size of (21,21), and if the patch is from noisy dataset, then a
Gaussian blurring of kernel size (61,61) is applied. That is, on a noisy
dataset we have less trust in the boundary locations given by the annotator,
hence we want it to be more fuzzy, or having lower importance in loss
computation.

Bootstrapping deals with missing annotations by pathologists. In
bootstrapping, we train the network for a few epochs, where it learns to
segment regions which can clearly be said to belong to a class or not with
confidence. Then, in addition to the ground truth mask used for segmentation,
the prediction of network itself, which acts as an auxiliary mask, is used to
compute the loss by deciding a weight map for it. The network’s prediction
masks are used only after training it for bootstarp period of $n$ number of
epochs, which is a hyperparameter.

For implementing the bootstrapping method, we compute bootstrap weight matrix
$w_{bs}$. In our experiments , on the $9$th iterations, the loss function
,$l_{bs}$, formed out of predicted outputs as mask, is spatially weighted by
$w_{bs}$, given in equation 2 as,

$w_{bs}=2*abs(0.5-p)$ (2)

where, p is the model prediction output after passing through a softmax layer.
So, $w_{bs}$ basically pushes those model predictions with higher confidence
(predictions close to 0 or 1) to higher magnitude and other predictions with
lower confidence (predictions close to 0.5) to lesser magnitude, and computes
$w_{bs}$.

###  III-D Loss Functions

We used weighted cross entropy loss to counter the imbalance between the
numbers of pixels belonging to foreground or background (approximately 1:10),
by using an inverse of their ratio. For a noisy dataset, this loss is then
multiplied by a confidence mask created using the fuzzy boundary and
bootstrapping loss techniques. We also countered the impact of missing context
near the edges of the patches by using center-cropped loss calculation. That
is, the loss for a pre-defined margin of a patch is discarded for training.

The weighted cross entropy loss $l_{wce}$ is given by the equation 3 as,

$l_{wce}=\sum(w_{bg}*p_{bg}*t_{bg}+w_{fg}*p_{fg}*t_{fg}),$ (3)

where $w_{bg}$ is the background pixels weight, and $w_{fg}$ is the foreground
pixels weight

The final loss $l$ is given by the equation 4 as follows:

$l_{total}=l_{wce}*w_{fb}+\mathbbm{1}_{epoch>n}*(l_{bs}*w_{bs})$ (4)

where $\mathbbm{1}$ is the indicator function. The indicator function
evaluates to one only after its condition hold true, or otherwise will be
zero. Hence, here only after $epoch>n$ becomes true the loss encountered due
to bootstrap loss gets activated.

Figure 3: (A) Input WSI of Lymph Node, (B) Target mask to segment Germinal
Centre (GC) regions, (C) U-Net model prediction overlay with mask and
image,(D) Attention U-Net model prediction overlay with mask and image, (E) MS
U-Net model prediction overlay with mask and image.

###  III-E Evaluation Metrics

For segmentation, a large WSI is often divided into constituent patches that
are individually segmented by a deep learning model. The segmentation mask
from these patches are then stitched back to finally create a WSI level
segmentation mask. For our analysis, we used intersection over union (IoU)
score as the evaluation metric at the WSI level. The value of IoU score ranges
in between $0-1$, and a larger value is desirable.

Along with the IoU score, for qualitative evaluation we also overlayed images
prediction with input WSI image and its ground truth. For overlaying ground
truth with prediction, we represent the true positive region, false positive
region and false negative region with green, red and blue regions,
respectively. This is overlaid over the input WSI image, as shown in the
results of figures 3(C)-(E) and 4 (C)-(H).

###  III-F Data Preparation

The dataset comprises of whole slide images (WSIs) of lymph node regions,
obtained from a private repository of breast cancer WSIs of lymph node data
collected from Guy’s hospital, London. Annotations were performed
independently by two pathologists. The pixel-wise annotations were carefully
created to identify and mark all regions of sinus for 50 WSIs. There are two
experiments mentioned below, one focusing on comparison of training results of
the proposed MS U-Net model with UNet and Attention Unet baselines, and
another focusing on comparison of training results obtained from different
robust methods like fuzzy boundary and bootstrapping.

For the first experiment, 20 WSIs were used for training, 8 WSIs were used for
validation, and the 5 WSIs were used for testing. All annotations for the
first experiment were prepared by extensive and carefully annotations by the
two expert pathologist. A total of 11,000 image patches were extracted in the
preprocessing stages of training the segmentation model. All experiments are
conducted at 10x resolution. A patch-wise training approach was followed, for
which there was a 50% overlapping region between adjacent extracted patches
from the WSIs. Patches of 1024x1024 size were extracted. In the second
experiment to improve the sinus region segmentation, in addition to the clean
20 WSIs used before for training, we introduced 22 more WSIs that had more
noisy or inaccurate annotations.

Figure 4: (A) Input WSI of Lymph Node, (B) Target mask to segment Sinus regions, (C) U-Net model prediction overlays,(D) Attention U-Net model prediction overlays, (E) MS U-Net model prediction overlays,(F) MS U-Net with fuzzy boundary (FB), (G) MS U-Net with bootstrapping (BS), (H) MS U-Net with FB and BS. TABLE I: Experiment 1 results: comparison of IoU score of Sinus and Germinal centre IoU scores between different architectures. WSI Identifier | UNet | Attention UNet | MS UNet | UNet | Attention UNet | MS UNet
---|---|---|---|---|---|---
Germinal Center | Sinus
Test WSI 1 | 0.6123 | 0.7192 | 0.8027 | 0.2222 | 0.4823 | 0.7218
Test WSI 2 | 0.7618 | 0.7912 | 0.8210 | 0.2836 | 0.4418 | 0.6710
Test WSI 3 | 0.4212 | 0.6614 | 0.7916 | 0.1983 | 0.3908 | 0.4856
Test WSI 4 | 0.8299 | 0.5982 | 0.9011 | 0.3422 | 0.3742 | 0.5122
Test WSI 5 | 0.4333 | 0.4602 | 0.6046 | 0.4500 | 0.5518 | 0.6330
Average | 0.6117 | 0.64604 | 0.7842 | 0.2993 | 0.4482 | 0.6047
TABLE II: Experiment 2 results: comparing the quantitative improvements of IoU score with Fuzzy Boundaries and Bootstrapping WSI Identifier | MS UNet | MS UNet Fuzzy Boundary | MS UNet Bootstrapping | MS UNet Fuzzy Boundary + Bootstrapping
---|---|---|---|---
Test WSI 1 | 0.7218 | 0.7456 | 0.7362 | 0.7480
Test WSI 2 | 0.6710 | 0.7243 | 0.6988 | 0.7312
Test WSI 3 | 0.4856 | 0.5828 | 0.5512 | 0.6104
Test WSI 4 | 0.5122 | 0.5800 | 0.5666 | 0.6241
Test WSI 5 | 0.6330 | 0.6012 | 0.6402 | 0.6433
Average | 0.5436 | 0.5880 | 0.5860 | 0.6259

##  IV Results

###  IV-A U-Net, Attention U-Net and MS U-Net Compared

The results for U-Net , Attention U-Net and MS U-Net model are shown below in
table I for different test images. The training was performed using Adam
optimizer, using constant learning rate of $10^{-3}$.

As seen from the IoU scores, the MS U-Net model outperforms the plain U-Net
model based on the same training method. This shows the ability of MS U-Net
model to better capture the multi-resolution information needed for the
segmentation of sinus regions. Since the germinal centers (GC) are easy to
spot compared to the variations observed in a sinus region, the scores for GC
region are higher compared to that of sinus regions.

To also do a quality analysis of model predictions, in figure 3 we have shown
one of the testing WSIs (Test WSI 1 in table I), its target mask region
indicating the sinus regions, and three images in (C),(D) and (E), shows the
overlap of the different model prediction with the ground truth mask,
overlayed on the input image. As seen U-Net and attention U-Net model results
contain much more false positives than MS U-Net model, which is even shown by
the difference in their mIoU scores.

###  IV-B MS U-Net with Noise Robust Training

The MS U-Net sinus prediction results obtained are compared using three
different schemes, as shown in table II – one by introducing the fuzzy
boundary loss to the MS U-Net model, another by introducing the bootstrapping
method, and lastly by combining the fuzzy boundary loss and bootstrapping
method. As it can be seen that the performance obtained using both the fuzzy
boundaries and bootstrapping method gives the best results. We also show the
qualitative results in terms of the segmentation masks given by the different
models in figure 4 for Test WSI 1 in table II

##  V Conclusion

Our experiments establish the advantage of modifying the U-Net models to
include stronger capabilities to handle multiple resolutions. We proposed a
simple and effective method modification that employs kernels with different
strides and fuses their feature maps to segment histopathology slides with
features at various scales. We further analyzed and offered a practical
solution that is robust to noisy annotations – those with inexact boundaries
and missing annotations. We relaxed the emphasis of cross-entropy loss at the
boundaries through Gaussian blurred confidence maps and an auxiliary mask
created using bootstrapping a model prediction’s predictions after training it
for a few epochs. We compared and contrasted the improvements of each of these
methods when the ground truth data is inherently noisy to show that their
combination gives the best results, indicating each of these have a role to
play in medical image segmentation.

## References

  * [1] John KC Chan,  “The wonderful colors of the hematoxylin–eosin stain in diagnostic surgical pathology,”  International journal of surgical pathology, vol. 22, no. 1, pp. 12–32, 2014.
  * [2] Nikhil Cherian Kurian, Amit Sethi, Anil Reddy Konduru, Abhishek Mahajan, and Swapnil Ulhas Rane,  “A 2021 update on cancer image analytics with deep learning,”  Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 11, no. 4, pp. e1410, 2021.
  * [3] Chandradeep Bhatt, Indrajeet Kumar, V Vijayakumar, Kamred Udham Singh, and Abhishek Kumar,  “The state of the art of deep learning models in medical science and their challenges,”  Multimedia Systems, vol. 27, no. 4, pp. 599–613, 2021.
  * [4] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in International Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241.
  * [5] Nabil Ibtehaz and M Sohel Rahman,  “Multiresunet: Rethinking the u-net architecture for multimodal biomedical image segmentation,”  Neural Networks, vol. 121, pp. 74–87, 2020.
  * [6] Jingdong Yang, Jintu Zhu, Hailing Wang, and Xin Yang,  “Dilated multiresunet: Dilated multiresidual blocks network based on u-net for biomedical image segmentation,”  Biomedical Signal Processing and Control, vol. 68, pp. 102643, 2021\.
  * [7] Zhiwu Lu, Zhenyong Fu, Tao Xiang, Peng Han, Liwei Wang, and Xin Gao,  “Learning from weak and noisy labels for semantic segmentation,”  IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 3, pp. 486–500, 2016.
  * [8] Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al.,  “Attention u-net: Learning where to look for the pancreas,”  arXiv preprint arXiv:1804.03999, 2018.
  * [9] David Joon Ho, Dig VK Yarlagadda, Timothy M D’Alfonso, Matthew G Hanna, Anne Grabenstetter, Peter Ntiamoah, Edi Brogi, Lee K Tan, and Thomas J Fuchs,  “Deep multi-magnification networks for multi-class breast cancer image segmentation,”  Computerized Medical Imaging and Graphics, vol. 88, pp. 101866, 2021\.
  * [10] Simindokht Jahangard, Mohammad Hossein Zangooei, and Maysam Shahedi,  “U-net based architecture for an improved multiresolution segmentation in medical images,”  arXiv preprint arXiv:2007.08238, 2020.
  * [11] Run Su, Deyun Zhang, Jinhuai Liu, and Chuandong Cheng,  “Msu-net: multi-scale u-net for 2d medical image segmentation,”  Frontiers in Genetics, vol. 12, pp. 639930, 2021.
  * [12] Yunhe Gao, Chang Liu, and Liang Zhao,  “Multi-resolution path cnn with deep supervision for intervertebral disc localization and segmentation,”  in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2019, pp. 309–317.
  * [13] Nima Tajbakhsh, Laura Jeyaseelan, Qian Li, Jeffrey N Chiang, Zhihao Wu, and Xiaowei Ding,  “Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation,”  Medical Image Analysis, vol. 63, pp. 101693, 2020.
  * [14] Davood Karimi, Haoran Dou, Simon K Warfield, and Ali Gholipour,  “Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis,”  Medical Image Analysis, vol. 65, pp. 101759, 2020.
  * [15] Shaode Yu, Mingli Chen, Erlei Zhang, Junjie Wu, Hang Yu, Zi Yang, Lin Ma, Xuejun Gu, and Weiguo Lu,  “Robustness study of noisy annotation in deep learning based medical image segmentation,”  Physics in Medicine & Biology, vol. 65, no. 17, pp. 175007, 2020\.


11institutetext: Translational Surgical Oncology, National Center for Tumor
Diseases, Dresden, Germany (11email<EMAIL_ADDRESS>22institutetext: Department for Visceral, Thoracic and Vascular Surgery,
University Hospital Carl-Gustav-Carus, TU Dresden, Germany 33institutetext:
Institute and Policlinic for Diagnostic and Interventional Radiology,
University Hospital Carl-Gustav-Carus, TU Dresden, Germany

# Non-Rigid Volume to Surface Registration using a Data-Driven Biomechanical
Model

Micha Pfeiffer 11 Carina Riediger 22 Stefan Leger 11 Jens-Peter Kühn 33 Danilo
Seppelt 33 Ralf-Thorsten Hoffmann 33 Jürgen Weitz 22 Stefanie Speidel 11

###### Abstract

Non-rigid registration is a key component in soft-tissue navigation. We focus
on laparoscopic liver surgery, where we register the organ model obtained from
a preoperative CT scan to the intraoperative partial organ surface,
reconstructed from the laparoscopic video. This is a challenging task due to
sparse and noisy intraoperative data, real-time requirements and many unknowns
- such as tissue properties and boundary conditions. Furthermore, establishing
correspondences between pre- and intraoperative data can be extremely
difficult since the liver usually lacks distinct surface features and the used
imaging modalities suffer from very different types of noise. In this work, we
train a convolutional neural network to perform both the search for surface
correspondences as well as the non-rigid registration in one step. The network
is trained on physically accurate biomechanical simulations of randomly
generated, deforming organ-like structures. This enables the network to
immediately generalize to a new patient organ without the need to re-train. We
add various amounts of noise to the intraoperative surfaces during training,
making the network robust to noisy intraoperative data. During inference, the
network outputs the displacement field which matches the preoperative volume
to the partial intraoperative surface. In multiple experiments, we show that
the network translates well to real data while maintaining a high inference
speed. Our code is made available online.

###### Keywords:

Liver Registration Soft-Tissue Surgical Navigation CNN

##  1 Introduction

a)b) Figure 1: Non-rigid registration using our CNN. Given the intraoperative
point cloud of a partial liver surface and the preoperative liver mesh (a),
our network estimates a displacement field, which deforms the preoperative
volume to match the intraoperative surface (b). Displacement magnitude is
encoded in shades of blue.

In navigated surgical interventions, the aim is to aid surgeons in finding
structures of interest - such as tumors, vessels or planned resection lines.
Often, a detailed, accurate and highly informative preoperative computer
tomography (CT) scan is available and the challenge is to align this model
with the intraoperative scene. Whenever deforming soft-tissue is involved,
this requires a non-rigid registration. However, usually, only limited, sparse
and noisy data can be acquired during the intervention. Furthermore, it is
usually very difficult to find one-to-one correspondences between the pre- and
intraoperative data, since they are subject to different types of noise and
can look substantially different. Together with the large deformation, only
partially visible surface and many unknown parameters - such as the organ’s
elastic properties, acting forces and boundary conditions - this makes the
soft-tissue registration a challenging problem.

This work explores a deep-learning based approach to performing a non-rigid
organ registration. We focus on laparoscopic liver surgery and register a
given preoperative liver volume mesh to an intraoperative, partial surface of
the same organ (obtained from a laparoscopic stereo video stream). We use a
fully convolutional neural network (CNN), which analyses the two meshes and
outputs a displacement field to register the preoperative organ volume to the
intraoperative surface (see Fig. 1). To reach this aim, the network must learn
to find a plausible solution for the surface correspondence problem, while at
the same time learning about biomechanical deformation in order to constrain
itself to physically realistic solutions. For training, a deformed
intraoperative state of each preoperative mesh is simulated, using the Finite
Element Method (FEM). We use synthetically generated, random organ-like
meshes, which allows the trained network to generalize to new patients without
the need to re-train for every liver. We call the approach _Volume-To-Surface
Registration Network_ (V2S-Net) and publish the code as well as the pretrained
network online111https://gitlab.com/nct_tso_public/Volume2SurfaceCNN.

####  1.0.1 Related Work

The application of deep learning to simulate organ deformation has recently
received a lot of attention, mainly due to the very low inference times [9,
10, 12, 14]. It has been shown that these data-driven models can achieve a
high accuracy [9, 12], can deal with partial surface information [2, 14] and
can even learn to deform real organs after training on synthetic data only
[14]. However, all of the mentioned methods require the use of boundary
conditions similar to the FEM. These are very difficult to obtain in a real
surgical setting, since, for example, forces are extremely difficult to
measure and estimating the surface displacement would require known surface
correspondences.

On the other hand, the machine learning community has developed a large number
of data-driven feature descriptors to efficiently describe and register 3D
structures [6, 17, 3]. While this shows the ability of neural networks to
interpret 3D data, in the case of organs, the lack of distinct features and
large organ deformation requires the incorporation of soft-tissue mechanics in
order to find correct correspondences.

In design, our method is similar to the work of Suwelack et al. [16], who
propose to morph the preoperative volume mesh into the intraoperative partial
surface by simulating electric charge on the surfaces and solving the problem
using the FEM. Similar to our work, they use the two meshes as input and
output a displacement field for the preoperative mesh. However, like many
previous approaches, their method requires manual assignment of boundary
conditions and parameters, can become unstable if wrong values are chosen and
their displacement estimation is much slower than ours.

##  2 Methods

The aim of the proposed V2S-Net is to learn to recover a displacement field
which deforms the input volume in such a way that it is well aligned with the
partial surface, when given only the preoperative volume mesh and a partial
intraoperative surface as input. We train the network on synthetic data, since
a real data set consisting of volume meshes as well as known displacement
fields is not available. In addition to simulating deformations, we also
generate the preoperative 3D meshes randomly, to ensure that the network will
translate directly to new, unseen liver meshes.

a)b)c)d)e)V2S-Net Figure 2: a) Random preoperative volume mesh $\text{V}_{P}$,
b) simulated intraoperative state $\text{V}_{I}$ (green) and partial surface
$S_{I}$ (orange), c) signed distance field of preoperative volume
$\text{SDF}_{P}$, d) distance field of partial surface $\text{DF}_{I}$, e)
known displacement U (magnitude), which the V2S-Net learns to predict from the
two distance fields. Afterwards, this displacement field can be used to infer
the position of internal structures, such as vessels and tumors.

###  2.1 Data Generation

####  2.1.1 Simulation

A random 3D surface mesh is generated by first creating an icosphere and then
applying a set of extrusion, boolean, remeshing and smoothing operators. We
use Gmsh [5] to fill the surface mesh with tetrahedral elements. The resulting
volume mesh is considered to be our preoperative volume organ mesh
$\text{V}_{P}$ (Fig. 2, a). Next, up to three random forces (max. magnitude
1.5 N) and a zero-displacement boundary condition are assigned to random areas
of the surface of the mesh. The steady-state deformation is calculated by the
Elmer [8] finite element solver, using a neo-Hookean hyperelastic material
model with a random Young’s Modulus (2 kPa to 5 kPa) and Poisson’s ratio of
0.35. The resulting deformed volume acts as the intraoperative state of the
organ $\text{V}_{I}$ (Fig. 2 b). For every vertex in $\text{V}_{P}$, we now
know the displacement vector that needs to be applied in order to reach the
intraoperative state $\text{V}_{I}$, resulting in the displacement field $u$.

We extract a random surface patch of this deformed volume mesh. To simulate
intraoperative noise and difference in imaging modalities, we resample this
patch, delete random vertices and add uniform random displacements to the
position of every vertex. Furthermore, random parts of the patch are deleted
to simulate areas where surrounding tissues (like the falciform ligament)
occlude the surface from the perspective of the laparoscope. The result is our
intraoperative partial surface mesh $\text{S}_{I}$.

The use of random values sometimes leads to the generation of meshes for which
the creation of tetrahedral elements fails or simulations for which the finite
element solver does not find a solution. These samples are discarded. We also
discard samples where the maximum displacement is larger than 20 cm or the
amount of visible surface is less than 10%, since we assume these cases to be
unrealistic.

####  2.1.2 Voxelization

To pass the meshes to the network, we represent them in the form of distance
fields on a regular grid. For this, we define a grid of $64^{3}$ points. At
each point, we calculate the distance to the nearest surface point of the
preoperative volume $\text{V}_{P}$ as well as the intraoperative surface
$\text{S}_{I}$. For the preoperative mesh, we flip the sign of all grid points
that lie within the organ volume, resulting in a signed distance field
$\text{SDF}_{P}$ for the preoperative and a distance field $\text{DF}_{I}$ for
the intraoperative mesh (see Fig. 2 c and d).

Additionally, we interpolate the target displacement field $u$ into the same
grid with a gaussian kernel, resulting in an interpolated vector field U. For
points outside the preoperative volume, U is set to $(0,0,0)^{T}$.

We use the outlined process to generate $80~{}000$ samples, which are split
into training data (90%) and validation data (10%). By flipping each training
sample along each combination of X-, Y- and Z-axes, we further increase the
amount of training data by a factor of eight, resulting in roughly $460~{}000$
samples.

###  2.2 Network Architecture and Training

To estimate the displacement field U, the network is given the full
preoperative volume $\text{SDF}_{P}$ and the partial intraoperative surface
$\text{DF}_{I}$. Formally, we want our network to estimate the function:

$F(\text{SDF}_{P},\text{DF}_{I})=\text{U}.$ (1)

Similar to [9] and [14], we choose a fully convolutional architecture with 3D
convolutions, an encoder-decoder structure and skip connections to allow for
an abstract low-resolution representation while preserving high-resolution
details. The precise architecture is shown in the supplementary material.
Following [14], we let the network output the displacement field
$\text{U}_{r,est}$ at multiple resolutions $r\in(64,32,16,8)$. These are
compared to the (downsampled) target displacement fields $\text{U}_{r}$ using
the mean absolute error $L_{r}$. We find that this process speeds up loss
convergence during training. The final loss $L$ is a weighted sum of these
errors:

$\begin{split}L_{r}&=\frac{1}{r^{3}}\sum_{i=1}^{r^{3}}{\left|\text{U}_{r}(i)-\text{U}_{r,est}(i)\right|}\\\
L&=10~{}L_{64}+L_{32}+L_{16}+L_{8}.\end{split}$ (2)

We train the network with a one cycle learning rate scheduler [15] and the
AdamW optimizer [7] for 100 epochs, after which the mean registration error on
the validation data has converged to roughly 6 mm.

##  3 Experiments and Results

Since our network is trained on randomly generated meshes, it is vital to test
on real patient data. However, reference data for real laparoscopic
interventions is very difficult to obtain, since interventional CT scans
during laparoscopy are very limited. To nevertheless capture all necessary
aspects of the registration process, we conduct three experiments: one
experiment with simulated liver deformations, one with real human liver
deformations under breathing motion and one qualitative experiment on data
from a laparoscopic setting. In all experiments, the used preoperative liver
meshes were extracted from patient CT scans automatically using a segmentation
CNN [1]. The process of estimating the displacement field (including upload to
the GPU) takes roughly 130 ms.

###  3.1 Liver Deformations (In Silico)

We generate a synthetic dataset as described in section 2.1. However, instead
of generating random meshes we use the mesh of a patient’s liver and simulate
1725 deformations and partial intraoperative surfaces. This new dataset is not
used for training and thus allows us to test how our method translates to a
real liver mesh: We apply the network to all samples and calculate the
displacement error for every point in the estimated displacement fields
$\text{U}_{64,est}$ (results in Fig. 3). When the visible surfaces are large
enough, registration errors tend to be small, even when the target
deformations become large. Smaller visible areas lead to more interesting
results, for example when the network cannot infer which part of the volume
should be matched to the partial surface (see Fig. 4).

Figure 3: Mean displacement error of each sample over the mean target
displacement. As expected, the registration error tends to increase with
target displacement and decrease with larger visible surfaces. With few
exceptions, samples with a displacement error larger than 1 cm are those with
very little visible surface. 0 cm6.4 cma)b) Figure 4:  From a real
preoperative liver mesh (grey), a deformed intraoperative liver state (purple)
is simulated. Given the preoperative mesh and a partial intraoperative surface
(yellow), the V2S-Net estimates the deformed state (images 2 and 4). The
magnitude of the error (red) indicates a successful registration for case a)
(max. error 1.5 cm) and an unsuccessful registration for case b) (max. error
6.4 cm). When the randomly picked visible surface contains no distinct
features (as is the case in b), or captures no areas of significant
deformation, the network may estimate an incorrect but plausible deformation,
resulting in the outliers in Fig. 3. This suggests that it is more important
which areas of the liver are visible than the size of the visible surface.

###  3.2 Liver with Breathing Deformation (In Vivo)

During breathing motion, the human liver moves and deforms considerably. To
assess whether our network translates to real human liver deformation, we
evaluate on human breathing data. We extract liver meshes from two CT scans
(one showing an inhaled and one an exhaled state) and let these represent the
preoperative volume $V_{P}$ and intraoperative surface $S_{I}$. We search for
clearly visible vessel bifurcations and mark their positions in both scans.
Given the distance fields of the two meshes, the network then estimates the
displacement of every voxel in the inhaled state. The resulting displacement
field $\text{U}_{64,est}$ is interpolated to the positions of the marked
bifurcation points (gaussian interpolation kernel, radius 1 cm) and is used to
displace them. We carry out this experiment for two patients. Average
displacements and remaining errors after registration are shown in Table 1 and
the result for Patient 1 is depicted in Fig. 5.

Table 1: Displacement of marked bifurcations (due to inhaling) and remaining errors after registration. | Avg. displacement (mm) | Avg. error (mm)
---|---|---
Patient 1 | 21.4 (max: 31.1) | 5.7 (max: 9.8)
Patient 2 | 28.3 (max: 32.7) | 4.8 (max: 5.4)
Figure 5: Registration for Patient 1. The liver is deformed by breathing
motion (inhaled and exhaled states, left). Four vessel bifurcations are marked
in the first CT scan (black) and in the second CT scan (red). The network uses
the surfaces to calculate a displacement field which registers the two liver
states to each other (right). Applying this displacement field to the markers
indicates a good approximation of the internal deformation. Results for
Patient 2 can be found in the supplementary material.

###  3.3 Laparoscopic Liver Registration (In Vivo)

To validate whether our method works for our target task of navigated
laparoscopy, we perform a qualitative evaluation on human in-vivo data
acquired from a da Vinci (Intuitive Surgical) stereo laparoscope. We first
reconstruct the intraoperative surface $S_{I}$ from the video stream. For
this, a 10 second camera-sweep of the liver is performed at the beginning of
the surgery. Since we lack positional information of the laparoscope, the
OrbSLAM2 [11] algorithm is used to estimate the camera pose for each frame.
Furthermore, the disparity between each pair of stereo frames is estimated
(using a disparity CNN [18]) and a semantic segmentation is performed on each
left camera frame to identify pixels which show the liver (using a
segmentation CNN [13]). We reproject the pixel information into the scene
using the disparity, camera calibration parameters and estimated camera pose,
obtaining a 3D point cloud with color and segmentation information. While the
camera moves, points which are close together and are similar in hue are
merged together. After the sweep, we discard points if they have not been
merged multiple times (i.e. points which were not seen from multiple view
points) or if they were not segmented as _liver_ in at least 70% of the frames
in which they were seen. Additionally, we use a moving least squares filter
(radius 0.5 cm) to smoothen the surface. After a manual rigid alignment of
$V_{P}$ and $S_{I}$, the distance fields for the pre- and intraoperative data
are calculated and our network is used to estimate a displacement of the liver
volume. Qualitative results are shown in Fig. 1 and in the supplementary
material.

##  4 Conclusion

In this work, we have shown that a CNN can learn to register a full liver
surface to a deformed partial surface of the same liver. The network was never
trained on a real organ or real deformation during the training process, and
yet it learned to solve the surface correspondence problem as well as the
underlying biomechanical constraints.

Since our method works directly on the surfaces, no assignment of boundary
conditions and no search for correspondences is necessary. The breathing
motion experiment shows that, even though the segmentation process creates
some artifacts in the surfaces, the V2S-Net finds a valid solution. The
network was able to find the displacement without the need for a prior rigid
alignment, likely because the full intraoperative surface was used. In cases
with less visible surface, we find that the solution depends on the prior
rigid alignment, which should be investigated further in future work.

In our method, we outsource the complex and time consuming simulation of soft-
tissue behavior to the data generation stage. This could be further exploited
by adding additional information to the simulations, such as inhomogeneous
material properties, surrounding and connecting tissues and more complex
boundary conditions. Where these properties are measurable for a patient, they
could be passed to the network as additional input channels.

Our results show that there may be cases where the solution is ambiguous, for
example when too little information is given and the network must guess how
the hidden side of the liver is deformed. This issue is likely inherent to the
ill-posed laparoscopic registration problem in itself and not confined to our
method. However, in contrast to conventional registration methods, neural
networks can estimate how confident they are of a solution [4]. This
probabilistic output could be used to assess how a solution should be
interpreted and could give additional information to the surgeon.

## References

  * [1] Bilic, P., Christ, P.F., Vorontsov, E., Chlebus, G., Chen, H., Dou, Q., Fu, C.W., Han, X., Heng, P.A., Hesser, J., Kadoury, S., Konopczynski, T.K., Le, M., Li, C., Li, X., Lipková, J., Lowengrub, J.S., Meine, H., Moltz, J.H., Pal, C.J., Piraud, M., Qi, X., Qi, J., Rempfler, M., Roth, K., Schenk, A., Sekuboyina, A., Zhou, P., Hülsemeyer, C., Beetz, M.J., Ettlinger, F., Grün, F., Kaissis, G.A., Lohöfer, F., Braren, R.F., Holch, J., Hofmann, F., Sommer, W.H., Heinemann, V., Jacobs, C., Mamani, G.E.H., van Ginneken, B., Chartrand, G., Tang, A., Drozdzal, M., Ben-Cohen, A., Klang, E., Amitai, M.M., Konen, E., Greenspan, H., Moreau, J., Hostettler, A., Soler, L., Vivanti, R., Szeskin, A., Lev-Cohain, N., Sosna, J., Joskowicz, L., Menze, B.H.: The liver tumor segmentation benchmark (lits). ArXiv abs/1901.04056 (2019)
  * [2] Brunet, J.N., Mendizabal, A., Petit, A., Golse, N., Vibert, E., Cotin, S.: Physics-based deep neural network for augmented reality during liver surgery. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2019\. Springer International Publishing (2019)
  * [3] Charles, R., Su, H., Kaichun, M., Guibas, L.: Pointnet: Deep learning on point sets for 3d classification and segmentation. pp. 77–85 (07 2017). https://doi.org/10.1109/CVPR.2017.16
  * [4] Gal, Y., Ghahramani, Z.: Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48. p. 1050–1059. ICML’16, JMLR.org (2016)
  * [5] Geuzaine, C., Remacle, J.F.: Gmsh: A 3-d finite element mesh generator with built-in pre- and post-processing facilities. International Journal for Numerical Methods in Engineering 79(11), 1309–1331 (2009). https://doi.org/10.1002/nme.2579, https://onlinelibrary.wiley.com/doi/abs/10.1002/nme.2579
  * [6] Griffiths, D., Boehm, J.: A review on deep learning techniques for 3d sensed data classification. Remote Sensing 11(12) (2019)
  * [7] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2017)
  * [8] Malinen, M., Råback, P.: Elmer Finite Element Solver for Multiphysics and Multiscale Problems, pp. 101–113. Multiscale Modelling Methods for Applications in Materials Science, Forschungszentrum Jülich (2013)
  * [9] Mendizabal, A., Márquez-Neila, P., Cotin, S.: Simulation of hyperelastic materials in real-time using deep learning. Medical Image Analysis 59, 101569 (10 2019). https://doi.org/10.1016/j.media.2019.101569
  * [10] Mendizabal, A., Tagliabue, E., Brunet, J.N., Dallálba, D., Fiorini, P., Cotin, S.: Physics-based Deep Neural Network for Real-Time Lesion Tracking in Ultrasound-guided Breast Biopsy. In: Computational Biomechanics for Medicine XIV. Shenzhen, China (Oct 2019), https://hal.inria.fr/hal-02311277
  * [11] Mur-Artal, R., Tardós, J.D.: ORB-SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras. IEEE Transactions on Robotics 33(5), 1255–1262 (2017). https://doi.org/10.1109/TRO.2017.2705103
  * [12] Pellicer-Valero, O.J., Rupérez, M.J., Martínez-Sanchis, S., Martín-Guerrero, J.D.: Real-time biomechanical modeling of the liver using machine learning models trained on finite element method simulations. Expert Systems with Applications 143, 113083 (2020). https://doi.org/https://doi.org/10.1016/j.eswa.2019.113083, http://www.sciencedirect.com/science/article/pii/S0957417419308000
  * [13] Pfeiffer, M., Funke, I., Robu, M.R., Bodenstedt, S., Strenger, L., Engelhardt, S., Roß, T., Clarkson, M.J., Gurusamy, K.S., Davidson, B.R., Maier-Hein, L., Riediger, C., Welsch, T., Weitz, J., Speidel, S.: Generating large labeled data sets for laparoscopic image processing tasks using unpaired image-to-image translation. In: MICCAI (2019)
  * [14] Pfeiffer, M., Riediger, C., Weitz, J., Speidel, S.: Learning soft tissue behavior of organs for surgical navigation with convolutional neural networks. International Journal of Computer Assisted Radiology and Surgery 14, 1147–1155 (2019)
  * [15] Smith, L., Topin, N.: Super-convergence: Very fast training of residual networks using large learning rates (08 2017)
  * [16] Suwelack, S., Röhl, S., Bodenstedt, S., Reichard, D., Dillmann, R., Santos, T., Maier-Hein, L., Wagner, M., Wünscher, J., Kenngott, H., Müller, B., Speidel, S.: Physics-based shape matching for intraoperative image guidance. Medical physics 41, 111901 (11 2014). https://doi.org/10.1118/1.4896021
  * [17] Wang, H., Guo, J., Yan, D.M., Quan, W., Zhang, X.: Learning 3d keypoint descriptors for non-rigid shape matching. In: The European Conference on Computer Vision (ECCV) (September 2018)
  * [18] Yang, G., Manela, J., Happold, M., Ramanan, D.: Hierarchical deep stereo matching on high-resolution images. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)