11institutetext: Institute of AI for Health, Helmholtz Munich – German
Research Centre for Environmental Health, Neuherberg, Germany 22institutetext:
Helmholtz Pioneer Campus, Helmholtz Munich– German Research Centre for
Environmental Health, Neuherberg, Germany 33institutetext: Technical
University of Munich, Munich, Germany 33email: bastian.rieck@helmholtz-
muenchen.de

# Capturing Shape Information with Multi-Scale Topological Loss Terms for 3D
Reconstruction

Dominik J. E. Waibel 1133 0000-0002-3753-8598 Scott Atwell 22
0000-0001-9943-3524 Matthias Meier 22 0000-0001-9455-4538 Carsten Marr 11
0000-0003-2154-4552 Bastian Rieck 112233 0000-0003-4335-0302

###### Abstract

Reconstructing 3D objects from 2D images is both challenging for our brains
and machine learning algorithms. To support this spatial reasoning task,
contextual information about the overall shape of an object is critical.
However, such information is not captured by established loss terms (e.g. Dice
loss). We propose to complement geometrical shape information by including
multi-scale topological features, such as connected components, cycles, and
voids, in the reconstruction loss. Our method uses cubical complexes to
calculate topological features of 3D volume data and employs an optimal
transport distance to guide the reconstruction process. This topology-aware
loss is fully differentiable, computationally efficient, and can be added to
any neural network. We demonstrate the utility of our loss by incorporating it
into SHAPR, a model for predicting the 3D cell shape of individual cells based
on 2D microscopy images. Using a hybrid loss that leverages both geometrical
and topological information of single objects to assess their shape, we find
that topological information substantially improves the quality of
reconstructions, thus highlighting its ability to extract more relevant
features from image datasets.

###### Keywords:

topological loss cubical complex 3D shape prediction

##  1 Introduction

Segmentation and reconstruction are common tasks when dealing with imaging
data. Especially in the biomedical domain, segmentation accuracy can have a
substantial impact on complex downstream tasks, such as a patient’s diagnosis
and treatment. 3D segmentation is a complex task in itself, requiring the
assessment and labelling of each voxel in a volume, which, in turn,
necessitates a high-level understanding of the object and its context.
However, complexity rapidly increases when attempting to reconstruct a 3D
object from a 2D projection since 3D images may often be difficult to obtain.
This constitutes an inverse problem with intrinsically ambiguous solutions:
each 2D image permits numerous 3D reconstructions, similar to how a shadow
alone does not necessarily permit conclusions to be drawn about the
corresponding shape. When addressed using machine learning, the solution of
such inverse problems can be facilitated by imbuing a model with additional
inductive biases about the structural properties of objects. Existing models
supply such inductive biases to the reconstruction task mainly via geometry-
based objective functions, thus learning a likelihood function
$f\colon\mathcal{V}\to\mathds{R}$, where $f(x)$ for $x\in\mathcal{V}$ denotes
the likelihood that a voxel $x$ of the input volume $\mathcal{V}$ is part of
the ground truth shape [18]. Loss functions to learn $f$ are evaluated on a
per-voxel basis, assessing the differences between the original volume and the
predicted volume in terms of overlapping labels. Commonly-used loss functions
include binary cross entropy (BCE), Dice loss, and mean squared error (MSE).
Despite their expressive power, these loss terms do not capture structural
shape properties of the volumes.

##### Our contributions.

Topological features, i.e. features that characterise data primarily in terms
of _connectivity_ , have recently started to emerge as a powerful paradigm for
complementing existing machine learning methods [13]. They are capable of
capturing shape information of objects at multiple scales and along multiple
dimensions. In this paper, we leverage such features and integrate them into a
novel differentiable ‘topology-aware’ loss term
$\operatorname{\mathcal{L}}_{\text{T}}$ that can be used to regularise the
shape reconstruction process. Our loss term handles arbitrary shapes, can be
computed efficiently, and may be integrated into general deep learning models.
We demonstrate the utility of $\operatorname{\mathcal{L}}_{\text{T}}$ by
combining it with SHAPR [29], a framework for predicting individual cell
shapes from 2D microscopy images. The new hybrid variant of SHAPR, making use
of both geometry-based and topology-based objective functions, results in
improved reconstruction performance along multiple metrics.

##  2 Related Work

Several deep learning approaches for predicting 3D shapes of single objects
from 2D information already exist; we aim to give a brief overview. Previous
work includes predicting natural objects such as air planes, cars, and
furniture from photographs, creating either meshes [11, 30], voxel volumes
[3], or point clouds [9]. A challenging biomedical task, due to the occurrence
of imaging noise, is tackled by Waibel et al. [29], whose SHAPR model predicts
the shape and morphology of individual mammalian cells from 2D microscopy
images. Given the multi-scale nature of microscopy images, SHAPR is an ideal
use case to analyse the impact of employing additional topology-based loss
terms for these reconstruction tasks.

Such loss terms constitute a facet of the emerging field of _topological
machine learning_ and _persistent homology_ , its flagship algorithm (see
Section 3.1 for an introduction). Previous studies have shown great promise in
using topological losses for image segmentation tasks or their evaluation
[25]. In contrast to our loss, existing work relies on prior knowledge about
‘expected’ topological features [4], or enforces a pre-defined set of
topological features based on comparing segmentations [15, 16].

##  3 Our Method: A Topology-Aware Loss

We propose a topology-aware loss term based on concepts from topological
machine learning and optimal transport. The loss term works on the level of
individual volumes, leveraging a valid metric between topological descriptors,
while remaining efficiently computable. Owing to its generic nature, the loss
can be easily integrated into existing architectures; see Fig. 1 for an
overview.

Figure 1: Given a predicted object and a 3D ground truth object, we calculate
topological features using cubical persistent homology, obtaining a set of
persistence diagrams. Each point in a persistence diagram denotes the creation
and destruction of a $d$-dimensional topological feature of the given object.
We compare these diagrams using $\operatorname{\mathcal{L}}_{\text{T}}$, our
novel topology-based loss, combining it with geometrical loss terms such as
binary cross entropy (BCE). Dotted components can be swapped out.

###  3.1 Assessing the Topology of Volumes

Given a volume $\mathcal{V}$, i.e. a $d$-dimensional tensor of shape
$n_{1}\times n_{2}\times\dots\times n_{d}$, we represent it as a _cubical
complex_ $C$. A cubical complex contains individual voxels of $\mathcal{V}$ as
vertices , along with connectivity information about their neighbourhoods via
edges , squares , and their higher-dimensional counterparts.111Expert readers
may recognise that cubical complexes are related to meshes and simplicial
complexes but use squares instead of triangles as their building blocks.
Cubical complexes provide a fundamental way to represent volume data and have
proven their utility in previous work [24, 28]. Topological features of
different dimensions are well-studied, comprising connected components (0D),
cycles (1D), and voids (2D), for instance. The number of $k$-dimensional
topological features is also referred to as the $k$th _Betti number_
$\beta_{k}$ of $C$. While previous work has shown the efficacy of employing
Betti numbers as a topological prior for image segmentation tasks [4, 15], the
reconstruction tasks we are considering in this paper require a multi-scale
perspective that cannot be provided by Betti numbers, which are mere feature
counts. We therefore make use of _persistent homology_ , a technique for
calculating multi-scale topological features [8]. This technique is
particularly appropriate in our setting: our model essentially learns a
likelihood function $f\colon\mathcal{V}\to\mathds{R}$. To each voxel
$x\in\mathcal{V}$, the function $f$ assigns the likelihood of $x$ being part
of an object’s shape. For a likelihood threshold $\tau\in\mathds{R}$, we
obtain a cubical complex $C^{(\tau)}:=\\{x\in\mathcal{V}\mid f(x)\geq\tau\\}$
and, consequently, a different set of topological features. Since volumes are
finite, their topology only changes at a finite number of thresholds
$\tau_{1}\geq\dotsc\geq\tau_{m}$, and we obtain a sequence of nested cubical
complexes $\emptyset\subseteq C^{(\tau_{1})}\subseteq
C^{(\tau_{2})}\subseteq\dots\subseteq C^{(\tau_{m})}=\mathcal{V}$, known as a
_superlevel set filtration_. Persistent homology tracks topological features
across all complexes in this filtration, representing each feature as a tuple
$(\tau_{i},\tau_{j})$, with $\tau_{i}\geq\tau_{j}$, indicating the cubical
complex in which a feature was being ‘created’ and ‘destroyed,’ respectively.
The tuples of $k$-dimensional features, with $0\leq k\leq d$, are stored in
the $k$th _persistence diagram_ $\mathcal{D}_{f}^{(k)}$ of the data set.222We
use the subscript $f$ to indicate the corresponding likelihood function; we
will drop this for notational convenience when discussing general properties.
Persistence diagrams thus form a multi-scale shape descriptor of all
topological features of a dataset. Despite the apparent complexity of
filtrations, persistent homology of cubical complexes can be calculated
efficiently in practice [28].

##### Structure of persistence diagrams.

Persistent homology provides information beyond Betti numbers: instead of
enforcing a choice of threshold $\tau$ for the likelihood function, which
would result in a fixed set of Betti numbers, persistence diagrams encode all
thresholds at the same time, thus capturing additional geometrical details
about data. Given a tuple $(\tau_{i},\tau_{j})$ in a persistence diagram, its
_persistence_ is defined as
$\operatorname{pers}\left(\tau_{i},\tau_{j}\right):=\left|\tau_{j}-\tau_{i}\right|$.
Persistence indicates the ‘scale’ over which a topological feature occurs,
with large values typically assumed to correspond to more stable features. The
sum of all persistence values is known as the _degree- $p$ total persistence_,
i.e.
$\operatorname{Pers}_{p}(\mathcal{D}_{f}):=\sum_{(\tau_{i},\tau_{j})\in\mathcal{D}_{f}}\left|\operatorname{pers}(\tau_{i},\tau_{j})^{p}\right|$.
It constitutes a stable summary statistic of topological activity [7].

##### Comparing persistence diagrams.

Persistence diagrams can be endowed with a metric by using optimal transport.
Given two diagrams $\mathcal{D}$ and $\mathcal{D}^{\prime}$ containing
features of the same dimensionality, their $p$th _Wasserstein distance_ is
defined as

$\operatorname{W}_{p}\left(\mathcal{D},\mathcal{D}^{\prime}\right):=\left(\inf_{\eta\colon\mathcal{D}\to\mathcal{D}^{\prime}}\sum_{x\in\mathcal{D}}\|x-\eta(x)\|_{\infty}^{p}\right)^{\frac{1}{p}},$
(1)

where $\eta(\cdot)$ denotes a bijection. Since $\mathcal{D}$ and
$\mathcal{D}^{\prime}$ generally have different cardinalities, we consider
them to contain an infinite number of points of the form $(\tau,\tau)$, i.e.
tuples of zero persistence. A suitable $\eta(\cdot)$ can thus always be found.
Solving Eq. 1 is practically feasible using modern optimal transport
algorithms [10].

##### Stability.

A core property of persistence diagrams is their stability to noise. While
different notions of stability exist for persistent homology [7], a recent
theorem [27] states that the Wasserstein distance between persistence diagrams
of functions $f,f^{\prime}\colon\mathcal{V}\to\mathds{R}$ is bounded by their
$p$-norm, i.e.

$\operatorname{W}_{p}\left(\mathcal{D}_{f}^{(k)},\mathcal{D}_{f^{\prime}}^{(k)}\right)\leq
C\left\|f-f^{\prime}\right\|_{p}\text{ for $0\leq k\leq d$},$ (2)

with $C\in\mathds{R}_{>0}$ being a constant that depends on the dimensionality
of $\mathcal{V}$. Eq. 2 implies that gradients obtained from the Wasserstein
distance and other topology-based summaries will remain bounded; we will also
use it to accelerate topological feature calculations in practice.

##### Differentiability.

Despite the discrete nature of topological features, persistent homology
permits the calculation of gradients with respect to parameters of the
likelihood function $f$, thus enabling the use of automatic differentiation
schemes [14, 19, 22]. A seminal work by Carrière et al. [2] proved that
optimisation algorithms converge for a wide class of persistence-based
functions, thus opening the door towards general topology-based optimisation
schemes.

###  3.2 Loss Term Construction

Given a true likelihood function $f$ and a predicted likelihood function
$f^{\prime}$, our novel generic topology-aware loss term takes the form

$\operatorname{\mathcal{L}}_{\text{T}}\left(f,f^{\prime},p\right):=\sum_{i=0}^{d}\operatorname{W}_{p}\left(\mathcal{D}_{f}^{(i)},\mathcal{D}_{f^{\prime}}^{(i)}\right)+\operatorname{Pers}\left(\mathcal{D}_{f^{\prime}}^{(i)}\right).$
(3)

The first part of Eq. 3 incentivises the model to reduce the distance between
$f$ and $f^{\prime}$ with respect to their topological shape information. The
second part incentivises the model to reduce overall topological activity,
thus decreasing the noise in the reconstruction. This can be considered as the
topological equivalent of reducing the _total variation_ of a function [23].
Given a task-specific geometrical loss term
$\operatorname{\mathcal{L}}_{\text{G}}$,333We dropped all hyperparameters of
the loss term for notational clarity.  such as a Dice loss, we obtain a
combined loss term as
$\operatorname{\mathcal{L}}:=\operatorname{\mathcal{L}}_{\text{G}}+\lambda\operatorname{\mathcal{L}}_{\text{T}}$,
where $\lambda\in\mathds{R}_{>0}$ controls the impact of the topology-based
part. We will use $p=2$ since Eq. 2 relates
$\operatorname{\mathcal{L}}_{\text{T}}$ to the Euclidean distance in this
case.

##### Calculations in practice.

To speed up the calculation of our loss term, we utilise the stability theorem
of persistent homology and downsample each volume to $M\times M\times M$
voxels using trilinear interpolation. We provide a theoretical and empirical
analysis of the errors introduced by downsampling in the Supplementary
Materials. In our experiments, we set $M=16$, which is sufficiently small to
have no negative impact on computational performance while at the same time
resulting in empirical errors $\leq 0.1$ (measured using Eq. 2 for $p=2$).

##  4 Experiments

We provide a brief overview of SHAPR before discussing the experimental setup,
datasets, and results. SHAPR is a deep learning method to predict 3D shapes of
single cells from 2D microscopy images [29]. Given a 2D fluorescent image of a
single cell and a corresponding segmentation mask, SHAPR predicts the 3D shape
of this cell. The authors suggest to train SHAPR with a combination of Dice
and BCE loss, with an additional adversarial training step to improve the
predictions of the model. We re-implemented SHAPR using PyTorch [21] to ensure
fair comparisons and the seamless integration of our novel loss function
$\operatorname{\mathcal{L}}_{\text{T}}$, employing ‘Weights & Biases’ for
tracking experiments [1]. Our code and reports are publicly available.444 See
https://github.com/marrlab/SHAPR˙torch.

##### Data.

For our experiments, we use the two datasets published with the original SHAPR
manuscript [29]. The first dataset comprises $825$ red blood cells, imaged in
3D with a confocal microscope [26]. Each cell is assigned to one of nine pre-
defined classes: sphero-, stomato-, disco-, echino-, kerato-, knizo-, and
acanthocytes, as well as cell clusters and multilobates. The second dataset
contains $887$ nuclei of human-induced pluripotent stem cells (iPSCs),
counterstained and imaged in 3D with a confocal microscope. Cells were
manually segmented to create ground truth objects. Both datasets include 3D
volumes of size $64\times 64\times 64$, 2D segmentation masks of size
$64\times 64$, and fluorescent images of size $64\times 64$.

###  4.1 Training and Evaluation

We trained our implementation of SHAPR for a maximum of $100$ epochs, using
early stopping with a patience of $15$ epochs, based on the validation loss.
For each run, we trained five SHAPR models in a round-robin fashion,
partitioning the dataset into five folds with a 60%/20%/20%
train/validation/test split, making sure that each 2D input image appears once
in the test set. To compare the performance of SHAPR with and without
$\operatorname{\mathcal{L}}_{\text{T}}$, we used the same hyperparameters for
all experiments (initial learning rate of $1\text{\times}{10}^{-3}$,
$\beta_{1}=0.9$, and $\beta_{2}=0.999$ for the ADAM optimiser). We optimised
$\lambda\in\\{$1\text{\times}{10}^{-3}$,$1\text{\times}{10}^{-2}$,\dots,$1\text{\times}{10}^{2}$\\}$,
the regularisation strength parameter for
$\operatorname{\mathcal{L}}_{\text{T}}$, on an independent dataset, resulting
in $\lambda=0.1$ for all experiments. We also found that evaluating Eq. 3 for
each dimension individually leads to superior performance; we thus only
calculate Eq. 3 for $i=2$. Finally, for the training phase, we augmented the
data with random horizontal or vertical flipping and $90^{\circ}$ rotations
with a 33% chance for each augmentation to be applied for a sample. The goal
of these augmentations is to increase data variability and prevent
overfitting.

Following Waibel et al. [29], we evaluated the performance of SHAPR by
calculating  (i) the _intersection over union_ (IoU) error,  (ii) the relative
volume error,  (iii) the relative surface area error, and  (iv) the relative
surface roughness error  with respect to the ground truth data, applying
Otsu’s method [20] for thresholding predicted shapes. We calculate the volume
by counting non-zero voxels, the surface area as all voxels on the surface of
an object, and the surface roughness as the difference between the surface
area of the object and the surface area of the same object after smoothing it
with a 3D Gaussian [29].

(a)
---
(b)

Figure 2: 2(a) Examples of predictions without (top row) and with (middle row)
$\operatorname{\mathcal{L}}_{\text{T}}$, our topological loss term. The third
row shows ground truth images. 2(b) $\operatorname{\mathcal{L}}_{\text{T}}$
improves predictions in relevant metrics, such as the IoU error, the relative
volume error, relative surface area error, and relative surface roughness
error.

###  4.2 Results

To evaluate the benefits of our topology-aware loss, we perform the same
experiment twice: first, using a joint BCE and Dice loss [29], followed by
adding $\lambda\operatorname{\mathcal{L}}_{\text{T}}$. Without
$\operatorname{\mathcal{L}}_{\text{T}}$, we achieve results comparable to the
original publication (IoU, red blood cell data: $0.63\pm 0.12\%$; IoU, nuclei
data: $0.46\pm 0.16\%$); minor deviations arise from stochasticity and
implementation differences between PyTorch and Tensorflow. We observe superior
performance in the majority of metrics for both datasets when adding
$\operatorname{\mathcal{L}}_{\text{T}}$ to the model (see Fig. 2 and Table 1);
the performance gains by $\operatorname{\mathcal{L}}_{\text{T}}$ are
statistically significant in all but one case. Notably, we find that
$\operatorname{\mathcal{L}}_{\text{T}}$ increases SHAPR’s predictive
performance unevenly across the classes of the red blood cell dataset (see
Fig. 2(a)). For spherocytes (round cells), only small changes in IoU error,
relative volume error, relative surface area error, and relative surface
roughness error (7% decrease, 2% decrease, and 3% decrease, respectively)
occur, whereas for echinocytes (cells with a spiky surface), we obtain a 27%
decrease in IoU error and 7% decrease in relative volume error. Finally, for
discocytes (bi-concave cells) and stomatocytes, we obtain a 2% decrease in IoU
error, a 3% decrease in volume error, a 11% decrease in surface area error,
and a 25% decrease in surface roughness error upon adding
$\operatorname{\mathcal{L}}_{\text{T}}$.

Table 1:  Median, mean ($\mu$) and standard deviation ($\sigma$) of several relative error measures for two datasets (lower values are better; winner shown in bold). The $\operatorname{\mathcal{L}}_{\text{T}}$ column indicates whether our new loss term was active. We also show the $p$-value of a paired Wilcoxon signed-rank test between error distributions.  |  | Red blood cell ($n=825$) | Nuclei ($n=887$)
---|---|---|---
Relative error | $\operatorname{\mathcal{L}}_{\text{T}}$ | Median | $\mu\pm\sigma$ | $p$ | Median | $\mu\pm\sigma$ | $p$
$1-\mathrm{IoU}$ | ✗ | $0.48$ |  $0.49$ p m 0.09 | $1.1\text{\times}{10}^{-19}$ | $0.62$ |  $0.62$ p m 0.11 | $0.491$
✓ | $0.47$ |  $0.47$ p m 0.10 | $0.61$ |  $0.61$ p m 0.11
Volume | ✗ | $0.31$ |  $0.35$ p m 0.31 | $1.2\text{\times}{10}^{-47}$ | $0.34$ |  $0.48$ p m 0.47 | $4.6\text{\times}{10}^{-14}$
✓ | $0.26$ |  $0.29$ p m 0.27 | $0.32$ |  $0.43$ p m 0.42
Surface area | ✗ | $0.20$ |  $0.24$ p m 0.20 | $39.5\text{\times}{10}^{-13}$ | $0.21$ |  $0.27$ p m 0.25 | $1.7\text{\times}{10}^{-8}$
✓ | $0.14$ |  $0.18$ p m 0.16 | $0.18$ |  $0.25$ p m 0.24
Surface roughness | ✗ | $0.35$ |  $0.36$ p m 0.24 | $9.1\text{\times}{10}^{-4}$ | $0.17$ |  $0.18$ p m 0.12 | $1.5\text{\times}{10}^{-6}$
✓ | $0.24$ |  $0.29$ p m 0.22 | $0.18$ |  $0.19$ p m 0.13

##  5 Discussion

We propose a novel topology-aware loss term
$\operatorname{\mathcal{L}}_{\text{T}}$ that can be integrated into existing
deep learning models. Our loss term is not restricted to specific types of
shapes and may be applied to reconstruction and segmentation tasks. We
demonstrate its efficacy in the reconstruction of 3D shapes from 2D microscopy
images, where the results of SHAPR are statistically significantly improved in
relevant metrics whenever $\operatorname{\mathcal{L}}_{\text{T}}$ is jointly
optimised together with geometrical loss terms. Notably,
$\operatorname{\mathcal{L}}_{\text{T}}$ does not optimise classical
segmentation/reconstruction metrics, serving instead as an inductive bias for
incorporating multi-scale information on topological features.
$\operatorname{\mathcal{L}}_{\text{T}}$ is computationally efficient and can
be adapted to different scenarios by incorporating topological features of a
specific dimension. Since our experiments indicate that the calculation of Eq.
3 for a single dimension is sufficient to achieve improved reconstruction
results in practice, we will leverage topological duality/symmetry theorems
[6] in future work to improve computational efficiency and obtain smaller
cubical complexes.

Our analysis of predictive performance across classes of the red blood cell
dataset (see Table 1) leads to the assumption that the topological loss term
shows the largest reconstruction performance increases on shapes that have
complex morphological features, such as echinocytes or bi-concavely shaped
cells (discocytes and stomatocytes). This implies that future extensions of
the method should incorporate additional geometrical descriptors into the
filtration calculation, making use of recent advances in capturing the
topology of multivariate shape descriptors [17].

####  5.0.1 Acknowledgements.

We thank Lorenz Lamm, Melanie Schulz, Kalyan Varma Nadimpalli, and Sophia
Wagner for their valuable feedback to this manuscript. The authors also are
indebted to Teresa Heiss for discussions on the topological changes induced by
downsampling volume data.

####  5.0.2 Funding.

Carsten Marr received funding from the European Research Council (ERC) under
the European Union’s Horizon 2020 Research and Innovation Programme (Grant
Agreement 866411).

####  5.0.3 Author contributions.

DW and BR implemented code and conducted experiments. DW, BR, and CM wrote the
manuscript. DW created figures and BR the main portrayal of results. SA and MM
provided the 3D nuclei dataset. BR supervised the study. All authors have read
and approved the manuscript.

##  Appendix 0.A Discussion: The Effects of Interpolation

As outlined in the main paper, we downsample our volumes from $64^{3}$ voxels
to $16^{3}$ voxels using _trilinear interpolation_. To assess the impact this
has on the calculation of persistent homology, we can make use of recent work
on the topological impact of changing image resolutions [12]. Following Heiss
et al. [12], we treat downsampling as a general function that transforms a
volume of side length $r_{1}\in\mathds{N}$ into a volume of side length
$r_{2}\in\mathds{N}$. Assuming that $r_{1}>r_{2}>0$ and $r_{2}\mid r_{1}$, we
refer to $a:=\nicefrac{{r_{1}}}{{r_{2}}}$ as the _compression factor_. In $d$
dimensions, every voxel of size $r_{2}$ contains $a^{d}$ voxels of size
$r_{1}$, making it possible to compare the two volumes by repeating the values
of the smaller volume.

While the bounds given by Heiss et al. [12] make use of the _bottleneck
distance_ instead of the Wasserstein distance, we can still use the
equivalence of norms to obtain bounds. To this end, we first observe that the
likelihood function $f$ that we use in the paper is Lipschitz continuous with
Lipschitz constant $L=1$. This is a direct consequence of the image of the
function being restricted to $[0,1]$. Corollary III.4 of Heiss et al. [12] now
states the following:555We use a slightly different formulation that is
aligned with the notation of our main paper. Moreover, we consider $r_{1}$ to
refer to the larger side length, as outlined above.

###### Corollary 1

If $f\colon\mathds{R}^{d}\to\mathds{R}$ is Lipschitz continuous with Lipschitz
constant $L$, then
$\operatorname{d_{B}}(\mathcal{D}_{f_{r_{1}}},\mathcal{D}_{f_{r_{2}}})\leq
Lr_{1}\sqrt{d}$, where $\operatorname{d_{B}}(\cdot,\cdot)$ refers to the
bottleneck distance between two persistence diagrams, and
$\mathcal{D}_{f_{r_{1}}},\mathcal{D}_{f_{r_{2}}}$ denote the persistence
diagrams of the likelihood functions for volumes with side lengths $r_{1}$ and
$r_{2}$, respectively.

This corollary makes use of the fact that
$\|f_{r_{1}}-f_{r_{2}}\|_{\infty}\leq Lr_{1}\sqrt{d}$, such that the bound
follows as a consequence of the stability theorem of persistent homology [5,
27]. Using the equivalence of norms in finite-dimensional vector spaces, we
know that for every $x\in\mathds{R}^{n}$,

$\|x\|_{\infty}\leq\|x\|_{2}\leq\sqrt{n}\|x\|_{\infty}.$ (4)

Treating the values of our likelihood function as a vector in $n:=r_{1}^{d}$
dimensions, we obtain an upper bound according to Eq. 4 as

$\|f_{r_{1}}-f_{r_{2}}\|_{2}\leq Lr_{1}\sqrt{d}\sqrt{r_{1}^{d}}.$ (5)

This is a worst-case bound; it does not incorporate the fact that as $r_{2}$
approaches $r_{1}$, interpolation errors decrease. Moreover, the bound does
not incorporate the structure of the interpolation scheme; for specific types
of interpolation schemes, tighter bounds can be derived.

##### Empirical errors.

In practice, we observe substantially smaller errors. Fig. 3 depicts empirical
interpolation errors in terms of the Wasserstein distance between persistence
diagrams arising from the original space and an interpolated variant. The
computational performance gains that we get from using only $16^{3}$ voxels
(instead of $64^{3}$ voxels) for the topological calculations are shown to be
accompanied by small approximation errors.

##### Code.

Along with our model, we also publish an additional script for readers
interested in reproducing these plots. To run the analysis for $s=8$, for
instance, you may use the following call:

    python -m scripts.analyse_interpolation -s 8 -p config/small-0D.json

Please refer to our code repository666See
https://github.com/marrlab/SHAPR˙torch.  for more details.

$32$$16$$8$$0$$0.1$$0.2$$0.3$$0.4$$0.5$$0.6$$0.7$$0.8$$0.9$$1$Side length of
interpolated volume
($r_{2}$)$\operatorname{W}_{2}\left(\mathcal{D}_{f_{r_{1}}},\mathcal{D}_{f_{r_{2}}}\right)$
Figure 3: Empirical observation errors for a small test data set. We show that
error with respect to the second Wasserstein distance $\operatorname{W}_{2}$
between the persistence diagrams generated from the original function
($r_{1}$) and a lower-resolution version with side length $r_{2}$. The mean
error for $r_{2}\in\\{16,32\\}$ is bounded by $\approx 0.1$, which we deem
satisfactory for subsequent calculations.

## References

  * [1] Biewald, L.: Experiment tracking with Weights and Biases (2020), https://www.wandb.com/
  * [2] Carrière, M., Chazal, F., Glisse, M., Ike, Y., Kannan, H., Umeda, Y.: Optimizing persistent homology based functions. In: Proceedings of the 38th International Conference on Machine Learning. pp. 1294–1303 (2021)
  * [3] Choy, C.B., Xu, D., Gwak, J., Chen, K., Savarese, S.: 3D-R2N2: A unified approach for single and multi-view 3D object reconstruction. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 628–644 (2016)
  * [4] Clough, J., Byrne, N., Oksuz, I., Zimmer, V.A., Schnabel, J.A., King, A.: A topological loss function for deep-learning based image segmentation using persistent homology. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020)
  * [5] Cohen-Steiner, D., Edelsbrunner, H., Harer, J.: Stability of persistence diagrams. Discrete & Computational Geometry 37(1), 103–120 (2007)
  * [6] Cohen-Steiner, D., Edelsbrunner, H., Harer, J.: Extending persistence using Poincaré and Lefschetz duality. Foundations of Computational Mathematics 9(1), 79–103 (2009)
  * [7] Cohen-Steiner, D., Edelsbrunner, H., Harer, J., Mileyko, Y.: Lipschitz functions have $\mathrm{L}_{p}$-stable persistence. Foundations of Computational Mathematics 10(2), 127–139 (2010)
  * [8] Edelsbrunner, H., Letscher, D., Zomorodian, A.J.: Topological persistence and simplification. Discrete & Computational Geometry 28(4), 511–533 (2002)
  * [9] Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3D object reconstruction from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
  * [10] Flamary, R., Courty, N., Gramfort, A., Alaya, M.Z., Boisbunon, A., Chambon, S., Chapel, L., Corenflos, A., Fatras, K., Fournier, N., Gautheron, L., Gayraud, N.T., Janati, H., Rakotomamonjy, A., Redko, I., Rolet, A., Schutz, A., Seguy, V., Sutherland, D.J., Tavenard, R., Tong, A., Vayer, T.: POT: Python optimal transport. Journal of Machine Learning Research 22(78), 1–8 (2021)
  * [11] Gkioxari, G., Malik, J., Johnson, J.: Mesh R-CNN. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019)
  * [12] Heiss, T., Tymochko, S., Story, B., Garin, A., Bui, H., Bleile, B., Robins, V.: The impact of changes in resolution on the persistent homology of images. In: IEEE International Conference on Big Data. pp. 3824–3834 (2021)
  * [13] Hensel, F., Moor, M., Rieck, B.: A survey of topological machine learning methods. Frontiers in Artificial Intelligence 4 (2021)
  * [14] Hofer, C.D., Graf, F., Rieck, B., Niethammer, M., Kwitt, R.: Graph filtration learning. In: Proceedings of the 37th International Conference on Machine Learning (ICML). pp. 4314–4323 (2020)
  * [15] Hu, X., Li, F., Samaras, D., Chen, C.: Topology-preserving deep image segmentation. In: Advances in Neural Information Processing Systems. vol. 32 (2019)
  * [16] Hu, X., Wang, Y., Fuxin, L., Samaras, D., Chen, C.: Topology-aware segmentation using discrete Morse theory. In: International Conference on Learning Representations (2021)
  * [17] Lesnick, M., Wright, M.: Interactive visualization of 2-D persistence modules arXiv:1512.00180 (2015)
  * [18] Minaee, S., Boykov, Y., Porikli, F., Plaza, A., Kehtarnavaz, N., Terzopoulos, D.: Image segmentation using deep learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(7), 3523–3542 (2021)
  * [19] Moor, M., Horn, M., Rieck, B., Borgwardt, K.: Topological autoencoders. In: Proceedings of the 37th International Conference on Machine Learning (ICML). pp. 7045–7054 (2020)
  * [20] Otsu, N.: A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics 9(1), 62–66 (1979)
  * [21] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: PyTorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32, pp. 8024–8035 (2019)
  * [22] Poulenard, A., Skraba, P., Ovsjanikov, M.: Topological function optimization for continuous shape matching. Computer Graphics Forum 37(5), 13–25 (2018)
  * [23] Rieck, B., Leitte, H.: Exploring and comparing clusterings of multivariate data sets using persistent homology. Computer Graphics Forum 35(3), 81–90 (2016)
  * [24] Rieck, B., Yates, T., Bock, C., Borgwardt, K., Wolf, G., Turk-Browne, N., Krishnaswamy, S.: Uncovering the topology of time-varying fMRI data using cubical persistence. In: Advances in Neural Information Processing Systems. vol. 33, pp. 6900–6912 (2020)
  * [25] Shit, S., Paetzold, J.C., Sekuboyina, A., Ezhov, I., Unger, A., Zhylka, A., Pluim, J.P.W., Bauer, U., Menze, B.H.: clDice \- A novel topology-preserving loss function for tubular structure segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16560–16569 (2021)
  * [26] Simionato, G., Hinkelmann, K., Chachanidze, R., Bianchi, P., Fermo, E., van Wijk, R., Leonetti, M., Wagner, C., Kaestner, L., Quint, S.: Red blood cell phenotyping from 3D confocal images using artificial neural networks. PLOS Computational Biology 17(5), 1–17 (2021)
  * [27] Skraba, P., Turner, K.: Wasserstein stability for persistence diagrams arXiv:2006.16824 (2020)
  * [28] Wagner, H., Chen, C., Vuçini, E.: Efficient computation of persistent homology for cubical data. In: Topological Methods in Data Analysis and Visualization II: Theory, Algorithms, and Applications, pp. 91–106. Springer (2012)
  * [29] Waibel, D.J.E., Kiermeyer, N., Atwell, S., Sadafi, A., Meier, M., Marr, C.: SHAPR - An AI approach to predict 3D cell shapes from 2D microscopic images bioRxiv:2021.09.29.462353 (2021)
  * [30] Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., Jiang, Y.G.: Pixel2mesh: Generating 3D mesh models from single RGB images. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 52–67 (2018)


1143 Theory or Model

Examples of MV layout designs based on factors of user perception (a-g) and
view content (h-j), which highlights broadening the design considerations from
purely perception-driven to intelligently content-driven.

# Toward Systematic Design Considerations of Organizing Multiple Views

Abdul Rahman Shaikh e-mail<EMAIL_ADDRESS>David Koop e-mail:
<EMAIL_ADDRESS>Hamed Alhoori e-mail<EMAIL_ADDRESS>Maoyuan Sun e-mail:
<EMAIL_ADDRESS>Northern Illinois University

###### Abstract

Multiple-view visualization (MV) has been used for visual analytics in various
fields (e.g., bioinformatics, cybersecurity, and intelligence analysis).
Because each view encodes data from a particular perspective, analysts often
use a set of views laid out in 2D space to link and synthesize information.
The difficulty of this process is impacted by the spatial organization of
these views. For instance, connecting information from views far from each
other can be more challenging than neighboring ones. However, most visual
analysis tools currently either fix the positions of the views or completely
delegate this organization of views to users (who must manually drag and move
views). This either limits user involvement in managing the layout of MV or is
overly flexible without much guidance. Then, a key design challenge in MV
layout is determining the factors in a spatial organization that impact
understanding. To address this, we review a set of MV-based systems and
identify considerations for MV layout rooted in two key concerns: perception,
which considers how users perceive view relationships, and content, which
considers the relationships in the data. We show how these allow us to study
and analyze the design of MV layout systematically.

###### keywords:

Multiple views, visual analytics, spatial layout.

Introduction

Multiple-view visualization (MV) has been commonly used in visual analysis
tools for sensemaking of data in various application domains, including
multimedia analysis (e.g., Canopy [9]), text analytics (e.g., IN-SPIRE [1] and
Jigsaw [37]), business intelligence (e.g., Spotfire [2] and Tableau [40]) and
cyber security [45]. To support sensemaking activities, users often work with
multiple views and spatially organize them, if allowed, on a display space
(e.g., placing a focused view in the center, moving a few related ones around,
and minimizing unused ones). Designing effective visualizations or MV systems
to explore a dataset effectively is challenging due to the restrictive
layouts, complicated coordination mechanisms, and interactions between
multiple dimensions of design spaces [43].

As prior research shows that humans can use space to think [5], the created
layout of MV can encode specific meanings that assist in the information
displayed in multiple views. However, current visual analysis tools either use
a fixed layout of MV [15, 19] or completely delegate organizing MV to users
[37, 27] (e.g., manually moving views). The former limits user involvement in
organizing MV, and the latter seems overly flexible as it does not provide any
scaffold to support users in manually organizing MV. While recent research
performed an in-depth study of the composition and configuration of MV [10],
their MV layout design recommendation considers little about the content in MV
and cross-view data relationships, which impacts the usage of MV [38].

This leads to an important design challenge for MV layout. Specifically, what
important factors should we consider for the design of MV layout? To address
this, we conducted a systematic review of 360 MV designs used in a prior study
[10]. Based on the review, we propose two major aspects to consider while
designing the layout of MV: perception and content. The former considers user
perception of MV. The latter regards data relations across MV.

We argue that a view’s content and the cross-view data relationship should be
a key consideration for designing the layout for an MV system. MV systems
should account for view content as layouts tailored to a prominent view can
help users be more productive [7]. Specifically, well-designed systems in this
respect can reduce the time spent exploring the dataset by effectively
conveying essential information to users by avoiding clutter in display space
[3]. An improved layout is also correlated to the productivity of and
interpretability by the user [28]. However, despite this key benefit, view
content or cross-view data relations are not extensively considered in
existing design techniques and tools for layouts of MV.

This study offers a novel approach to the design layout of MV systems based on
user perception and view content. We identify factors related to them to build
a practical design layout for MV systems. We further discuss design layouts in
detail based on these attributes and factors. Our work suggests layouts to
assist designers in planning an initial point in their layout design and
consider content as an essential part of their layout design process. Our
contributions in this paper are as follows:

  * •

We describe how user perception and view content are essential attributes to
build concrete design guidelines for MVs.

  * •

We present layouts that help designers consider perception and content when
designing MVs.

##  1 Related Work

###  1.1 Multiple View

The field of MV [25, 6, 32, 31] has been explored extensively in the past few
years. Researchers have focused on encouraging the use of MV [29], providing
guidelines for using MVs [43], investigating collaborative computational
analysis on MVs[16, 11], exploring multiple coordinated views [30, 34], and
creating MV tools or systems to investigate and gain insight into datasets.
Numerous MV systems have been proposed in recent years, including ComVis [23],
a tool that facilitates new visualization techniques and prototypes; SightBi
[39], a visual analytics system that explores cross-view data relationships
using biclusters and visualizes them to explore insights into coordinated
activities for sensemaking; JigSaw [37], a visual analytic system with MVs
containing documents along with entities that can be visually connected to
enable analysts to examine the documents quickly and investigate them
efficiently. These MV tools and systems have distinct layout designs that
provide analysis of data through different perceptions.

Prior research has analyzed the perception and cognitive performance of users
while using different tools [13, 14]. Andrews et al. [5] investigated how
increased space affects a user’s cognitive thinking and decision-making
processes while performing a sensemaking task. They found that factors like
design space, design layout, and analytics provenance alter a user’s cognitive
thinking, thus affecting their performance in a sensemaking process. North and
Shneiderman [24] proposed a taxonomy with multiple window coordination
capabilities through which users can create visual environments and
coordination links between windows. They found that this coordination between
multiple windows was associated with improved user performance, detection of
unexpected relationships, and desktop unification (i.e., showing complex
information on multiple displays with appropriate coordination).

###  1.2 Multiple View Layout

Researchers have surveyed the design layout of MV systems and identified a set
of different layouts proposed over recent years. Chen et al. [10] analyzed
designs from 360 multiple view systems with different views, layout designs,
and display spaces proposed in research articles. Similarly, Al-maneea and
Roberts [4] extracted 491 images of MV systems from several conferences,
journals, and workshops to explore different view layouts proposed across
multiple studies. However, little to no research has been published that
considers the view content or the cross-view data relationship in designing
layouts of MV systems.

Several MV systems proposed over the years contain less than five views,
simple visualizations, and non-complex layouts often with fixed views [10].
For example, BirdVis [15] is an MV system in which the layout is a side-by-
side display of fixed views. Hofman et al. [19] proposed a system with three
fixed views with a central salience view containing a spiral linked view,
controls on the left, and a streamgraph view on the right. The MV system
proposed by Bögl et al. [8] has four fixed views with two main (salient) views
on the top with controls and list views positioned at the bottom.

Apart from simple layouts, a few complex MV systems have been proposed over
the last few years, which often use flexible views. For example, Jigsaw [37]
can accommodate many flexible views in its display space where views can be
re-positioned or resized causing overlaps. Zhao et al. [47] proposed a system
with flexible resizing of views and a salient view at the center surrounded by
its related views. The SimilarityExplorer [27] system has flexible views
similar to Jigsaw but with overlaps that cause users to move views often to
analyze two or more focused views simultaneously. In summary, many proposed
systems have a fixed or overly flexible layout, as there is no systematic
design consideration in organizing MVs.

##  2 Design Considerations

We reviewed a dataset of 360 images of MVs by coding the attributes affecting
layout design. We found the typical design for MV layouts focuses on
attributes that affect user perception, often related to the view (e.g.,
position or size of the view) or viewport (display space) and the content of a
view (e.g., visualizations, control panels, and textual data). We propose that
content and perception are essential attributes to consider when designing
layouts for MV systems.

We suggest the layout design of an MV system considering the factors of the
two coded attributes: user perception and view content. A user’s perception of
the view, while working with MVs, affects their cognitive ability to perform
an analysis (e.g., a view in a large size or a centered-position can imply it
is more important than other views). User perception also affects user
attention, interpretation, insights, and performance, impacting the decision-
making process. Similarly, the content present in each view and the
relationship between such content also influence a user’s analytical process.
A view’s content is integral to an MV system since it effectively communicates
information through appropriate visualizations. The relationship between the
content of views provides insightful information and assists users in
decision-making. In cross-view data relationships, content in one view may be
updated in response to a user action in another view, or elements may be
linked between views. (e.g., two views with a cross-view data relationship may
convey knowledge more efficiently if they are appropriately positioned near
each other rather than separated).

However, current design processes overlook the cross-view data relationship
(relationship between the content of two or more views). Additionally, we
discuss how MV layouts can be improved by integrating cross-view data
relationships. Table 1 provides an overview of the driving factors of each
design layout suggested in this work.

Table 1: Proposed Layout Designs (refer to Figure Toward Systematic Design
Considerations of Organizing Multiple Views) and their layout factors
categorized based on perception (see Section 2.1) and content (see Section
2.2).

Factors |  (a) |  (b) |  (c) |  (d) |  (e) |  (f) |  (g) |  (h) |  (i) |  (j)
---|---|---|---|---|---|---|---|---|---|---
Separation |  |  ✕ |  |  |  |  |  |  |  |
Position |  ✕ |  |  |  ✕ |  ✕ |  |  ✕ |  |  ✕ |
Salience |  ✕ |  ✕ |  |  |  ✕ |  ✕ |  |  |  ✕ |
View content |  ✕ |  ✕ |  ✕ |  ✕ |  ✕ |  ✕ |  ✕ |  ✕ |  |
Cross-view |  ✕ |  ✕ |  ✕ |  ✕ |  ✕ |  ✕ |  ✕ |  |  ✕ |

###  2.1 Perception

Visual perception is how people process and organize the visual information
presented to them; improving a person’s perception allows them to efficiently
recognize patterns and gain understanding from data [44]. Perception aids
cognition and benefits from encoding information through visual channels like
color hue and length. In MV layouts, the visual properties of and between the
_views_ significantly contribute to how this information is understood. Thus,
we define a _perception-driven layout_ as a layout that organizes views to
improve visual perception and thus reduce cognitive load. This can be achieved
by differentiating different views, drawing user attention to important views,
and helping users navigate through views.

We have categorized a number of factors, including _separation_ , _relative
position_ , and _salience_ , which contribute to the perception of MVs. The
most important property of an MV system is _separation_. It allows the user to
distinguish between two or more views. This does not mean that views cannot
overlap, but if they do, users need to be able to clearly identify individual
views. However, in some MV systems, users may not always need to distinguish
between individual views (e.g., if two views are very similar and can provide
more information if perceived as one). The second factor influencing
perception-driven layouts is the _relative position_ of the views. A set of
views that are randomly laid out instead of aligned in a grid affects the way
relationships between views are understood. Similarly, two views located far
away from each other are often interpreted as unrelated, and a view in a
central position may be understood as a base for those that are eccentric. The
third factor we have identified is _salience_ which captures how particular
views draw more attention than others. For example, if most views use a
grayscale colormap, but one view uses vivid colors, that view is more salient.
Other properties influencing saliency include whether a view is minimized or
maximized or is larger or smaller in size. These factors intersect as well; a
central view that is also larger than others may be identified as the most
important view in the visualization.

###  2.2 Content

We define content-aware layouts as those being focused on each view’s content
(e.g., types of visualizations and control panels) and the relationship of
content between different views. In content-aware layouts, we focus on the
relationship between views to reveal hidden patterns or relationships between
content elements. We categorize the content-aware factor as the cross-view
data relationship between views. As the content of the views impacts an MV
system’s analysis process, we propose content-aware layouts that consider view
content and cross-view data relationships when designing layouts.

The cross-view data relationship focuses on exploring the connection between
the content of views, such as linking elements between views or coordinating
between views (content is updated in a view when a user interacts with other
views). Content-aware layouts can be useful in specific analyses, such as
investigating relationships, exploring patterns, filtering cross-views,
finding similar items, or linking distinct items across views. In a scenario
where a display space has many views, cross-view data relationships can help
with the layout by separating/focusing on relevant views for analysis.
Additionally, the layout can be adjusted to focus on the cross-view data
relationship to find similar or different items in different views, and then
position the views according to this relationship for a more effective
analysis. We suggest and describe four content-aware layouts based on the two
factors: view content and cross-view data relationship.

##  3 Layouts

We define six perception-driven and three content-aware layouts as provided in
Figure Toward Systematic Design Considerations of Organizing Multiple Views to
serve as a guiding point for designers of MV systems. Furthermore, we discuss
the characteristics of each layout as well as their usefulness.

###  3.1 Perception-Driven Layout

Standard Layouts - The standard layouts adapted widely by most MV systems are
flexible and fixed layouts. In the flexible layout (Figure Toward Systematic
Design Considerations of Organizing Multiple Views (a)) designers provide no
constraints on the views in the display space; there are no fixed positions,
fixed sizes, or predefined salient views. The user can easily drag, resize,
and position views anywhere in the display space, which means MVs can be
displayed in different sizes. Separate views can be positioned to overlap or
appear side by side in this layout. Therefore, the user has a high-level of
control over the views. MV systems, such as Jigsaw [37] and SimilarityExplorer
[27], implemented flexible layouts.

Conversely, in a fixed layout (Figure Toward Systematic Design Considerations
of Organizing Multiple Views (b)), designers assign a fixed position and size
to all views in the display space. When a user opens or considers a view, it
is displayed at its associated position and cannot be moved to another
position or resized. Separate views do not overlap in this type of layout, and
views are usually positioned adjacently. Designers can fix a salient view in
an attractive location (such as the center of the display space), thereby
directing users’ attention to it. The user has minimal control of views with
the layout predefined by the designer. A fixed-layout has been designed and
used in MV systems, such as BirdVis [15] and SwiftTuna [21].

Cascade Layout - A cascade layout (Figure Toward Systematic Design
Considerations of Organizing Multiple Views (c)) allows a designer to organize
each view in a system into a stack of views known as a _cascaded stack_. The
views in the stack are arranged so the user can observe portions of content in
the view. The user can focus on one or more specific views by selecting them
from the cascaded stack, which maximizes the selected view. This layout is
similar to the cascade option on Windows 10; the layout allows for fixed or
dynamic sizes, adjustable positions, and multiple salient views. The layout
can accommodate multiple views within a display space with minimal clutter as
separate views can be piled on top and overlapped. An exampled MV system
implementing a cascade layout is [18].

Focused Layout - In the focused layout (Figure Toward Systematic Design
Considerations of Organizing Multiple Views (d)), designers provide a _focused
window_ at the center of the display space that allows users to focus or
explore one or several specific views. A border surrounds the focused window
to visualize views not present within it. Depending on the display space, the
views can be fixed or not in size, and their positions can be flexible.
Current MV systems adopting this layout usually include supplemental views or
controls on the side, often related to the salient view in the center. The
designers use this layout when the user should focus on a particular view;
some examples of MV systems are [36, 47, 8].

Split Layout - In the split layout (Figure Toward Systematic Design
Considerations of Organizing Multiple Views (e)), the display space is split
into one or more windows – a style used to organize several different views.
The user can split the display space into multiple separate windows, and views
can be moved between the windows. This layout is similar to the split-screen
option available in Mac OS and Windows 10. The views from the split windows
can have different layouts, allowing for flexible or fixed sizes, positions,
and multiple salient views. The split layout has been used by many MV systems
[42, 26].

Stacked Layout - In the stacked layout (Figure Toward Systematic Design
Considerations of Organizing Multiple Views (f)), designers can arrange views
on top of one another as they are sequentially displayed in the viewing space.
The position of the views in this layout is flexible, although the size of the
views is fixed with no specific salient view. This layout can be designed
according to a row or column stacked structure in which the views are arranged
in a row or column format, respectively. It allows flexibility to arrange and
remove views as required. The user can prioritize the views, analyze them
sequentially and perform step-by-step analysis efficiently. The stack data
structure inspires this layout. Examples of MV systems that are designed
similar to this layout are [17, 22].

Tab Layout - The tab layout (Figure Toward Systematic Design Considerations of
Organizing Multiple Views (g)) includes views that are entirely minimized in
the form of a tab such that only the name of the view is visible. This layout
has flexible positions, flexible sizes, and multiple salient views. When a
user hovers over a tab, the view is shown. This allows the user to either
maximize the view for a more focused view or remove it from the display space.
Maximized or focused views can be minimized to form tabs. This layout is
influenced by the tab option available on most operating systems. Some MV
systems designed based on this layout are [41, 35]

###  3.2 Content-Aware

Hierarchical Layout - Designers can implement the hierarchical layout (Figure
Toward Systematic Design Considerations of Organizing Multiple Views (h)) to
create a tree-like structure between views with a root view and its leaf
views. The design of the layout enables users to keep linked views together
and focus on several connections simultaneously. Views can be added to the
display space multiple times, allowing them to become either the root view or
the leaf view of a tree structure. The linking of items across different views
and the relation of content present in views through these links is an
essential consideration in this layout. A user can keep track of the views
explored in the analytical process, which can help a user retrace their steps.
The layout is based on the tree data structure and can be helpful in tasks
that require linking of views, hierarchy-based tasks, or step-based tasks. An
example of an MV system designed based on this layout is GraphTrail[12].

Composite Layout - In the composite layout (Figure Toward Systematic Design
Considerations of Organizing Multiple Views (i)), designers can provide a
system that allows users to utilize as much space as possible by presenting
content in vacant areas or by integrating views. The user can incorporate
content into vacant spaces caused by any overlap between views. The user can
also choose particular views and combine them to create a combination of the
selected views. The layout is focused on using most of the space available to
provide essential content in the same display space. Designers may also choose
to leave space between views to depict related views. Similar to the OnSet
technique [33], this layout is a result of considering common patterns among
views. Designers can use this layout to display MVs in small display spaces
and assist in identifying commonalities among views. The various strategies or
patterns that designers can consider to implement this layout are given in
[20].

Categorized Layout - Designers can utilize the categorized layout (Figure
Toward Systematic Design Considerations of Organizing Multiple Views (j)) to
group similar or related views in the MV system. Similarities between the
views need to be considered for producing this layout. The similarity between
views can be observed by similar visualization in views or through the
presentation of different perspectives of the same elements. The content of
the views are categorized, and similar views are displayed together in a
separate window. This layout enables the user to focus on a specific category
of related views and can be used for finding similar or dissimilar elements.
[46] depicts an MV system utilizing this layout to categorize different
visualizations into separate areas of the display space.

##  4 Use Case

Figure 1: An example of an MV system with a poor layout (A) and a revised,
improved layout (B) using a systematic layout design.

MV systems use multiple views to improve understanding of data, but most
designers do not formally examine the relationships between the views (cross-
view relationships) in a layout. Neglecting the impacts on perception and
understanding can lead to issues as shown in Figure 1(A). The issues can be
noticed by considering the factors related to perception and content. The
island visualizations shown in (1) are not separated, leading to potential
confusion as to their relative geographic locations. Here, views are not
aligned, and the histogram view (5) blocks the visibility of some of the
content (2). Considering the _separation_ factor, the issue of overlap caused
by (4) and the confusion in (1) is noticeable. The _relative position_ of (5),
(3), and (6) result in overlaps and disjointed views. In addition, _cross-view
data relationship_ is overlooked. Views (1) and (6) use linked highlighting
(shown using the red dot), but they are not adjacent, meaning users may
overlook the connection. Finally, The histograms in (3) communicate the
distribution of attributes that are also shown in the binned scatterplot in
(2), but again they are not connected in the layout.

These issues can be addressed by implementing the suggested layouts in the MV
system. Implementing the categorized layout fixes the overlap issue by
categorizing (4) and (5) in a separate area in the display space. It also
places views (1) and (6) side by side so that the user can easily identify the
connection between the views; using the split layout, there is a separation in
(1) to distinguish each visualization in the view. In addition, we combine
views (2) and (3) into a joint plot to better communicate the relationship
between attributes by utilizing the composite layout. The layout of the MV
system is now improved (see Figure 1 (B)), conveying the information
effectively in a small display space. This shows the transition of an MV
system from a poor layout to an improved layout, considering the factors and
layouts proposed in this work. Additionally, it demonstrates that a designer
can integrate designs from one or more layouts when designing an MV system.

##  5 Discussion and Future Work

Current MV systems are designed in a way that usually requires manual or semi-
directed management of view layout. By considering the content of views and
their cross-view data relationships, it is possible to improve guidance for
the design of MV layout. The factors and suggested MV layouts can also
facilitate the transition from a user-directed layout to a steered, content-
aware layout management that incorporates layout as a type of information
synthesis.

This paper investigates the challenges of layout design for MV systems and
suggests factors to consider when designing MV systems. We do believe that
this framework is effective in addressing challenges in layout design.
However, our approach has some limitations; we do not claim to categorize
_all_ layouts or factors, nor do we rank them. Some design factors for future
study may include links between views, the number of related elements, and the
semantic relevance of views. Using a proposed layout or optimizing a layout
for a particular factor may not lead to an improved MV layout. We believe that
these findings can stimulate more research on the importance of layout design
in MVs, taking into account view content and cross-view data relationships.
Our future research will explore users’ real-time cognitive performance based
on their ability to analyze datasets using the design layouts described. We
intend to develop more content-driven design layouts to support complex MV
systems. To further validate the layouts and factors, we will implement
layouts similar to those shown in Figure 1 and study user performance on
analytical tasks.

###### Acknowledgements.

This research is supported in part by the NSF Grants IIS-2002082 and
SMA-2022443.

## References

  * [1] In-spire visual document analysis.  https://in-spire.pnnl.gov/.
  * [2] C. Ahlberg.  Spotfire: an information exploration environment.  ACM SIGMOD Record, 25(4):25–29, 1996.
  * [3] K. Ajani, E. Lee, C. Xiong, C. N. Knaflic, W. Kemper, and S. Franconeri.  Declutter and focus: Empirically evaluating design guidelines for effective data communication.  IEEE Transactions on Visualization and Computer Graphics, 2021.
  * [4] H. M. Al-maneea and J. C. Roberts.  Towards quantifying multiple view layouts in visualisation as seen from research publications.  In 2019 IEEE Visualization Conference (VIS), pp. 121–121, 2019\.
  * [5] C. Andrews, A. Endert, and C. North.  Space to think: Large high-resolution displays for sensemaking.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, p. 55–64, 2010.
  * [6] L. Bavoil, S. P. Callahan, P. J. Crossno, J. Freire, C. E. Scheidegger, C. T. Silva, and H. T. Vo.  Vistrails: Enabling interactive multiple-view visualizations.  In IEEE Visualization., pp. 135–142. IEEE, 2005.
  * [7] T. Boger, S. B. Most, and S. L. Franconeri.  Jurassic mark: Inattentional blindness for a datasaurus reveals that visualizations are explored, not seen.  In IEEE Visualization Conference, pp. 71–75. IEEE, 2021.
  * [8] M. Bögl, P. Filzmoser, T. Gschwandtner, T. Lammarsch, R. A. Leite, S. Miksch, and A. Rind.  Cycle plot revisited: Multivariate outlier detection using a distance-based abstraction.  In Computer Graphics Forum, vol. 36, pp. 227–238, 2017.
  * [9] R. Burtner, S. Bohn, and D. Payne.  Interactive visual comparison of multimedia data through type-specific views.  In Visualization and Data Analysis, vol. 8654, pp. 253–267. SPIE, 2013.
  * [10] X. Chen, W. Zeng, Y. Lin, H. M. Ai-Maneea, J. Roberts, and R. Chang.  Composition and configuration patterns in multiple-view visualizations.  IEEE Trans. on Vis. & Computer Graphics, 27(2):1514–1524, 2020\.
  * [11] H. Chung, C. North, J. Z. Self, S. Chu, and F. Quek.  Visporter: facilitating information sharing for collaborative sensemaking on multiple displays.  Personal and Ubiquitous Computing, 18(5):1169–1186, 2014.
  * [12] C. Dunne, N. Henry Riche, B. Lee, R. Metoyer, and G. Robertson.  Graphtrail: Analyzing large multivariate, heterogeneous networks while supporting exploration history.  In Proc. of the CHI Conference on Human Factors in Computing Systems, pp. 1663–1672, 2012.
  * [13] A. Endert, P. Fiaux, and C. North.  Semantic interaction for sensemaking: inferring analytical reasoning for model steering.  IEEE Transactions on Visualization and Computer Graphics, 18(12):2879–2888.
  * [14] A. Endert, M. S. Hossain, N. Ramakrishnan, C. North, P. Fiaux, and C. Andrews.  The human is the loop: new directions for visual analytics.  Journal of Intelligent Information Systems, 43(3):411–435, 2014\.
  * [15] N. Ferreira, L. Lins, D. Fink, S. Kelling, C. Wood, J. Freire, and C. Silva.  Birdvis: Visualizing and understanding bird populations.  IEEE Trans. on Vis. & Computer Graphics, 17(12):2374–2383, 2011\.
  * [16] C. Görg, Z. Liu, J. Kihm, J. Choo, H. Park, and J. Stasko.  Combining computational analyses and interactive visualization for document exploration and sensemaking in jigsaw.  IEEE Transactions on Visualization and Computer Graphics, 19(10):1646–1663, 2013.
  * [17] A. H. Hajizadeh, M. Tory, and R. Leung.  Supporting awareness through collaborative brushing and linking of tabular data.  IEEE Transactions on Visualization and Computer Graphics, 19(12):2189–2197, 2013.
  * [18] F. Heimerl, S. Koch, H. Bosch, and T. Ertl.  Visual classifier training for text document retrieval.  IEEE Transactions on Visualization and Computer Graphics, 18(12):2839–2848, 2012.
  * [19] J. Hofmann, M. Größler, M. Rubio-Sánchez, P.-P. Pichler, and D. J. Lehmann.  Visual exploration of global trade networks with time-dependent and weighted hierarchical edge bundles on gpu.  In Computer Graphics Forum, vol. 36, pp. 273–282, 2017.
  * [20] W. Javed and N. Elmqvist.  Exploring the design space of composite visualization.  In IEEE Pacific Vis. Symposium, pp. 1–8. IEEE, 2012.
  * [21] J. Jo, W. Kim, S. Yoo, B. Kim, and J. Seo.  Swifttuna: Responsive and incremental visual exploration of large-scale multidimensional data.  In IEEE Pacific Visualization Symposium, pp. 131–140. IEEE, 2017\.
  * [22] D. Maljovec, B. Wang, P. Rosen, A. Alfonsi, G. Pastore, C. Rabiti, and V. Pascucci.  Rethinking sensitivity analysis of nuclear simulations with topology.  In IEEE Pacific Vis. Symposium, pp. 64–71. IEEE, 2016.
  * [23] K. Matkovic, W. Freiler, D. Gracanin, and H. Hauser.  Comvis: A coordinated multiple views system for prototyping new visualization technology.  In International Conf. InfoVis., pp. 215–220. IEEE, 2008.
  * [24] C. North and B. Shneiderman.  A taxonomy of multiple window coordination.  Technical report, 1997.
  * [25] C. North and B. Shneiderman.  Component-based, user-constructed, multiple-view visualization.  In CHI’01 Extended Abstracts on Human Factors in Computing Systems, pp. 201–202, 2001.
  * [26] G. Palmas, M. Bachynskyi, A. Oulasvirta, H.-P. Seidel, and T. Weinkauf.  Movexp: A versatile visualization tool for human-computer interaction studies with 3d performance and biomechanical data.  IEEE Trans. on Vis. & Comp. Graph., 20(12):2359–2368, 2014.
  * [27] J. Poco, A. Dasgupta, Y. Wei, W. Hargrove, C. Schwalm, R. Cook, E. Bertini, and C. Silva.  Similarityexplorer: A visual inter-comparison tool for multifaceted climate data.  In Computer Graphics Forum, vol. 33, pp. 341–350, 2014.
  * [28] I. Reppa and S. Mcdougall.  When the going gets tough the beautiful get going: aesthetic appeal facilitates task performance.  Psychonomic Bulletin & Review, 22, 01 2015.
  * [29] J. C. Roberts.  On encouraging multiple views for visualization.  In Proc. of IEEE Conf. on Information Visualization, pp. 8–14. IEEE, 1998.
  * [30] J. C. Roberts.  State of the art: Coordinated & multiple views in exploratory visualization.  In Fifth International Conf. on Coordinated & Multiple Views in Exploratory Visualization, pp. 61–71. IEEE, 2007.
  * [31] J. C. Roberts, H. Al-maneea, P. W. Butcher, R. Lew, G. Rees, N. Sharma, and A. Frankenberg-Garcia.  Multiple views: different meanings and collocated words.  In Computer Graphics Forum, vol. 38, pp. 79–93, 2019.
  * [32] Y. S. Ryu, B. Yost, G. Convertino, J. Chen, and C. North.  Exploring cognitive strategies for integrating multiple-view visualizations.  In Proc. of the Human Factors and Ergonomics Society Annual Meeting, vol. 47, pp. 591–595, 2003.
  * [33] R. Sadana, T. Major, A. Dove, and J. Stasko.  Onset: A visualization technique for large-scale binary set data.  IEEE Transactions on Visualization and Computer Graphics, 20(12):1993–2002, 2014.
  * [34] M. Scherr.  Multiple and coordinated views in information visualization.  Trends in Information Visualization, 38:1–33, 2008.
  * [35] M. Sedlmair, A. Frank, T. Munzner, and A. Butz.  Relex: Visualization for actively changing overlay network specifications.  IEEE Trans. on Visualization and Computer Graphics, 18(12):2729–2738, 2012.
  * [36] P. M. Simon and C. Turkay.  Hunting high and low: Visualising shifting correlations in financial markets.  In Computer Graphics Forum, vol. 37.
  * [37] J. Stasko, C. Görg, and Z. Liu.  Jigsaw: supporting investigative analysis through interactive visualization.  Information Vis., 7(2):118–132, 2008.
  * [38] M. Sun, A. Namburi, D. Koop, J. Zhao, T. Li, and H. Chung.  Towards systematic design considerations for visualizing cross-view data relationships.  IEEE Trans. on Vis. and Computer Graphics, 2021.
  * [39] M. Sun, A. R. Shaikh, H. Alhoori, and J. Zhao.  Sightbi: Exploring cross-view data relationships with biclusters.  IEEE Transactions on Visualization and Computer Graphics, 28(1):54–64, 2021.
  * [40] Tableau.  https://www.tableau.com/.
  * [41] A. Unger, S. Schulte, V. Klemann, and D. Dransch.  A visual analysis concept for the validation of geoscientific simulation models.  IEEE Trans. on Vis. and Computer Graphics, 18(12):2216–2225, 2012\.
  * [42] S. Van den Elzen and J. J. Van Wijk.  Multivariate network exploration and presentation: From detail to overview via selections & aggregations.  IEEE Trans. on Vis. & Computer Graphics, 20(12):2310–2319, 2014\.
  * [43] M. Q. Wang Baldonado, A. Woodruff, and A. Kuchinsky.  Guidelines for using multiple views in information visualization.  In Proc. of the Working Conference on Advanced visual Interfaces, pp. 110–119, 2000.
  * [44] C. Ware.  Information visualization: perception for design.  Morgan Kaufmann, 2019.
  * [45] H. Zhang, M. Sun, D. Yao, and C. North.  Visualizing traffic causality for analyzing network anomalies.  In Proceedings of the ACM International Workshop on Security and Privacy Analytics, pp. 37–42, 2015.
  * [46] J. Zhang, E. Yanli, J. Ma, Y. Zhao, B. Xu, L. Sun, J. Chen, and X. Yuan.  Visual analysis of public utility service problems in a metropolis.  IEEE Trans. on Vis. and Computer Graphics, 20(12):1843–1852, 2014\.
  * [47] J. Zhao, F. Chevalier, C. Collins, and R. Balakrishnan.  Facilitating discourse analysis with interactive visualization.  IEEE Transactions on Visualization and Computer Graphics, 18(12):2639–2648, 2012.