(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext:  Fondazione Human Technopole, Viale Rita Levi-Montalcini 1,
20157 Milan, Italy 11email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# denoiSplit: a method for joint image splitting
and unsupervised denoising

Ashesh 11 0000-0003-3778-0576 Florian Jug 11 0000-0002-8499-5812

###### Abstract

In this work we present $\text{denoi}\mathbb{S}\text{plit}$, a method to
tackle a new analysis task, _i.e_. the challenge of joint semantic image
splitting and unsupervised denoising. This dual approach has important
applications in fluorescence microscopy, where semantic image splitting has
important applications but noise does generally hinder the downstream analysis
of image content. Image splitting involves dissecting an image into its
distinguishable semantic structures. We show that the current state-of-the-art
method for this task struggles in the presence of image noise, inadvertently
also distributing the noise across the predicted outputs. The method we
present here can deal with image noise by integrating an unsupervised
denoising sub-task. This integration results in improved semantic image
unmixing, even in the presence of notable and realistic levels of imaging
noise. A key innovation in $\text{denoi}\mathbb{S}\text{plit}$ is the use of
specifically formulated noise models and the suitable adjustment of KL-
divergence loss for the high-dimensional hierarchical latent space we are
training. We showcase the performance of $\text{denoi}\mathbb{S}\text{plit}$
across $4$ tasks on real-world microscopy images. Additionally, we perform
qualitative and quantitative evaluations and compare results to existing
benchmarks, demonstrating the effectiveness of using
$\text{denoi}\mathbb{S}\text{plit}$: a single Variational Splitting Encoder-
Decoder (VSE) Network using two suitable noise models to jointly perform
semantic splitting and denoising.

##  1 Introduction

Fluorescence microscopy remains a cornerstone in the exploration of cellular
and sub-cellular structures, enabling scientists to visualize biological
processes at a remarkable level of detail [9, 23]. The ability to distinguish
and analyze multiple structures within a single sample, though, requires a
multiplexed imaging protocol that requires extra time and effort [23]. To
address these downsides, and enable more efficient and new kinds of
investigations, a powerful method for semantic image splitting was recently
introduced [1].

Figure 1: Teaser Figure. In this work we use a variational encoder-decoder
network to jointly solve an usupervised denoising and image splitting task and
show that our approach outperforms sensibly chosen baselines.

Building upon this previous work [1], we address a key challenge that
persisted: noise in microscopy images and its adverse effect on the quality of
image splitting predictions. Recognizing the need for a method capable of
handling noisy input images while maintaining the integrity of the semantic
splitting task, we introduce a technique that not only builds on the strengths
of $\mu\text{Split}$ [1] but also incorporates unsupervised denoising
capabilities, for example as in [16, 22, 21, 20]. Figure 1 outlines the
overall approach we are proposing.

Combined, these ingredients lead to a new method,
$\text{denoi}\mathbb{S}\text{plit}$. It refines the process of image
decomposition, ensuring that even under high levels of pixel noises present in
the entire body of available training data, the semantic integrity of the
semantically split image components (the predictions) is well preserved.
Additionally, $\text{denoi}\mathbb{S}\text{plit}$ can assess data uncertainty
by sampling from the learned posterior of possible splitting solutions,
followed by evaluating the inter-sample variability. Additionally, in Section
4, we show how to use this possibility to predict the expected error
$\text{denoi}\mathbb{S}\text{plit}$ makes on a given input. In supplementary
Sec S3, we pen down a concrete pipleline which microscopists can follow to
assess the possibility of using our work without making any change to their
microscope configuration.

In summary, we believe that this work will open new avenues for the efficient
and detailed analysis of complex biological samples, for example in the
context of fluorescent microscopy.

\begin{overpic}[width=346.89731pt]{imgs/full_frame_biosr.png}
\put(51.0,100.0){\scriptsize High SNR}
\put(38.3,100.0){\scriptsize\mbox{$\text{denoi}\mathbb{S}\text{plit}$} }
\put(29.0,100.0){\scriptsize\mbox{$\mu\text{Split}$} }
\put(17.3,100.0){\scriptsize GT} \put(5.0,100.0){\scriptsize Input}
\put(-1.6,97.7){\scriptsize A} \put(-1.6,72.4){\scriptsize B}
\put(-1.6,47.1){\scriptsize C} \put(-1.6,21.8){\scriptsize D}
\par\put(0.1,84.3){\scriptsize$c_{0}$} \put(6.3,81.6){\scriptsize$c^{N}_{0}$}
\put(0.0,78.5){\scriptsize$c_{1}$} \put(6.3,75.3){\scriptsize$c^{N}_{1}$}
\par\put(0.1,58.9){\scriptsize$c_{0}$} \put(6.3,56.0){\scriptsize$c^{N}_{0}$}
\put(0.0,52.9){\scriptsize$c_{1}$} \put(6.3,49.7){\scriptsize$c^{N}_{1}$}
\par\put(0.1,33.4){\scriptsize$c_{0}$} \put(6.3,30.8){\scriptsize$c^{N}_{0}$}
\put(0.0,27.3){\scriptsize$c_{1}$} \put(6.3,24.6){\scriptsize$c^{N}_{1}$}
\par\put(0.1,8.7){\scriptsize$c_{0}$} \put(6.3,6.0){\scriptsize$c^{N}_{0}$}
\put(0.0,2.5){\scriptsize$c_{1}$} \put(6.3,-0.2){\scriptsize$c^{N}_{1}$}
\par\end{overpic} Figure 2: Qualitative Results. We show examples of noisy
inputs, individual noisy channel training data (GT), and predictions by one of
the baselines ($\mu\text{Split}$) and our own results obtained with
$\text{denoi}\mathbb{S}\text{plit}$ for all four tasks (A: MT vs. CCPs, B: ER
vs. CCPs, C: MT vs. ER, and D: F-actin vs. ER). Additionally we show high SNR
channel images (not used during training) and show PSRN values w.r.t. these
images. Additionally, we plot histograms of various panels for comparison (see
legend on the right). The bottom cell in the first column of each panel shows
the used noise models (see main text for details). The superimposed plots
(green) show the distribution of noisy observations ($c_{i}^{N}$) for two
clean signal intensities.

##  2 Related Work

###  2.1 Image Denoising

Image denoising is a task that has a long and exciting history. Classical
methods, such as Non-Local Means [5] or BM3D [6], were frequently and very
successfully used before neural network based approaches have been introduced
towards the end of the last decade [26, 28, 27, 15].

The advent of deep learning saw people exploit different aspects of noise and
the way networks learn to enable denoising. Noise, while usually being
undesired, is simultaneously much harder to predict, as was elegantly
demonstrated in [25], leading to a zero-shot denoiser. In the case specific to
pixel noises, _i.e_. all forms of noise that are independent per image pixel
(given the signal at that pixel) [23], the impossibility to predict the noise
was exploited in various ways, leading to important contributions such as
Noise2Void [15], Noise2Self [3], or Self2Self [14]. Another well known
approach close to this family of approaches is Noise2Noise [17], capable of
denoising even more complex noises that can correlate beyond the confines of
single pixels.

Slightly later, to further improve denoising performance, Probabilistic
Noise2Void [16, 22] introduced, and DivNoising [21] and Hierarchical
DivNoising (HDN) [20] reused the idea of suitably measured or trained pixel
noise models. Such noise model are, in essence, a collection of probability
distributions mapping from a true pixel intensity to observed noisy pixel
measurements (and vice-versa).

###  2.2 Image Decomposition

Image decomposition is the inverse problem of splitting a given input image
that is the superposition (_i.e_. the pixel-wise sum) of two constituent image
channels. While the sum of two values is not uniquely invertible, if for each
summand a prior on its value exists, a unique solution can exist. In a similar
vain, having learned structural priors of the appearance of the two
constituent image channels, an input image (a grid of observed pixels that are
each a sum of two values) can be split into two pixel grids such that each one
satisfies the respective structural prior. In computer vision, reflection
removal, dehazing, deraining _etc_. are some of the applications [8, 7, 2, 4]
for which image splitting can be used.

More recently, image splitting in fluorescence microscopy was experiencing
heightened attention, likely due to the direct applicability and potential
utility a well working approach can bring to this microscopy modality that
finds wide-spread use in biological investigations. In particular, a method
called $\mu\text{Split}$ [1] demonstrated impressive image splitting
performance on several datasets, suggesting that it is ready to be used in
biological research projects.

However, $\mu\text{Split}$’s requires relatively noise-free data for training
and prediction, which limits its potential utility (see also Section 5 or
Figure 2).

###  2.3 Uncertainty Calibration

The ability to correctly assess the quality of predictions is of obvious
utility. Ideally, a predictive system capable of co-predicting a confidence
value, has the property that the predicted confidence scales with the average
error of the prediction. If the relationship between error and confidence is
close to the identity, we call the uncertainty predictions of this system
calibrated.

Early works tried to use the deviation of the prediction as a proxy for the
confidence in the made predictions [27]. Other works tried to calibrate the
predicted standard deviation with the expected error (_i.e_. the RMSE) [18].
Earlier calibration works were mostly concerned with classification tasks
[19]. However, in [18], those approaches were reformulated in the context of
regression.

In [18], the authors propose a way to evaluate calibration. They train a
separate branch to predict a standard deviation per pixel that expresses its
prediction uncertainty. For evaluating the calibration quality, the authors
cluster examples on the basis of the predicted standard deviation values.
Within each cluster, the predicted uncertainty is then compared with the
empirical uncertainty (RMSE loss). To further improve the calibration, the
authors propose a simple, yet effective scaling methodology wherein they learn
a scalar parameter on re-calibration data, _i.e_. a subset of data not
included in the training data. (In our experiments, we use the validation data
for this purpose.) This scalar gets multiplied to the predicted uncertainty
values, which then reduces the calibration error.

##  3 Problem Formulation

Lets denote a noise free dataset containing $n$ pairs of images as
$D=(C_{1},C_{2})$, with each $C_{i}$ containing $n$ two-channel images
$C_{i}=(c_{i,j}|1\leq j\leq n)$. Lets define a corresponding set of images
$X=(x_{j}|1\leq j\leq n)$, such that all $x_{j}=c_{1,j}+c_{2,j}$ are the
pixel-wise sum of the two corresponding channel images.

While $D$ is typically not available (or even observable), in practice we can
only observe noisy data, denoted here by $D^{N}=(C_{1}^{N},C_{2}^{N})$.
Analogous to before, we define $X^{N}=(x_{j}^{N}|1\leq j\leq n)$, such that
$x_{j}^{N}=c_{1,j}^{N}+c_{2,j}^{N}$ are the pixel-wise sum of the noisy
channel observations.

Given one $x_{j}^{N}\in X^{N}$ of $D^{N}$, the task at hand is to predict the
noise free and unmixed tuple $(c_{1,j},c_{2,j})$. We shall denote the
predictions made by a trained $\text{denoi}\mathbb{S}\text{plit}$ network by
$(\hat{c}_{1,j},\hat{c}_{2,j})$.

Whenever above notions are used in a context that makes the $j$ in the
subscript redundant, we allow ourselves to omit them for brevity and
readability.

For evaluation purposes, we will in later sections use high-quality microscopy
datasets that contain minimal levels of noise as surrogates for $D$, $X$,
$C_{1}$, and $C_{2}$, but we never use them during training, and only their
noisy counterparts are used.

##  4 Our Approach

In the following sections we describe the main ingredients of
$\text{denoi}\mathbb{S}\text{plit}$, namely the hierarchical network structure
we use (Section 4.1), the changed loss term for variational training of the
splitting task (Section 4.2), the noise models we employ to enable the joint
unsupervised denoising (Section 4.3), and an uncertainty calibration
methodology allowing us to estimate the prediction error introduced by
aleatoric uncertainty in a given input image (Section 4.4).

###  4.1 Network Architecture and Training Objective

In this work, we employ an altered Hierarchical VAE (HVAE) network
architecture. HVAEs were originally described in [24] and later adapted for
image denoising in [20] and for image splitting in [1]. In general terms,
HVAEs learn a hierarchical latent space, with the lowest hierarchy level
encoding detailed pixel-level structure, while higher hierarchy levels capture
increasingly larger scale structures in the training data.

For $\text{denoi}\mathbb{S}\text{plit}$, we modify the HVAE architecture to
not any longer being used as an autoencoder. Instead, our outputs are the two
unmixed channel images $(\hat{c}_{1},\hat{c}_{2})$, motivating us to call the
resulting architecture a Variational Splitting Encoder-Decoder (VSE) Network.

Our objective is to maximize the likelihood over the noisy two channel dataset
we train on, _i.e_., finding decoder parameters $\boldsymbol{\theta}$ such
that

$\boldsymbol{\theta}=\operatorname*{arg\,max}_{\boldsymbol{\theta}}\sum_{1\leq
j\leq n}\log P(c_{1,j}^{N},c_{2,j}^{N};\boldsymbol{\theta}).$ (1)

Using the modified evidence lower bound (ELBO), as proposed in [1] and
assuming conditional independence of the two predictions given the latent
space embedding, we maximize

$\>E_{q(z|x;\boldsymbol{\phi})}[\log P(c_{1}|z;\boldsymbol{\theta})+\log
P(c_{2}|z;\boldsymbol{\theta})]\\\ -KL(q(z|x;\boldsymbol{\phi}),P(z)),$ (2)

where $q(z|x;\boldsymbol{\phi})$ is the distribution parameterized by the
encoder network output $\text{Enc}_{\boldsymbol{\phi}}(x)$,
$P(c_{1}|z;\boldsymbol{\phi})$ is the distribution parameterized by decoder
network output $\text{Dec}_{\boldsymbol{\theta}}(z)$ and $KL()$ denotes the
Kullback-Leibler divergence loss. As in [1], $P(z)$ factorizes over the
different hierarchy levels in the network. Please refer to [1] for a more
thorough derivation. Details about the training, hyperparameters used, and
used VSE Networks can be found in the supplementary Sec S2.

Similar to the way noise models had been employed in the context of denoising
[20], we model the two log likelihood terms $\log
P(c_{1}|z;\boldsymbol{\theta})$ and $\log P(c_{2}|z;\boldsymbol{\theta})$
using noise models which we describe in detail in Section 4.3 and naturally in
our open code repository.

\begin{overpic}[width=433.62pt]{imgs/sampling_figure_423_143.png}
\put(87.0,28.3){\scriptsize High SNR} \put(74.5,28.3){\scriptsize MMSE}
\put(61.0,28.3){\scriptsize$S_{1}$ - $S_{2}$} \put(45.3,28.3){\scriptsize
Sample 2} \put(32.5,28.3){\scriptsize Sample 1} \put(20.0,28.3){\scriptsize
GT} \put(6.0,28.3){\scriptsize Input} \put(-1.2,26.5){\scriptsize A}
\end{overpic}

\begin{overpic}[width=433.62pt]{imgs/sampling_figure_423_144.png}
\put(-1.2,26.5){\scriptsize B} \end{overpic}

\begin{overpic}[width=433.62pt]{imgs/sampling_figure_716_145.png}
\put(-1.2,26.5){\scriptsize C} \end{overpic}

Figure 3: Variational Sampling and Calibration. The VSE Network in
$\text{denoi}\mathbb{S}\text{plit}$ is capable of sampling from a learned
posterior. Here we show cropped inputs ($256\times 256$), two corresponding
prediction samples, the difference between the two samples ($S_{1}-S_{2}$),
the MMSE prediction, and otherwise unused high SNR microscopy for three tasks,
namely ER vs. CCPs, ER vs. MT, and CCPs vs. MT. The MMSE predictions are
computed by averaging $50$ samples. As before, we show PSNR w.r.t. high SNR
patches. The dot plots in the first column show are calibration plots,
showcasing that the error estimate we propose works well (see main text).

###  4.2 Hierarchical KL Loss Weighing for Variational Training

In $\mu\text{Split}$, the authors showed SOTA performance on a multitude of
splitting tasks. However, used datasets were close to noise-free, making the
task at hand simpler then the one we outlined in Section 3. When
$\mu\text{Split}$ is trained on noisy datasets, the resulting channel
predictions are themselves noisy. After analyzing this matter, we concluded
that a modified KL loss can help to reduce the amount of noise being
reconstructed by the decoder.

In more technical terms, let $Z$ be a hierarchical latent space and $Z[i]$
denote the latent space embedding at $i$-th hierarchy level, having shape
$(c,h_{i},w_{i})$, with $c$ being channel dimension, and $h_{i}$, $w_{i}$ the
height and width of the latent space embedding. Now let $\text{KL}_{i}$ denote
the KL-divergence loss tensor computed on $Z[i]$, which has the same shape as
$Z[i]$ itself.

In $\mu\text{Split}$, the corresponding scalar loss term $kl_{i}$ is defined
as

$\text{kl}_{i}=\alpha\cdot\sum_{j,h,w}\frac{\text{KL}_{i}[j,h,w]}{h_{i}\cdot
w_{i}},$ (3)

with $\alpha$ being a suitable constant. Observe that the denominator makes
each $\text{kl}_{i}$ be the average of all values in $\text{KL}_{i}$, making
the respective values not scale with the size of $Z[i]$, even though lower
hierarchy levels ($Z[i]$ for smaller $i$) have more entries. However, this
also means that the KL loss for the individual pixels in these lower hierarchy
levels is given less weight. Hence, smaller structures, such as noise itself,
can more easily seep through such pixel-near hierarchy levels.

In this work, we diverge from this formulation and return to a more classical
setup where we compute the scalar loss term for the $i$-th hierarchy level
$Z[i]$ as

$\text{kl}_{i}=\alpha\cdot\sum_{j,h,w}\text{KL}_{i}[j,h,w].$ (4)

The decisive difference is that this changed formulation gives more weight to
the KL loss at lower hierarchy levels, leading to more strongly enforcing the
unit variance Gaussians nature the KL loss enforces, and therefore hindering
noise from being as easily represented during training. We refer to this
architecture as Altered $\mu\text{Split}$ and show qualitative and
quantitative results in Section 5 and Tables 1.

The next section extends on Altered $\mu\text{Split}$ by adding unsupervised
denoising, adding the last ingredient to the
$\text{denoi}\mathbb{S}\text{plit}$ approach we present in this work.

###  4.3 Adding Suitable Pixel Noise Models

As briefly introduced in Section 2, pixel noise models are a collection of
probability distributions mapping from a true pixel intensity to observed
noisy pixel measurements (and vice-versa) [16]. They have previously been
successfully used in the context of unsupervised denoising [21, 20] and we
intend to employ them for this purpose also in the setup we are presenting
here. We use the fact that, given a measured (noisy) pixel intensity, a pixel
noise model returns a distribution over clean signal intensities and their
respective probability of being the underlying true pixel value.

We incorporate this likelihood function into the loss of our overall setup,
encouraging $\text{denoi}\mathbb{S}\text{plit}$ to predict pixel intensities
that maximize this likelihood and thereby values that are consistent with the
noise properties of the given training data.

Since $\text{denoi}\mathbb{S}\text{plit}$, in contrast to existing denoising
applications, predicts two images (the two unmixed channels), we employ two
noise models and add two likelihood terms to our overall loss.

More formally, in VAEs [13] and HVAEs, the generative distribution over pixel
intensities is modelled as a Gaussian distribution with its variance either
clamped to $1$ or also learned and predicted. We change our VSE Network to
only predict the true pixel intensity and replace the above mentioned Gaussian
distribution by distributions defined in two noise models
$\mathit{P}^{\text{nm}}_{1}(c^{N}_{1}|c_{1})$ and
$\mathit{P}^{\text{nm}}_{2}(c^{N}_{2}|c_{2})$, one for each respective unmixed
output channel. These noise models are pixel-wise independent, _i.e_.,

$\mathit{P_{i}}^{\text{nm}}(c^{N}_{i}|c_{i})=\prod_{k}\mathit{P}^{\text{nm}}_{i}(c^{N}_{i}[k]|c_{i}[k]),\text{
}i\in\\{1,2\\},$ (5)

where $c^{N}_{i}[k]$ is the noisy pixel intensity for the $k$-th pixel and
$c_{i}[k]$ the corresponding noise-free intensity value. This independence
makes them particularly suitable for microscopy data where Poisson and
Gaussian noise are the predominant pixel-noises one desires to remove.

Since we now directly predict the noise-free pixel values, the output of the
decoder can directly be interpreted as
$\text{Dec}_{\boldsymbol{\theta}}(z)=(c_{1},c_{2})$ and the total loss for
$\text{denoi}\mathbb{S}\text{plit}$ now becomes

$\>E_{q(z|x;\boldsymbol{\phi})}[\log P^{\text{nm}}(c^{N}_{1}|c_{1})+\log
P^{\text{nm}}(c^{N}_{2}|c_{2})]\\\ -KL(q(z|x;\boldsymbol{\phi}),P(z)).$ (6)

In [21], two ways for creation of noise models are described and the decision
to pick which method depends upon whether or not one has access to the
microscope from which data was acquired. In the supplementary Sec S4, we
describe the process of noise model generation and do a quantitative
comparison between these two methodologies.

\begin{overpic}[width=433.62pt]{imgs/comparison_with_hdn.png}
\put(70.5,100.0){\scriptsize High SNR}
\put(53.7,100.0){\scriptsize\mbox{$\text{denoi}\mathbb{S}\text{plit}$} }
\put(36.3,100.0){\scriptsize\mbox{{HDN$\oplus$\mbox{$\mu\text{Split}$}}} }
\put(24.0,100.0){\scriptsize GT} \put(6.0,100.0){\scriptsize Input}
\put(-1.2,97.5){\scriptsize A} \put(-1.2,64.0){\scriptsize B}
\put(-1.2,30.7){\scriptsize C} \end{overpic} Figure 4: Comparison to
Sequential Baseline. For each panel (ER vs. CCPs, CCPs vs. MT, and ER vs. MT)
we show the full mixed input image, a ($256\times 256$) inset, corresponding
noisy training data crops, the results of the sequential denoising and
splitting baseline (HDN$\oplus$$\mu\text{Split}$) and our end-to-end results
obtained with $\text{denoi}\mathbb{S}\text{plit}$. All predictions show the
MMSE, obtained by averaging $50$ sampled predictions. White arrows indicate
locations where the baseline under-performs. Note that such small locations
might contribute little to evaluations via PSNR, but makes a huge difference
for the downstream analysis of investigated biological structures contained in
such microscopy data.

###  4.4 Computing Calibrated Data Uncertainties

The idea of calibration is for those network setups which produce both
prediction and a measure of uncertainty for the prediction.

Networks that can co-assess the uncertainty of their predictions are called
calibrated, when the predicted uncertainties are in-line with the measured
prediction error. To improve the calibration of a given system, one can find a
suitable transformation from uncertainty predictions to measured errors
(_e.g_., the RMSE). After such a transformation is found, an ideal calibrated
plot would be tightly fitting $y=x$, with y and x being the error and
estimated uncertainty, respectively. See for example Figure 3. Since VSE
networks, similar to VAEs, are variational inference systems, we can sample
the latent encoding and thereby sample from an approximate posterior
distribution of possible solutions given the data uncertainty (aleatoric
uncertainty). In this section, our intention is to utilize this ability to
predict a reliable uncertainty term for our results.

For this, we adapt the calibration methodology of [18]. In contrast to the
approach described there, we propose to use the variability in posterior
samples to estimate a pixel-wise standard deviation. More specifically, we
sample $k=50$ predictions for each input image and compute the pixel-wise
standard deviations $\sigma_{1}$ and $\sigma_{2}$ for the two predicted image
channels $\hat{c}_{1}$ and $\hat{c}_{2}$, respectively. This gives us
uncertainty predictions.

Next, we calibrate these uncertainty predictions by scaling them appropriately
with the help of two learnable scalars, $\alpha_{1}$ and $\alpha_{2}$.
Following [18], we assume that pixel intensities come from a Gaussian
distribution. The mean and the standard deviation of this distribution are the
pixel intensities of MMSE prediction, _i.e_. the image obtained after
averaging $k=50$ predictions, and the scaled $\sigma$, respectively. We learn
the scalars $\alpha_{1}$ and $\alpha_{2}$ by minimizing the negative log
likelihood over the re-calibration dataset. It is important to note that the
presented calibration procedure does not alter the original predictions, but
instead learns a mapping that best predicts the measured error.

To evaluate the quality of the resulting calibration, we sort the scaled
standard deviations $\sigma_{i}\cdot s_{i}$ for each pixel in a predicted
channel, and build a histogram over $l=30$ equally sized bins $B_{i}^{j}$. We
then compute the root mean variance (RMV) for each bin $j$ and channel $i$ as

$\text{RMV}_{i}(j)=\sqrt{\frac{1}{|B^{j}_{i}|}\sum_{k\in
B^{j}_{i}}(\sigma^{k}_{i}\cdot\alpha_{i})^{2}}.$ (7)

For each bin $B_{j}$ and each channel $i$, we compute the RMSE as

$\text{RMSE}_{i}(j)=\sqrt{\frac{1}{|B^{j}_{i}|}\sum_{k\in
B^{j}_{i}}(c_{i}[k]-\hat{c}_{i}[k])^{2}}.$ (8)

As in Section 3, $c_{i}[t]$ and $\hat{c}_{i}[t]$ denote the noise free pixel
intensity and the corresponding prediction for $i$-th channel. In Sec. 5, we
plot the RMSE vs. RMV for multiple tasks, observing that the plots closely
resemble the identity $y=x$. Following[18], we don’t use the training dataset
for re-calibration. We use the validation dataset for re-calibration and we
show the calibration plots on test dataset.

In supplementary Sec S6, we inspect the relevance of using sampling for
estimating unconstrained uncertainty.

##  5 Experiments and Results

###  5.1 The BioSR dataset

The BioSR dataset [11] is a comprehensive dataset comprising of fluorescence
microscopy images of multiple labeled cell structures. For the purpose of our
experiments, we have picked four structures, namely clathrin-coated pits
(CCPs), microtubules (MTs), endoplasmic reticulum (ER) and F-actin.

Since the raw data quality is very high and only little image noise is present
in the individual micrographs, we add Gaussian noise and Poisson noise of
various levels to this raw data. The artificially noisy images are used to
train $\text{denoi}\mathbb{S}\text{plit}$, while the raw data is shown to
convince the reader of the validity of our approach and to compute evaluation
metrics (see Figs. 2 and 4 and Tab. 1).

In supplementary Sec S1, we evaluate our model on 5 more tasks and these tasks
have been created from BioSR, Hagen et al. [10] and PaviaATN [1].

###  5.2 Synthetic Noise Levels

We work with $4$ levels of zero-mean Gaussian noise and two levels of Poisson
noise. For Gaussian noise, we compute the standard deviation of the input data
$X^{N}$ for each of the tasks and scale the noise relative to one standard
deviation. Specifically, the $4$ scaling factors are $\\{1,1.5,2,4\\}$.

In cases where Poisson noise is added, and since it is already signal
dependent, we use use a constant factor of $1000$ to hit a realistic looking
level of Poisson noise. We also consider the case where Poisson noise is not
added which we denote by Poisson level of $0$ in Tab. 1.

To remove any remaining room for misinterpretations, we provide pseudo-code
for the synthetic noising procedure in the supplementary Sec S2.

###  5.3 Baselines

We conducted all experiments with two baseline setups, $\mu\text{Split}$ and
HDN$\oplus$$\mu\text{Split}$. In the original $\mu\text{Split}$ work [1],
authors introduce three architectures, but only one of them employs
variational inference and is therefore the most suitable to compare against.
More specifically, we choose the HVAE \+ Regular-LC setup as described in
their work. Here, we simply refer to this baseline as $\mu\text{Split}$. Note
that the $\mu\text{Split}$ baseline makes use of a KL-loss and is therefore
enabled to at least partially remove noise from a given noisy input image.

The second baseline, to which we refer to as HDN$\oplus$$\mu\text{Split}$, is
a sequential application of Hierarchical DivNoising (HDN), one of the leading
unsupervised denoising methods for microscopy datasets [20], and the
$\mu\text{Split}$ setup from above. Hence, here we first denoise all input
images $x_{i}^{N}$ and the respective two channel images
$c_{1,j}^{N},c_{2,j}^{N}$. Note that each set of the three kinds of image are
denoised with a separately and specifically trained HDN.

Next, we use the denoised predictions of $X^{N},C_{1}^{N},C_{2}^{N}$ to train
a $\mu\text{Split}$ network as we did for the first baseline. The expectation
from this baseline is to give denoised splitting results which we also show in
Fig. 4. We also show the denoised HDN predictions in the supplementary Sec S7.

Task | Model | training | Noise level parameters
---|---|---|---
$\lambda=0$ | $\lambda=1000$
$[h]$ | $\sigma=1$ | 1.5 | 2 | 4 | $\sigma=1$ | 1.5 | 2 | 4
T1 | $\mu\text{Split}$ | 7 | 30.3 | 28.4 | 27.4 | 25.9 | 29.4 | 28.1 | 27.3 | 25.9
0.853 | 0.748 | 0.66 | 0.42 | 0.844 | 0.750 | 0.667 | 0.437
HDN$\oplus$$\mu\text{Split}$ | 11 | 37.3 | 34.9 | 33.8 | 29.4 | 36.3 | 34.3 | 33.3 | 29.4
0.982 | 0.969 | 0.959 | 0.872 | 0.978 | 0.965 | 0.954 | 0.874
Altered $\mu\text{Split}$ (ours) | 1.3 | 38.9 | 36.7 | 34.4 | 30.9 | 36.9 | 35.5 | 34.8 | 31.1
0.988 | 0.980 | 0.965 | 0.909 | 0.982 | 0.974 | 0.968 | 0.912
$\text{denoi}\mathbb{S}\text{plit}$ (ours) | 1.5 | 39.7 | 36.8 | 35.4 | 31.1 | 37.9 | 36.3 | 35.0 | 31.2
0.989 | 0.978 | 0.969 | 0.912 | 0.984 | 0.977 | 0.967 | 0.912
T2 | $\mu\text{Split}$ | 6.3 | 26.0 | 23.9 | 22.7 | 21.0 | 25.2 | 23.7 | 22.7 | 21.1
0.800 | 0.699 | 0.593 | 0.356 | 0.780 | 0.691 | 0.613 | 0.386
HDN$\oplus$$\mu\text{Split}$ | 10 | 30.1 | 28.4 | 27.6 | 25.3 | 29.6 | 28.4 | 27.4 | 25.2
0.909 | 0.873 | 0.845 | 0.731 | 0.904 | 0.874 | 0.835 | 0.738
Altered $\mu\text{Split}$ (ours) | 1.5 | 30.4 | 28.8 | 26.9 | 23.4 | 29.9 | 27.4 | 27.9 | 24.4
0.915 | 0.879 | 0.809 | 0.620 | 0.903 | 0.833 | 0.845 | 0.677
$\text{denoi}\mathbb{S}\text{plit}$ (ours) | 1.6 | 30.5 | 29.2 | 28.2 | 25.1 | 30.0 | 29.0 | 27.0 | 24.8
0.916 | 0.886 | 0.860 | 0.714 | 0.909 | 0.885 | 0.815 | 0.702
T3 | $\mu\text{Split}$ | 7.2 | 30.5 | 28.3 | 27.3 | 25.6 | 29.6 | 28.1 | 27.2 | 25.6
0.880 | 0.793 | 0.713 | 0.46 | 0.877 | 0.800 | 0.722 | 0.476
HDN$\oplus$$\mu\text{Split}$ | 11 | 38.4 | 35.9 | 34.3 | 29.3 | 36.8 | 34.9 | 33.8 | 29.3
0.981 | 0.966 | 0.951 | 0.844 | 0.975 | 0.962 | 0.948 | 0.843
Altered $\mu\text{Split}$ (ours) | 1.4 | 38.9 | 35.8 | 35.0 | 30.4 | 37.4 | 35.6 | 34.3 | 30.4
0.985 | 0.968 | 0.960 | 0.867 | 0.979 | 0.968 | 0.953 | 0.865
$\text{denoi}\mathbb{S}\text{plit}$ (ours) | 1.6 | 40.1 | 37.3 | 35.7 | 30.6 | 38.1 | 36.6 | 35.2 | 30.7
0.986 | 0.973 | 0.962 | 0.872 | 0.981 | 0.971 | 0.958 | 0.872
T4 | $\mu\text{Split}$ | 7 | 25.9 | 24.3 | 23.6 | 22.4 | 25.2 | 24.2 | 23.5 | 22.4
0.777 | 0.664 | 0.556 | 0.331 | 0.729 | 0.640 | 0.554 | 0.321
HDN$\oplus$$\mu\text{Split}$ | 10.7 | 28.8 | 27.9 | 27.4 | 25.8 | 28.2 | 27.6 | 27.2 | 25.7
0.852 | 0.817 | 0.790 | 0.725 | 0.840 | 0.810 | 0.787 | 0.716
Altered $\mu\text{Split}$ (ours) | 1.3 | 29.4 | 28.5 | 27.5 | 25.9 | 29.0 | 27.8 | 27.3 | 25.8
0.858 | 0.824 | 0.786 | 0.718 | 0.849 | 0.794 | 0.780 | 0.710
$\text{denoi}\mathbb{S}\text{plit}$ (ours) | 1.5 | 29.6 | 28.7 | 27.6 | 26.0 | 29.0 | 28.5 | 27.9 | 26.1
0.868 | 0.835 | 0.787 | 0.725 | 0.854 | 0.828 | 0.799 | 0.727
Table 1: Quantitative Results. We show quantitative evaluations for all joint
denoising and splitting experiments. The four corresponding tasks are
abbreviated as T1: ER vs. CCPs; T2: ER vs. MT; T3: CCPs vs. MT, T4: F-actin
vs. ER. For all experiments, we show the PSNR (sub-row 1) and MS-SSIM (sub-row
2) metrics across $8$ noise levels: Gaussian noise levels of
$\sigma\in\\{1,1.5,2,4\\}$ and Poisson noise levels of
$\lambda\in\\{0,1000\\}$. The best performance per task and noise level is
shown in bold. The third column additionally shows the training time on a
single Tesla-V100 GPU (in hours). Not only does
$\text{denoi}\mathbb{S}\text{plit}$ perform best, it does at the same time
require considerably less training time.

###  5.4 Qualitative and Quantitative Evaluation of Results

We show the quality of results our methods can obtain in Figs. 2, 3 and 4.

In Fig. 2, we show predictions on entire input images ($960\times 960$
pixels). We can see that $\mu\text{Split}$ indeed unmixes the given inputs and
even partially reduces noise. Still, the results by
$\text{denoi}\mathbb{S}\text{plit}$ have a much higher resemblance to the high
SNR microscopy images shown in the rightmost column, even though they have
never been presented during training (which was conducted only on noisy images
as shown in the second column).

In Fig. 3, we show zoomed $256\times 256$ portions of full predictions to
allow readers to also appreciate the prediction quality of smaller structures
contained in the data. Additionally, we show two posterior samples ($S_{1}$
and $S_{2}$) as well as their highlighted differences ($S_{1}-S_{2}$). The
second to last column shows the average of $50$ posterior samples (the
approximate MMSE [21]).

In Fig. 4, we also show $256\times 256$ insets on inputs and results by
HDN$\oplus$$\mu\text{Split}$ and $\text{denoi}\mathbb{S}\text{plit}$, showing
that fine details are better preserved by our proposed method. Note that it is
these very details that make all the difference when such methods are used on
fluorescence microscopy data for the sake of scientific downstream analysis.

For quantitative quality evaluations, we use the well known and established
PSNR and Multiscale-SSIM metrics [23] to evaluate all results of our
experiments and report these results in Tab. 1.

These quantitative evaluations clearly show that our proposed methods improve
considerably over $\mu\text{Split}$, and as well over the sequential denoising
and splitting baseline HDN$\oplus$$\mu\text{Split}$. However, since joint
denoising-splitting is an ill-posed problem, it is not difficult to create
arbitrarily hard splitting tasks: one just needs to have the two constituent
channels contain less discriminatory information. In supplementary Sec S8, we
work with a more challenging task to show the limitations of our approach.

##  6 Discussion and Conclusion

Our work presents $\text{denoi}\mathbb{S}\text{plit}$, the first method that
takes on the challenge of joint semantic image splitting and unsupervised
denoising. This advancement is crucial, considering the limitations
microscopists face in acquiring high-SNR images, often due to practical
constraints such as sample sensitivity and limitations in imaging technology.
Unlike in a sequential approach of image denoising followed by training and
applying $\mu\text{Split}$ on denoised data,
$\text{denoi}\mathbb{S}\text{plit}$ streamlines the process into a single end-
to-end model, which is on one hand reducing the complexity and computational
resources required for training and inference, on the other hand leads to
better results.

Supporting this conclusion, we presented qualitative and quantitative results,
showing that $\text{denoi}\mathbb{S}\text{plit}$ outperforms
$\mu\text{Split}$, the current SOTA for semantic image splitting, when the
available training data is subject to pixel-noises. Since pixel-noises are
ubiquitous in biomedical light microscopy, our method addresses an important
shortcoming, enabling many potential microscopy users to benefit from the
method we devised. Additionally, we have shown that
$\text{denoi}\mathbb{S}\text{plit}$ better preserves high-frequency structures
of biological samples than the above mentioned sequential pipeline (_i.e_.
SOTA denoising followed by $\mu\text{Split}$). This is much desired for the
downstream analysis of cellular and tissue features in biomedical
applications.

Additionally, by leveraging the possibility of performing variational
sampling, $\text{denoi}\mathbb{S}\text{plit}$ offers microscopists the
opportunity to explore the learned posterior of possible solutions to the
joint task at hand. We also present a way to use this sampling capability of
$\text{denoi}\mathbb{S}\text{plit}$ to predict the expected error
$\text{denoi}\mathbb{S}\text{plit}$ makes on a given input. To this end, we
have shown how to calibrate the diversity of samples to best align with the
empirically measured error, an approach that can be applied whenever some
amount of relatively noise-free calibration data is available. We believe that
this is a key contribution of this work, because it enables automated
assessment of the data uncertainty in processed images, allowing biologist
users without a deep appreciation of the technical details of
$\text{denoi}\mathbb{S}\text{plit}$ to remove images from their dataset that
should likely not be trusted and therefore not be used for downstream analysis
in the context of biomedical investigations.

In the future, we want to explore improved backbone architectures that can
further improve the quality of predictions and achievable calibrations.
Additionally, domain adaptation techniques to fine-tune existing models on
noisy data from slightly different image domains or microscopy modalities
would promise to further ease the applicability of
$\text{denoi}\mathbb{S}\text{plit}$ by biomedical researchers. This continuous
development aims to bridge the gap between computational imaging methods and
the practicalities and limitations of modern microscopy, benefiting our
overall goal of elevating the rate of scientific discovery in the life-
sciences by conducting cutting-edge methods research.

## Acknowledgements

This work was supported by the European Commission through the Horizon Europe
program (IMAGINE project, grant agreement 101094250-IMAGINE and AI4LIFE
project, grant agreement 101057970-AI4LIFE) as well as the compute
infrastructure of the BMBF-funded de.NBI Cloud within the German Network for
Bioinformatics Infrastructure (de.NBI) (031A532B, 031A533A, 031A533B,
031A534A, 031A535A, 031A537A, 031A537B, 031A537C, 031A537D, 031A538A).
Additionally, the authors also want to thank Damian Dalle Nogare of the Image
Analysis Facility at Human Technopole for useful guidance and discussions and
the IT and HPC teams at HT for the compute infrastructure they make available
to us.

## References

  * [1] Ashesh, Krull, A., Sante, M.D., Pasqualini, F.S., Jug, F.: $\mu$Split: image decomposition for fluorescence microscopy (2023)
  * [2] Bahat, Y., Irani, M.: Blind dehazing using internal patch recurrence. In: 2016 IEEE International Conference on Computational Photography (ICCP). pp. 1–9 (May 2016)
  * [3] Batson, J., Royer, L.: Noise2Self: Blind denoising by Self-Supervision pp. 1–16 (Jan 2019)
  * [4] Berman, D., Treibitz, T., Avidan, S.: Non-local image dehazing. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1674–1682. IEEE (Jun 2016)
  * [5] Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. 2005 IEEE computer society (2005)
  * [6] Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (Aug 2007)
  * [7] Dekel, T., Rubinstein, M., Liu, C., Freeman, W.T.: On the effectiveness of visible watermarks (2017)
  * [8] Gandelsman, Y., Shocher, A., Irani, M.: “Double-DIP” : Unsupervised image decomposition via coupled deep-image-priors (2019), accessed: 2022-2-14
  * [9] Ghiran, I.C.: Introduction to fluorescence microscopy. Methods Mol. Biol. 689, 93–136 (2011)
  * [10] Hagen, G.M., Bendesky, J., Machado, R., Nguyen, T.A., Kumar, T., Ventura, J.: Fluorescence microscopy datasets for training deep neural networks. Gigascience 10(5) (May 2021)
  * [11] Jin, L., Liu, J., Zhang, H., Zhu, Y., Yang, H., Wang, J., Zhang, L., Xu, Y., Kuang, C., Liu, X.: Deep learning permits imaging of multiple structures with the same fluorophores. bioRxiv (2023)
  * [12] Kingma, D.P., Salimans, T., Welling, M.: Improving variational inference with inverse autoregressive flow. CoRR abs/1606.04934 (2016), http://arxiv.org/abs/1606.04934
  * [13] Kingma, D.P., Welling, M.: An introduction to variational autoencoders (Jun 2019)
  * [14] Ko, J., Lee, S.: Self2Self+: Single-Image denoising with Self-Supervised learning and image quality assessment loss (Jul 2023)
  * [15] Krull, A., Buchholz, T.O., Jug, F.: Noise2Void - learning denoising from single noisy images. arXiv cs.CV, 2129–2137 (Nov 2018)
  * [16] Krull, A., Vicar, T., Prakash, M., Lalit, M., Jug, F.: Probabilistic Noise2Void: Unsupervised Content-Aware denoising. Frontiers in Computer Science 2, 60 (Feb 2020)
  * [17] Lehtinen, J., Munkberg, J., Hasselgren, J., Laine, S., Karras, T., Aittala, M., Aila, T.: Noise2Noise: Learning image restoration without clean data. arxiv.org (Mar 2018)
  * [18] Levi, D., Gispan, L., Giladi, N., Fetaya, E.: Evaluating and calibrating uncertainty prediction in regression tasks. Sensors 22(15) (2022)
  * [19] Platt, J.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv. Large Margin Classif. 10 (06 2000)
  * [20] Prakash, M., Delbracio, M., Milanfar, P., Jug, F.: Interpretable unsupervised diversity denoising and artefact removal (Apr 2021)
  * [21] Prakash, M., Krull, A., Jug, F.: DivNoising: Diversity denoising with fully convolutional variational autoencoders. ICLR 2020 (Jun 2020)
  * [22] Prakash, M., Lalit, M., Tomancak, P., Krull, A., Jug, F.: Fully unsupervised probabilistic Noise2Void. arXiv eess.IV (Nov 2019)
  * [23] Shroff, H., Testa, I., Jug, F., Manley, S.: Live-cell imaging powered by computation. Nat. Rev. Mol. Cell Biol. (Feb 2024)
  * [24] Sønderby, C.K., Raiko, T., Maaløe, L., Sønderby, S.K., Winther, O.: Ladder variational autoencoders. Adv. Neural Inf. Process. Syst. 29, 3738–3746 (Jan 2016)
  * [25] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. Int. J. Comput. Vis. 128(7), 1867–1888 (Jul 2020)
  * [26] Weigert, M., Royer, L., Jug, F., Myers, G.: Isotropic reconstruction of 3D fluorescence microscopy images using convolutional neural networks. In: Medical Image Computing and Computer-Assisted Intervention - MICCAI 2017. pp. 126–134. Springer International Publishing (2017)
  * [27] Weigert, M., Schmidt, U., Boothe, T., M uuml ller, A., Dibrov, A., Jain, A., Wilhelm, B., Schmidt, D., Broaddus, C., Culley, S., Rocha-Martins, M., Segovia-Miranda, F., Norden, C., Henriques, R., Zerial, M., Solimena, M., Rink, J., Tomancak, P., Royer, L., Jug, F., Myers, E.W.: Content-aware image restoration: pushing the limits of fluorescence microscopy. Nature Publishing Group 15(12), 1090–1097 (Dec 2018)
  * [28] Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (Jul 2017)

Supplementary Material denoiSplit: a method for joint image splitting
and unsupervised denoising
Ashesh, Florian Jug

Jug Group, Fondazione Human Technopole, Milano, Italy,

Task | Model | training | Noise level parameters
---|---|---|---
$\lambda=0$ | $\lambda=1000$
$[h]$ | $\sigma=1$ | 1.5 | 2 | 4 | $\sigma=1$ | 1.5 | 2 | 4
T5 | $\mu\text{Split}$ | 5.5 | 25.1 | 23.7 | 23.1 | 22.1 | 24.8 | 23.8 | 23.0 | 22.1
0.728 | 0.633 | 0.537 | 0.341 | 0.697 | 0.593 | 0.536 | 0.307
HDN$\oplus$$\mu\text{Split}$ | 8 | 27.3 | 26.6 | 26.1 | 24.6 | 27.4 | 26.4 | 26.1 | 24.3
0.793 | 0.756 | 0.731 | 0.645 | 0.794 | 0.739 | 0.722 | 0.639
Altered $\mu\text{Split}$ (ours) | 1.3 | 27.9 | 26.8 | 25.9 | 24.5 | 27.4 | 26.5 | 25.9 | 24.4
0.799 | 0.741 | 0.698 | 0.601 | 0.780 | 0.731 | 0.697 | 0.595
$\text{denoi}\mathbb{S}\text{plit}$ (ours) | 1.5 | 27.7 | 27.0 | 25.9 | 24.5 | 27.5 | 27.1 | 26.0 | 24.4
0.798 | 0.762 | 0.707 | 0.604 | 0.787 | 0.761 | 0.710 | 0.606
T6 | $\mu\text{Split}$ | 5 | 30.5 | 28.7 | 27.8 | 26.7 | 29.9 | 28.7 | 27.9 | 26.8
0.869 | 0.779 | 0.685 | 0.422 | 0.866 | 0.788 | 0.708 | 0.443
HDN$\oplus$$\mu\text{Split}$ | 7 | 34.6 | 33.1 | 32.1 | 28.8 | 33.8 | 32.6 | 32.0 | 28.8
0.938 | 0.907 | 0.885 | 0.785 | 0.926 | 0.899 | 0.882 | 0.783
Altered $\mu\text{Split}$ (ours) | 1.3 | 35.5 | 33.3 | 32.5 | 30.4 | 34.4 | 32.8 | 32.4 | 30.1
0.950 | 0.911 | 0.890 | 0.825 | 0.935 | 0.898 | 0.887 | 0.819
$\text{denoi}\mathbb{S}\text{plit}$ (ours) | 1.5 | 36.3 | 34.4 | 33.1 | 30.4 | 34.7 | 33.8 | 32.9 | 29.9
0.945 | 0.927 | 0.895 | 0.830 | 0.939 | 0.918 | 0.890 | 0.826
Table S.1: Quantitative Results. We show quantitative evaluations for two more
tasks which are abbreviated as T5: F-actin vs. MT and T6: F-actin vs CCPs. For
all experiments, we show the PSNR (sub-row 1) and MS-SSIM (sub-row 2) metrics
across $8$ noise levels: Gaussian noise levels of $\sigma\in\\{1,1.5,2,4\\}$
and Poisson noise levels of $\lambda\in\\{0,1000\\}$. The best performance per
task and noise level is shown in bold. The third column additionally shows the
training time on a single Tesla-V100 GPU (in hours).

##  S.1 Performance on more splitting tasks

In this section, we train our models and baselines on five more tasks. We
train two tasks from BioSR dataset. Specifically, we add F-actin _vs_. CCPs
and F-actin _vs_. Microtubules tasks. In Tab. S.1, we present the quantitative
evaluation on these two tasks. Similar to the results from the Table 1 of main
manuscript, here as well, we find our methods, specifically
$\text{denoi}\mathbb{S}\text{plit}$ to outperform others in most cases.

We worked on three additional joint denoising-splitting tasks from two other
datasets which we describe next.

#### Hagen et al. Actin-Mitochondria Dataset

We picked the high resolution Actin and Mitochondria channels from Hagen et
al. [10] which were also used in [1]. Additionally, we also worked with the
low SNR version of these channels provided in the dataset as yet another task.
Note that in this task, the channels had the real microscopy noise present in
them.

#### PaviaATN dataset

[1] We worked with the Actin and Tubulin channel provided by the dataset. It
is worth noting that in terms of PSNR, we picked the hardest of the three
tasks worked upon by Ashesh et al. [1].

We provide the results on tasks generated from PaviaATN and Hagen et al.
datasets in Tab. S.2. We discuss more on these results in Sec. S.8. We show
the full-frame predictions for tasks not shown in the main manuscript in Figs.
S.4, S.7, S.5 and S.6.

##  S.2 Hyperparameter and Training details

As stated in the main manuscript, our $\text{denoi}\mathbb{S}\text{plit}$ and
Altered $\mu\text{Split}$ are built on top of $\mu\text{Split}$ architecture.
In addition to the major changes which are discussed in the main manuscript,
we have enabled free bits [12] parameter and have set $\textit{free bits}=1$.
We also do not use any additional low-resolution inputs.

$\text{denoi}\mathbb{S}\text{plit}$, Altered $\mu\text{Split}$ and HDN have
been trained with the learning rate of $0.001$, batch size of $32$, patch size
of $128$, max epoch of $400$ and with $16$ bit precision. For every task,
$80\%$ of the data was allocated as training data, $10\%$ of the data as
validation data and $10\%$ as the test data. For $\mu\text{Split}$ baseline
and used as part of HDN$\oplus$$\mu\text{Split}$, we use the same training
configuration as mentioned in [1]. For PSNR, we use the range invariant PSNR
formulation which is commonly used in this field [1, 27].

We have provided code with this supplement where information about how Poisson
noise is added is present in the file vanilla_dloader.py:L175. Specifically,
we use numpy python package to add poisson noise as

$\text{data}=\text{np.random.poisson}(data/poisson\\_factor)*poisson\\_factor$
(1)

We use $poisson\\_factor=1000$. Finally, in Tab. S.3, we provide the actual
$\sigma$ values which were used to add Gaussian noise in different tasks.

##  S.3 Practical relevance of $\text{denoi}\mathbb{S}\text{plit}$

Here, we outline the intended usecase for our work. Similar to [1], our work
also aims to enable microscopists in extracting multiple structures using a
single fluorescence marker. There are two important practical considerations
which our work addresses.

Firstly, different microscopy projects, depending upon the nature of
underlying specimen and microscope type, have different tolerance for the
amount of laser power and the dwell time which can be used during acquisition.
This roughly translates to the amount of noise which will be present in the
acquired micrographs. In this work, we cater to this necessity by working with
different noise levels.

Secondly, in most cases, there will be a need to purchase a single fluorescent
marker which can bind to both cellular structure types one is interested in.
Additionally, even after the purchase, it might still be challenging to get a
decent staining of both structures with the marker. An imperfect staining can
lead to under-expression of one of the two structure types in the imaged
micrographs. Therefore, there is also an investment of time and expertise in
getting a proper staining. That being the case, it makes sense to first
inspect the feasibility of the approach before making the investment of buying
a new marker followed by getting the staining correct. Our method provides the
way to get proof-of-concept splitting without making any of these investments.

We envision that microscopists should image individual channels in the same
noise regime as is permitted in their project. They should then train the
$\text{denoi}\mathbb{S}\text{plit}$ and inspect the prediction quality. If
there is a room for adjustment in power and dwell time, they can re-acquire in
a different noise regime and train the $\text{denoi}\mathbb{S}\text{plit}$
again. Note that this acquisition can be done with a single-color setup and
therefore can be done using their existing microscope configuration.

In case they find the model performance satisfactory in some feasible noise
regime, they can then order the relevant fluorescent marker and can
subsequently label their structures with it to get superimposed images.
Finally, they need to finetune the trained network to this slightly different
input data and then they can start using $\text{denoi}\mathbb{S}\text{plit}$.
This problem of finetuning the model is outside the scope of this work and
will be taken up in our future work.

$\text{denoi}\mathbb{S}\text{plit}$ also enables sampling which, as stated in
the main manuscript, will enable microscopists to inspect the predictions to
get a visual feeling about uncertain areas in prediction. To showcase this, we
provide with this supplement, a sampling.gif file where one could see 50
samples on a randomly choosen input from MT _vs_. ER task.

##  S.4 Noise model generation

Depending upon the physical availability of microscope, [21] described two
ways in which the noise model can be generated. In this section, our
motivation is to assess the performance difference which one should expect
between these two ways. We find that different choices of noise model
generation do not lead to very different performances. This result enables
$\text{denoi}\mathbb{S}\text{plit}$ to be used across a wide variety of
scenarios.

###  S.4.1 Physical availability of Microscope

The simpler and the better case is when we have access to the microscope which
generated the noisy data. In this case, one needs to image the same content
$N$ times which would result in acquisition of $N$ noisy versions of the same
underlying specimen. The high SNR version is then computed by simply taking
the pixelwise average of these noisy samples. In this situation, for every
clean signal value, one has $N$ noisy intensities. This data is then used to
train the Gaussian mixture based noise model.

To simulate this condition, for every intensity value in the range
$[0,65535]$, we obtain multiple noisy intensity values by adding the noise
(Gaussian and/or Poisson) multiple times independently.

###  S.4.2 Physical unavailability of Microscope

In case the microscope is not available, which is true for all publicly
available microscopy datasets, [21] proposed a bootstrap noise model approach.
In this method, the idea was to denoise the noisy data using some
unsupervised/self-supervised denoising technique and use the noisy data and
the predicted denoised data to generate the noise model.

To simulate this condition, we added noise to the noise free training data.
This gave us the noisy data and clean data pair which we used to train the
noise model. To showcase the applicability of our method to all publicly
available datasets, we have generated all noise models using this approach.

It is worth noting that in the way described above, there is a source of error
which has not been captured. This is the error introduced by the denoising
process for getting the clean data from its noisy counterpart. To assess the
effect of denoising, we generate the noise model in yet another way. In this
way, we added the noise to the noise free training data. We then use N2V [15],
an off-the-shelf denoiser to denoise the images. We use the noisy data and the
denoised images to generate the noise model.

Figure S.1: Quantitative comparison of different noise model generation
methodologies: Here, we compare three ways in which noise models can be
generated. (a) denoiSplit$+S_{\infty}$: When for every clean pixel intensity,
one has access to multiple noisy intensities. This corresponds to the case
when one has access to the microscope which has generated the data. (b)
denoiSplit$+S1$: This corresponds the case when one has access to clean data
and its corresponding noisy data. (c) denoiSplit$+N2V$: This corresponds to
the case when one has access to just noisy data. We use N2V to denoise them
which we use as clean data. We compare $\text{denoi}\mathbb{S}\text{plit}$
trained using each of the three noise model variants. We show PSNR performance
on two tasks.

###  S.4.3 Performance comparison among different noise model generation
procedures

To assess how performance varies depending upon the methodology used for noise
model creation which have been described above subsections, we train
$\text{denoi}\mathbb{S}\text{plit}$ three times, each time with its noise
model computed using a different way. In Fig. S.1, we show the performance
comparison. denoiSplit$+S_{\infty}$ denotes the case when one has access to
microscope. denoiSplit$+S1$ denotes the case when one works with a noisy and
its clean data pair. denoiSplit$+N2V$ denotes the case when one obtains the
clean data after denoising with N2V. We train them on two tasks namely ER
_vs_. CCPs and ER _vs_. MT over four Gaussian noise levels with Poisson noise
($\lambda=1000$) enabled. As can be observed from the plots, we don’t see a
large difference between the three approaches.

This performance evaluation encourages $\text{denoi}\mathbb{S}\text{plit}$ to
be used on publicly available datasets and for proof-of-concept evaluations
since it shows that having access to the microscope does not give large
performance improvements and the bottleneck in denoising-splitting is
essentially the splitting task.

Figure S.2: Problem comparison: Joint Denoising-Splitting _vs_. Denoising In
this figure, we compare the PSNR with respect to high-SNR micrographs for two
image restoration tasks: (a) Self-supervised denoising for which we use HDN
and (b) joint denoising-splitting where $\text{denoi}\mathbb{S}\text{plit}$ is
used. We evaluate the two models over multiple noise levels of Gaussian noise
(x-axis) with Poisson($\lambda=1000$) noise present in all cases. For
$\text{denoi}\mathbb{S}\text{plit}$, we use three tasks namely: ER _vs_. MT,
CCPs _vs_. ER and CCPs vs MT. We show one plot for each structure type (ER,
Microtubules and CCPs). Two things are evident: (a) Judging just from PSNR
numbers, we can say that Denoising is a simpler task than joint Denoising-
Splitting. (b) Performance of $\text{denoi}\mathbb{S}\text{plit}$ for one
channel depends on the other channel as well. For example, for
$\text{denoi}\mathbb{S}\text{plit}$, PSNR on ER channel (first plot) is higher
when the task is CCPs _vs_. ER (green) as opposed to ER _vs_. MT (orange)
task.  Figure S.3: In this figure, we show that as we increase the sample
count to get the uncalibrated estimate of pixelwise uncertainty, the
calibration diagram as shown in this plot becomes better and better, _i.e_.,
gets more and more closer to $y=x$.

##  S.5 Comparison between denoising task and splitting task

In this section, we compare between two computer vision tasks which are of
relevance to us: (a) unsupervised denoising and (b) joint denoising-splitting.
From Fig. S.2, we observe that unsupervised denoising is arguably a simpler
task when compared to joint denoising-splitting. The PSNR between the
prediction and high SNR micrographs is much better for HDN as compared to
$\text{denoi}\mathbb{S}\text{plit}$ for most cases. We note that it is
expected because in joint denoising-splitting, besides denoising, which is the
sole task in unsupervised denoising, one needs to additionally do the job of
image decomposition. But more interestingly, we observe that the prediction
quality of one channel depends upon the other channel.

We note that this observation is of considerable importance because it opens
up the question of best pairing strategy: which two structures should be
imaged by a single color fluorescent marker? This will be part of our future
work.

##  S.6 On usefulness of using sampling for calibration

In this section, we investigate our choice for estimating un-calibrated
uncertainty using sampled predictions. For this, we estimate the uncalibrated
pixelwise uncertainty using varying number of samples. We then follow our
calibration procedure and learn the channelwise scalar to get the calibration
plot. As can be observed in Fig. S.3, as we increase the number of samples,
the calibrated plot also improves thereby validating our choice.

##  S.7 Qualitative evaluation of HDN denoising

Here, we qualitatively evaluate the denoising behaviour of HDN. We show three
random input patches and the corresponding channel first and channel second
crops from 6 different splitting tasks in Figs. S.10, S.8, S.9, S.11, S.13 and
S.12. We show the results on noisy dataset having Poisson($\lambda=1000)$
noise and Gaussian noise of relative scale $1.5$. Since main manuscript also
shows the qualitative figures on this noise level, we believe this can be used
together with the figures present in main manuscript to better understand the
behaviour of HDN$\oplus$$\mu\text{Split}$.

##  S.8 Failure cases: Avenues for future work

In this section, we inspect the worse performing cases in our work. One clear
example is Actin _vs_. Tubulin task from PaviaATN [1] dataset whose
quantitative evaluation is present in Tab. S.2. However, as can be seen from
Fig. S.14, we find performance of both HDN$\oplus$$\mu\text{Split}$ and
$\text{denoi}\mathbb{S}\text{plit}$ to be unsatisfactory for it to be used by
microscopists. We note a striking difference of this task with the tasks from
BioSR dataset. Looking at $128\times 128$ input patches for tasks from BioSR
data, one could visually form an opinion about which structures in the input
patch should belong to first channel and which to the second channel. In case
of Actin _vs_. Tubulin task, we observe that making this opinion is much more
difficult since local structures are much less discriminative. It is only when
looking at a larger context, one can form some opinion about which structures
should be present in which channel. There seems to be another factor related
to the nature of the structures which the channels are composed of. Informally
speaking, individual channels have a "surface" like structure in Actin and
Tubulin images of PaviaATN dataset as opposed to "curved lines", "mesh" and
"dots" like structures in BioSR dataset.

Between $\text{denoi}\mathbb{S}\text{plit}$ and HDN$\oplus$$\mu\text{Split}$,
we observe that for this task, $\text{denoi}\mathbb{S}\text{plit}$ has more
tiling artefacts. It is not surprising for that to be the case because in [1],
the Lateral contextualization (LC) approach which incorporates context in a
memory efficient way, worked well on Actin _vs_. Tubulin. Compared to
HDN$\oplus$$\mu\text{Split}$, our model is naturally at disadvantage because
we have disabled the LC module but HDN$\oplus$$\mu\text{Split}$ uses it. We
believe that increasing the patch size can help our
$\text{denoi}\mathbb{S}\text{plit}$ reduce the tiling artefacts.

In general, we also find cases where $\mu\text{Split}$ has retained some fine
structures, albeit with noise, which the denoising based approaches have
omitted from the prediction. We argue this to be a natural consequence of
restricting the expressivity of latent spaces with KL divergence loss, which
is pivotal for denoising.

As joint denoising-Splitting is a new task, there is much that needs to be
done. We humbly acknowledge the challenges mentioned above which we hope to
tackle in our future works.

\begin{overpic}[width=433.62pt]{sup_imgs/108_realnoise_hagen_actin-60x-noise2-lowsnr_mito-60x-noise2-lowsnr.png} \put(86.0,40.0){\scriptsize High SNR} \put(66.0,40.0){\scriptsize\mbox{$\text{denoi}\mathbb{S}\text{plit}$} } \put(48.0,40.0){\scriptsize\mbox{$\mu\text{Split}$} } \put(30.0,40.0){\scriptsize GT} \put(8.0,40.0){\scriptsize Input} \end{overpic} Figure S.4: Qualitative Results Actin _vs_. Mito: In this figure, we show full frame prediction on Actin _vs_. Mitochondria task. Here, the noise in the target channels is not synthetic but is real microscopy noise. We show noisy input (column one), individual noisy channel training data (column two), and predictions by one of the baselines $\mu\text{Split}$ (column three) and our own results obtained with $\text{denoi}\mathbb{S}\text{plit}$ (column four). Additionally we show high SNR channel images (not used during training) as the last column and show PSNR values w.r.t. these images. Additionally, we plot histograms of pixel intensities various panels for comparison (see legend on the right). The second row, first column shows the used noise models. The superimposed plots (green) show the distribution of noisy observations ($c_{i}^{N}$) for two clean signal intensities.  Model | Tasks
---|---
T7 | T8 | T9
1 | 1.5 | 2 | 4 | 1 | 1.5 | 2 | 4 |
$\mu\text{Split}$ | 22.6 | 21.1 | 20.2 | 19.1 | 28.7 | 27.0 | 26.2 | 25.2 | 26.5
0.555 | 0.442 | 0.361 | 0.189 | 0.905 | 0.825 | 0.747 | 0.489 | 0.887
HDN$\oplus$$\mu\text{Split}$ | 27.8 | 27.3 | 27.0 | 26.4 | - | - | - | - | 28.1
0.880 | 0.871 | 0.865 | 0.843 | - | - | - | - | 0.760
(Ours) Altered $\mu\text{Split}$ | 26.3 | 26.2 | 25.9 | 25.2 | 34.7 | 34.2 | 33.6 | 32.3 | 31.1
0.838 | 0.826 | 0.820 | 0.807 | 0.975 | 0.971 | 0.966 | 0.951 | 0.887
(Ours) denoisplit | 26.4 | 26.1 | 26.0 | 25.2 | 35.5 | 33.7 | 33.5 | 32.2 | 31.0
0.835 | 0.827 | 0.825 | 0.807 | 0.979 | 0.974 | 0.968 | 0.951 | 0.887
Table S.2: T7: Actin vs Tubulin from PaviaATN dataset, T8: Actin vs Mito, T9:
Actin vs Mito low SNR. For Hagen et al. dataset tasks (T8 and T9), HDN
training was quite unstable and crashed multiple times due to NaNs. Due to
this reason, there are no entries for HDN$\oplus$$\mu\text{Split}$ for the
task T8. For task T9, we were able to train the HDN$\oplus$$\mu\text{Split}$
with a lower hierarchy count (3 as opposed to default 6). For T9, similar to
[1], we also upper bounded the log of variance of the latent space to $20$
across all hierarchy levels for further stability.
\begin{overpic}[width=433.62pt]{sup_imgs/6675_F-actin_a_ER.png}
\put(86.0,40.0){\scriptsize High SNR}
\put(66.0,40.0){\scriptsize\mbox{$\text{denoi}\mathbb{S}\text{plit}$} }
\put(48.0,40.0){\scriptsize\mbox{$\mu\text{Split}$} }
\put(30.0,40.0){\scriptsize GT} \put(8.0,40.0){\scriptsize Input}
\end{overpic} Figure S.5: Qualitative Results F-actin _vs_. ER: In this
figure, we show full frame prediction on F-actin _vs_. ER task. We show noisy
input (column one), individual noisy channel training data (column two), and
predictions by one of the baselines $\mu\text{Split}$ (column three) and our
own results obtained with $\text{denoi}\mathbb{S}\text{plit}$ (column four).
Additionally we show high SNR channel images (not used during training) as the
last column and show PSNR values w.r.t. these images. Additionally, we plot
histograms of pixel intensities various panels for comparison (see legend on
the right). The second row, first column shows the used noise models. The
superimposed plots (green) show the distribution of noisy observations
($c_{i}^{N}$) for two clean signal intensities.
\begin{overpic}[width=433.62pt]{sup_imgs/39_6450_F-actin_a_Microtubules.png}
\put(86.0,40.0){\scriptsize High SNR}
\put(66.0,40.0){\scriptsize\mbox{$\text{denoi}\mathbb{S}\text{plit}$} }
\put(48.0,40.0){\scriptsize\mbox{$\mu\text{Split}$} }
\put(30.0,40.0){\scriptsize GT} \put(8.0,40.0){\scriptsize Input}
\par\end{overpic} Figure S.6: Qualitative Results F-actin _vs_. MT: In this
figure, we show full frame prediction on F-actin _vs_. MT task. We show noisy
input (column one), individual noisy channel training data (column two), and
predictions by one of the baselines $\mu\text{Split}$ (column three) and our
own results obtained with $\text{denoi}\mathbb{S}\text{plit}$ (column four).
Additionally we show high SNR channel images (not used during training) as the
last column and show PSNR values w.r.t. these images. Additionally, we plot
histograms of pixel intensities various panels for comparison (see legend on
the right). The second row, first column shows the used noise models. The
superimposed plots (green) show the distribution of noisy observations
($c_{i}^{N}$) for two clean signal intensities.
\begin{overpic}[width=433.62pt]{sup_imgs/83_4575_F-actin_a_CCPs.png}
\put(86.0,40.0){\scriptsize High SNR}
\put(66.0,40.0){\scriptsize\mbox{$\text{denoi}\mathbb{S}\text{plit}$} }
\put(48.0,40.0){\scriptsize\mbox{$\mu\text{Split}$} }
\put(30.0,40.0){\scriptsize GT} \put(8.0,40.0){\scriptsize Input}
\end{overpic} Figure S.7: Qualitative Results F-actin _vs_. CCPs: In this
figure, we show full frame prediction on F-actin _vs_. CCPs task. We show
noisy input (column one), individual noisy channel training data (column two),
and predictions by one of the baselines $\mu\text{Split}$ (column three) and
our own results obtained with $\text{denoi}\mathbb{S}\text{plit}$ (column
four). Additionally we show high SNR channel images (not used during training)
as the last column and show PSNR values w.r.t. these images. Additionally, we
plot histograms of pixel intensities various panels for comparison (see legend
on the right). The second row, first column shows the used noise models. The
superimposed plots (green) show the distribution of noisy observations
($c_{i}^{N}$) for two clean signal intensities.

Figure S.8: Qualitative performance of HDN on ER _vs_. CCPs task input and its
two constituent channels. Left panel shows the denoising performance on input
for our splitting task and central and right panel shows its denoising
performance on its two constituent channels. Within each panel, we show three
random patches (rows) of size $256\times 213$. Specifically, we show the input
(first column), denoised predictions (second column) and high SNR patch (last
column).

Figure S.9: Qualitative performance of HDN on ER _vs_. MT task input and its
two constituent channels. Left panel shows the denoising performance on input
for our splitting task and central and right panel shows its denoising
performance on its two constituent channels. Within each panel, we show three
random patches (rows) of size $256\times 213$. Specifically, we show the input
(first column), denoised predictions (second column) and high SNR patch (last
column).

Figure S.10: Qualitative performance of HDN on CCPs _vs_. MT task input and
its two constituent channels. Left panel shows the denoising performance on
input for our splitting task and central and right panel shows its denoising
performance on its two constituent channels. Within each panel, we show three
random patches (rows) of size $256\times 213$. Specifically, we show the input
(first column), denoised predictions (second column) and high SNR patch (last
column).

Figure S.11: Qualitative performance of HDN on F-actin _vs_. CCPs task input
and its two constituent channels. Left panel shows the denoising performance
on input for our splitting task and central and right panel shows its
denoising performance on its two constituent channels. Within each panel, we
show three random patches (rows) of size $256\times 213$. Specifically, we
show the input (first column), denoised predictions (second column) and high
SNR patch (last column).

Figure S.12: Qualitative performance of HDN on F-actin _vs_. MT task input and
its two constituent channels. Left panel shows the denoising performance on
input for our splitting task and central and right panel shows its denoising
performance on its two constituent channels. Within each panel, we show three
random patches (rows) of size $256\times 213$. Specifically, we show the input
(first column), denoised predictions (second column) and high SNR patch (last
column).

Figure S.13: Qualitative performance of HDN on F-actin _vs_. ER task input and its two constituent channels. Left panel shows the denoising performance on input for our splitting task and central and right panel shows its denoising performance on its two constituent channels. Within each panel, we show three random patches (rows) of size $256\times 213$. Specifically, we show the input (first column), denoised predictions (second column) and high SNR patch (last column). Task | $\sigma=1$ | $\sigma=1.5$ | $\sigma=2$ | $\sigma=4$
---|---|---|---|---
ER _vs_. CCPs | 3400 | 5100 | 6800 | 13600
ER _vs_. MT | 4450 | 6675 | 8900 | 17800
CCPs _vs_. MT | 3150 | 4725 | 6300 | 12600
F-actin _vs_. ER | 4450 | 6675 | 8900 | 17800
F-actin _vs_. CCPs | 3050 | 4575 | 6100 | 12200
F-actin _vs_. MT | 4300 | 6450 | 8600 | 17200
Table S.3: Gaussian $\sigma$ values for the different tasks. Note that they
have been estimated by computing the standard deviation on the input images of
these tasks.

Figure S.14: PaviaATN Actin _vs_. Tubulin task Here, we show performance of
$\text{denoi}\mathbb{S}\text{plit}$ and HDN$\oplus$$\mu\text{Split}$ for six
random input patches of size $500\times 500$ in six panels. Within each panel,
we show the full input frame and its crop for which we do the predictions
(column one). Next two columns have the predictions of
HDN$\oplus$$\mu\text{Split}$ and $\text{denoi}\mathbb{S}\text{plit}$
respectively. The last column is the high SNR ground truth. We observe that
the splitting performance of both HDN$\oplus$$\mu\text{Split}$ and
$\text{denoi}\mathbb{S}\text{plit}$ does not reach the quality at which
microscopists would find it useful. Between HDN$\oplus$$\mu\text{Split}$ and
$\text{denoi}\mathbb{S}\text{plit}$, we see more tiling artefacts for
$\text{denoi}\mathbb{S}\text{plit}$.