# Correcting for objective sample refractive index mismatch in extended field
of view selective plane illumination microscopy

Steven J. Sheppard  Peter T. Brown  Douglas P. Shepherd
<EMAIL_ADDRESS>Center for Biological Physics and Department of
Physics, Arizona State University, Tempe, AZ

###### Abstract

Selective plane illumination microscopy (SPIM) is an optical sectioning
imaging approach based on orthogonal light pathways for excitation and
detection. The excitation pathway has an inverse relation between the optical
sectioning strength and the effective field of view (FOV). Multiple approaches
exist to extend the effective FOV, and here we focus on remote focusing to
axially scan the light sheet, synchronized with a CMOS camera’s rolling
shutter. A typical axially scanned SPIM configuration for imaging large
samples utilizes a tunable optic for remote focusing, paired with air
objectives focused into higher refractive index media. To quantitatively
explore the effect of remote focus choices and sample space refractive index
mismatch on light sheet intensity distributions, we developed a computational
model integrating ray tracing and field propagation. We validate our model’s
performance against experimental light sheet profiles for various SPIM
configurations. Our findings indicate that optimizing the position of the
sample chamber relative to the excitation optics can enhance image quality by
balancing aberrations induced by refractive index mismatch. We validate this
prediction using a homebuilt, large sample axially scanned SPIM configuration
and calibration samples.

##  I Introduction

Selective plane illumination microscopy (SPIM), or light sheet fluorescence
microscopy (LSFM), encompasses techniques that enable rapid volumetric imaging
of samples. SPIM approaches commonly involve two orthogonal objectives, one
delivering light to the sample (excitation pathway) and one collecting emitted
fluorescence (emission pathway) [1, 2, 3]. Shaping the excitation laser into a
“light sheet” that is thin along the detection axis enables optical
sectioning, leading to high-contrast images, even in densely labeled samples.
Successful applications of SPIM span a wide range, including imaging of entire
zebrafish embryos [2], tracking single cell dynamics in early mouse
development[4], observing neural activity in the brains of small organisms
[5], imaging of the central nervous system of mice [6], and extracting the
hydrodynamic properties of flagella by observing their free diffusion [7].
These applications demonstrate SPIM’s capability to provide detailed, high-
resolution images while minimizing photodamage and photobleaching.

All light sheet generation strategies have an intrinsic coupling of the width
at the focus and the axial extent of the focus. One strategy to extend the
axial extent for a given focus width is to use self-healing beams, such as
Bessel, Lattice, or Airy beams [8, 9, 10]. Gaussian beam propagation defines
the the beam waist $\omega_{0}=\frac{2\lambda}{\pi NA_{exc}}$ and confocal
parameter (twice the Rayleigh length) $2\times
z_{R}=2\times\frac{2\lambda}{\pi NA^{2}_{exc}}$ to parameterize the focus
width and propagation length. Recently, both simulations and experiments have
settled that Gaussian and “sinc” (given by a flat top profile) beams have
larger $z_{R}$ than the Gaussian prediction, allowing for high-contrast
imaging volumes similar to certain self-healing beams [8, 9]. The theoretical
discrepancy arises because Gaussian predictions rely on the paraxial wave
equation, which assumes small angles and does not account for diffraction
effects at larger angles. Predominantly, prior simulations characterized
various static and scanned light sheet modalities by propagating analytic
electric field amplitude and phase distributions using beam propagation
methods, to generate the 3D intensity distribution about the focal plane [8,
9, 11, 12]. Experimentally, deviation from a Gaussian focus results from
overfilling the pupil, leading to a more uniform line intensity in the pupil
and a $sinc$ rather than a Gaussian distribution at the focus [10].

For samples that extend beyond the $\omega_{0},z_{R}$ for a given optical
configuration, there will be a spatially variant resolution and contrast,
Figure 1. Two common strategies to achieve a spatially invariant imaging
system are to (1) limit the camera field of view (FOV) to the invariant region
of the excitation or (2) increase the confocal parameter by increasing the
excitation beam’s $\omega_{0}$. The first strategy limits the effective
imaging rate, and the second limits the optical sectioning, contrast, and
potential resolution gains.

Figure 1: (A-B) 200 z-stack (dz=0.5) ZX maximum intensity projections
(gamma=0.75) of 1 beads embedded in agar. (A) Acquired using a static light
sheet, illustrating a limited FOV. (B) Acquired using axially-scanned light
sheet microscopy to extend the FOV. (C) Illustrating the translation a focus
and selection of the in-focus and the out-of-focus regions. (D) Spatially
variant PSF originating from various regions of a static light sheet.

To fully exploit SPIM’s potential space-time bandwidth, several innovative
strategies have emerged to create a spatially invariant imaging system by
extending the region of “best” excitation. Conceptually, these approaches
synthesize an extended, high-quality FOV by imaging different parts of the
sample with a different light sheet, depicted in Figure 1. The differences
between methods lie in the mode of extension, ranging from physically
translating the sample through the light sheet focus [13], acquiring
independent images of the light sheet in different positions [14], or
continuously sweeping the light sheet position via remote focus [15, 16],
synchronously with the digital readout of a programmable CMOS detector [17].

Inspired by recent insights into light sheet excitation profiles obtained by
modeling and experiment, we sought to undertake a more comprehensive effort to
model light sheet profiles, subject to remote focusing, in refractive index
(RI) mismatched samples. Specifically, we aim to better understand the
interplay between the specific remote focusing implementation and
heterogeneous RI pathway on the excitation beam’s 3D intensity distribution.
Heterogeneous RI can arise from mismatches in the excitation objective versus
sample chamber, imaging media, sample, or some combination of all of these. To
further explore the consequences of instrument or experimental design choices
on the resulting image quality, we present here a computational approach to
simulate the 3D light sheet intensity distribution that integrates the remote
focus pathway aberrations as well as the effects of a converging beam
propagating through an index mismatch.

We focus on air immersion excitation objectives paired with a glass chamber
and aqueous imaging media, a combination commonly found when imaging large
samples. Both our computational and experimental results confirm that low
numerical apertures (NA) remote focusing with a tunable optic is a robust
method to extend the FOV with uniform image contrast, even in the presence of
a RI mismatch. However, even at moderate NA ($\sim 0.3$), we find significant
changes to excitation beam focus location and beam intensity profile for
tunable optic remote focusing. In the presence of heterogeneous RI, our model
predicts that there exists a combination of applied light sheet displacement
and excitation lens to sample chamber spacing that minimizes aberrations in
the excitation path. We experimentally validated the model predictions by
characterizing the PSF across the FOV for static light sheets, demonstrating
that there exist optimal combinations of light sheet position and objective-
chamber spacing.

##  II Methods

###  II.1 Optical modeling

To accurately model both aberrations caused by the light sheet optical train
and diffraction during beam propagation, we developed a model which
incorporates both ray tracing and field propagation. First, we launched a
collection of collimated rays from a starting plane. We model the intensity
distribution by stochastically sampling rays from either a Gaussian or uniform
distribution to simulate Gaussian and flat-top laser profiles. Then, we ray
traced through the optical system by applying Snell’s law of refraction at
each surface while tracking the optical path length (OPL) of each ray. This
allows us to account for aberrations induced by the optical system.

We modelled optical elements as flat or spherical surfaces with material
refractive indices [18] according to the manufacturers specifications and a
propagation wavelength of 561. Objectives are modeled as perfect lenses which
transform a ray’s height and angle in the front focal plane (FFP) to the angle
and height in the back focal plane (BFP) according to

$\displaystyle\frac{h_{2}}{f}$ $\displaystyle=$ $\displaystyle
n_{1}\sin\theta_{1}$ (1) $\displaystyle n_{2}\sin\theta_{2}$ $\displaystyle=$
$\displaystyle-\frac{h_{1}}{f}$ (2)

where $f$ is the vacuum focal length and we suppose the lens is immersed in
media with refractive index $n_{1}$ and $n_{2}$ the left and right
respectively. The FFP is $n_{1}f$ to the left of the lens, and the BFP is
$n_{2}f$ to the right of the lens. This construction ensures that an imaging
system composed from two perfect lenses satisfies the Abbe sine condition, but
does not account for either field-dependent or field-independent aberrations.

After ray tracing through the final optical surface, we obtain the electric
field on a grid and propagate it through the light sheet focus using the
exact-transfer function (angular spectrum) method [19] that relates the
electric field at axial position $z$ to the initial field,

$E(x,y,z)=\mathcal{F}^{-1}\left[\mathcal{F}\left[E(x,y,z=0)\right]e^{\imath
z\sqrt{\left(\frac{2\pi n}{\lambda}\right)^{2}-k_{x}^{2}-k_{y}^{2}}}\right].$
(3)

Here, $\mathcal{F}$ represents a 2D Fourier transform over the $x$-, and
$y$-coordinates. This approach entails utilizing a scalar approximation which
is valid due to the moderately low numerical apertures commonly used in light
sheet illumination [20].

For the optical pathways considered here, we assumed radial symmetry and only
ray traced a one-dimensional slice through each optical pathway. This gives us
a collection of points $\\{(r_{i},OPL_{i})\\}_{i=1,...,N}$ (see fig. 2C). From
these discrete points, we create an interpolating function for the optical
path length, $OPL_{\text{int}}(r)$. To handle the amplitude, we assume the
density of rays is proportional to the flux through the surface. First, we
generate a histogram over radial positions, then create an interpolating
function from this, $\phi_{\text{int}}(r)$. In order to satisfy energy
conservation and accommodate a spatially varying Poynting vector, we create an
interpolation function for the ray angles, $\Theta_{int}(r)$. Finally, we
estimate the electric field on a 2D grid $(x_{i},y_{j})$ as equation 4 . We
choose grid spacing of $\qty{100}{\nano}\sim\lambda/4n_{\text{water}}$, which
is Nyquist sampled with respect to the maximum propagating spatial frequency.

$E(x_{i},y_{j})=\sqrt{\frac{\phi_{\text{int}}\left(\sqrt{x_{i}^{2}+y_{j}^{2}}\right)}{\cos\left(\Theta_{int}\left(\sqrt{x_{i}^{2}+y_{j}^{2}}\right)\right)}}\exp\left[i\
OPL_{\text{int}}\left(\sqrt{x_{i}^{2}+y_{j}^{2}}\right)\right]$ (4)

To determine the asymmetric light sheet intensity from these radially
symmetric simulations, we combined two simulations using excitation beams
having Gaussian waists matching the two asymmetric light sheet waists,
$\omega_{0}^{(x)}$ and $\omega_{0}^{(y)}$, equation 6. We combine these two
solutions in analogy to the following connection between intensities valid for
Gaussian beams, equation 5.

$\displaystyle I_{G}(x,y,z;\omega_{0}^{i})$ $\displaystyle=$
$\displaystyle\left(\frac{\omega_{0}^{i}}{\omega^{i}(z)}\right)^{2}\exp\left[-2\frac{\left(x^{2}+y^{2}\right)}{\omega^{i}(z)^{2}}\right]$
(5) $\displaystyle I_{asy}(x,y,z;\omega_{0}^{(x)},\omega_{0}^{(y)})$
$\displaystyle=$
$\displaystyle\frac{\omega_{0}^{(x)}\omega_{0}^{(y)}}{\omega^{(x)}(z)\omega^{(y)}(z)}\exp\left[\frac{-2x^{2}}{{\omega^{(x)}(z)}^{2}}+\frac{-2y^{2}}{{\omega^{(y)}(z)}^{2}}\right]$
(6) $\displaystyle=$
$\displaystyle\sqrt{I_{G}\left(\sqrt{2}x,0,z;\omega_{0}^{(x)}\right)}\sqrt{I_{G}\left(0,\sqrt{2}y,z;\omega_{0}^{(y)}\right)}.$
(7)

We assume a similar relationship holds for our model profiles $I_{M}$. In
practice, we simulate the light sheet propagation along the axis with the
tight light sheet focus, and assume an exact Gaussian form along the tall axis
of the light sheet. We expect that optical aberrations only weakly affect this
axis, and we find simulations using this approach agree well with our
experiments.

$I_{\text{LS}}(x,y,z;\omega_{0}^{(x)},\omega_{0}^{(y)})\approx\frac{\omega_{0}^{(x)}}{\omega^{(x)}(z)}\exp\left[-\frac{2x^{2}}{\omega^{(x)}(z)^{2}}\right]\sqrt{I_{M}\left(0,\sqrt{2}y,z;\omega_{0}^{(y)}\right)}$
(8)

Figure 2: (A) Light sheet model excitation pathway, where _z_ is aligned with
the propagation direction and _x_ along the narrow axis of the light sheet.
(B) The initial ray flux, $\phi(r)$, and path length difference, $\Delta
OPL(r)$. (C) The ray $\phi(r)$ and $\Delta OPL(r)$ input for computing the
electric field to be propagated using the exact-transfer function. The
resulting intensity, $I(r)$, and electric field phase amplitude, $\Phi(r)$ are
shown. (D) Focus quality was estimated by fitting the wavefront in a pseudo
objective’s back focal plane.

Before determining the light sheet intensity profile, we first characterize
the focal plane using ray tracing. Determining the focal plane in the presence
of aberrations is complicated by the spread in optical axis intersection
between paraxial and marginal rays. For small aberrations, the best-fitting
Gaussian reference sphere is centered at the midpoint between the two ray
intersections [21]. Beyond this, diffraction effects must be considered. In
our optical model, the focal plane is defined as the midpoint between the
paraxial and marginal ray intersections with the optical axis.

To characterize wavefront aberrations, we estimated the effective objective
pupil wavefront by re-sampling the rays with an objective whose parameters
depended on the resulting focal plane and immersion media. The action of the
objective eliminated the curvature attributed to focusing, resulting a flat
pupil plane to characterize wavefront aberrations. We defined the wavefront
$W(r)$ as the optical path length difference with respect to the the paraxial
ray. Wavefront distributions were fit using an $8^{th}$ order polynomial. The
polynomial coefficients, $C_{i}$, are closely related to the Zernike
aberration representation. Here we only include azimuthally symmetric
aberrations, as our simulation is symmetric. For example, $C_{4}$ is a direct
measure of spherical aberration while defocus is characterized by a
combination of $C2$ and $C_{4}$[22]. To quantify wavefront aberrations, we
defined an aberration function by subtracting the piston term. For a holistic
interpretation of the focus quality, we calculated the wavefront root-mean-
square deviation (RMS) and Strehl ratio [23, 22].

$\displaystyle W(\rho)$ $\displaystyle=$
$\displaystyle\text{OPL}(\rho)-\text{OPL}(0)=\sum_{i=0}^{8}C_{i}\rho^{i}$ (9)
$\displaystyle\Phi(\rho)$ $\displaystyle=$ $\displaystyle W(\rho)-C_{0}$ (10)
$\displaystyle S$ $\displaystyle=$
$\displaystyle\frac{1}{\pi^{2}}\left|\int_{0}^{2\pi}\int_{0}^{1}e^{ik_{o}\Phi(\rho)}\rho\
d\rho d\phi\right|^{2}$ (11) $\displaystyle RMS$ $\displaystyle=$
$\displaystyle\sqrt{\frac{1}{\pi}\int_{0}^{2\pi}\int_{0}^{1}\left|\Phi(\rho)-\overline{\Phi}\right|^{2}\rho\
d\rho d\phi}$ (12)

where $\rho$ and $\phi$ describe the normalized pupil position, and
$\overline{\Phi}$ is the mean wavefront aberration.

Additionally, we employed ray tracing to characterize the axial spread of rays
(fig. 5). The difference in optical axis intersection between marginal and
paraxial rays is a method to estimate spherical aberration [24].

### Simulation of light sheet illuminations

We simulated the remote focus pathway used in the light sheet acquisitions as
shown in figure 2A. Each model included a thick lens of variable focal length
to simulate an electrotunable lens (ETL), two achromatic doublet lenses
($L_{1}$, $L_{2}$) and an objective ($EXC_{obj}$). We assume the light sheet
propagates through a cuvette, which we model as a flat (plano-plano) glass
surface of thickness 1.25 and RI=$1.4585$. The region inside the cuvette is
filled with water, RI=$1.33$. The RI mismatch at the cuvette surface shifts
the focus and reduces the effective NA.

The ETL was modeled using a flat first surface, a spherical second surface of
variable curvature and fixed edge thickness. The ETL was parameterized by the
edge thickness and the lens power in diopters (^-1). The lens material RI was
set to 1.3 to match the manufacturers specifications, and edge thickness was
estimated to be 10. To map the ETL power or focal length to the second surface
radius of curvature, $R_{2}$, we applied the lens makers equation for a plano-
convex or concave lens.

To align the lenses and ensure the ETL and objective are in conjugate planes,
we deployed ABCD matrix methods to identify the lens’ principle and focal
planes. After ray tracing each component, we determined the intensity
distribution about the focus as described above and extracted the central
slice, $I_{M}(x,0,z)$ for comparison to experimental data.

All computation, except for data acquisition, was performed using a Linux Mint
20 server with two Xeon E5-2650 CPUs (Intel) with $16$ cores each, 1 RAM, and
two GeForce RTX 3090 Ti (Nvidia) with 24 of memory.

###  II.2 ETL remote focus optimization

To investigate the effects of using an ETL for remote focusing we used ray
tracing to characterize the axial aberrations. The remote focus pathway
includes the same components as described above, with and without the cuvette
and with models for multiple commercial objectives (0.14 NA Mitutoyo
378-802-6, 0.3 NA Nikon CFI Plan Fluor MRH00105). In each case we tested all
physically attainable configurations of cuvette position and ETL curvature.
The cuvette offset positions spanned the space between the lenses surface
element and the lenses working distance. In the presence of the cuvette, the
ETL power is constrained to keep the focal plane within the cuvette bounds.
The light sheet displacement or focal shift is calculated with respect to the
last cuvette surface.

### Experimental Setup

We modified an existing remote focus SPIM setup to capture light sheet images
in transmission [25, 26], as illustrated in Figure 3A. To validate the ETL
optimization model, we imaged beads embedded in agar using the optical pathway
in Figure 3B. A National Instruments DAQ (USB-6343) was used to synchronize
the ETL and illumination. Below, we detail the illumination pathway, followed
by the detection optics.

Figure 3: (A) Optical pathway used to image light sheets in transmission. (B)
Optical pathway used to characterize PSFs to validate ETL optimization model.
$L_{i}$ are achromat doublets, Ap. is the adjustable aperture used to modulate
the NA. $L_{cyl}$ is the cylindrical lens, TL is the tube lens. The dotted
line represents conjugate planes, red and blue indicating conjugate to the
$\text{BFP}_{EO}$ and $\text{FFP}_{EO}$ respectively.

Both excitation optical pathways utilized an excitation source (Oxxius L4Cc)
with four wavelengths (488, 523, 561, 638) combined into a single fiber
output, with 561 used for light sheet measurements and 488 used to image
beads. For all measurements, the fiber output was collimated using 250 focal
length achromatic lens (Thorlabs AC508-250-A-ML), the effective NA was
modulated using an adjustable rectangular slit (Ealing 74-1137-000) conjugate
to the $\text{BFP}_{EO}$, and an ETL (Optotune EL-16-40-TC) was utilized for
the remote focusing unit.

In the excitation optical pathway used for light sheet characterization, Fig.
3A, the beam was collimated and expanded by a factor of 4 before passing
through the aperture (Thorlabs AC508-250-A-ML, Thorlabs AC254-050-A-ML,
AC508-200-A-ML) and a truncated Gaussian line profile was generated in the
$\text{BFP}_{EO}$ by focusing a cylindrical lens (Thorlabs ACY254-200-A) on
the ETL and relayed to the $\text{BFP}_{EO}$ (Thorlabs AC508-150-A-ML,
AC508-100-A-ML) before being focused by a 0.3 NA air objective (Nikon
MRH00105).

The excitation optical pathway used for standard SPIM operation used a
cylindrical lens (ACY254-50-A) to generate a line profile conjugate to the
imaging plane. The line profile is refocused on the ETL and relayed to the BFP
of a 0.14 NA air objective (Thorlabs AC254-050-A-ML, AC508-180-A-ML,
AC508-150-A-ML, Mitutoyo 378-802-6).

For both measurements, the detection optics were mounted on an adjustable
motorized stage (Thorlabs KST101) to record z-stacks for light-sheet
characterization or perform auto focusing during SPIM operation. Light sheets
were imaged in transmission using a 0.5 NA long working distance objective
(Mitutoyo 378-805-3) and a 200 tube lens (Thorlabs TTL-200) focused on a
scientific CMOS camera (Photometrics Iris15). The effective pixel size of 85
was adequate to Nyquist sample all the light sheets considered here. For
imaging beads, the detection optics were replaced with a 0.5 NA 2x
magnification objective and 1x tube lens (Olympus MV PLAPO 2XC, Olympus MV
PLAPO 1X), forming an image on the same scientific CMOS camera (Teledyne
Photometrics Iris15). Fluorescence excitation was isolated using a 30 filter
wheel (Finger Lakes Instrumentation HS0433417) and barrier filter (Semrock
FF01-536/40-32-D).

The control computer for all experiments was a Dell Precision 3630 running
Microsoft Windows 10 Enterprise. This computer has a Core i7-9700K CPU (Intel)
with $8$ cores, 64 RAM, and a GeForce RTX 3060 GPU (Nvidia) with 12 of memory.
The navigate software package [27] was used to collect the static and ASLM
data shown in Fig. 1A-B and Fig. 5. Micromanager [28] was used to acquire
light sheets in transmission.

###  II.3 Light sheet profile acquisitions

We measured light sheets focusing in air and through an RI mismatch as
expected during normal imaging conditions. The RI mismatch was created using a
10 x 20 x 45 (22.5 path length) glass cuvette filled with water or $1\%$ agar.
For each, we modulated the effective NA by adjusting the long axis extent of
the laser elliptical focus in the objective back focal plane then acquired a
z-stack for a range of ETL powers. The position of the index mismatch is fixed
such that when the ETL is flat, the focus is centered in the cuvette. The
remote focus distance is measured relative to the flat ETL’s light sheet
position. After adjusting the aperture, the ETL current was reset to zero, and
the detection stage adjusted to center the light sheet. To acquire data, we
first moved the detection stage to align the FOV to the desired remote focus
distance, then adjusted the ETL power to recenter the light sheet focus. For
each NA, we captured z-stacks for remote focus positions ranging from ±4.0.
The remote focus range for the higher effective NAs was limited by
aberrations. The z-stack range was adjusted to observe out of focus features
of the light sheets.

The 3D raw data was processed to extract a light sheet “slice” which we could
compare to our simulation results. First, we identified the focal plane using
the peak intensity. The light sheet slice or ZX plane with the greatest
average intensity was chosen to compare with our model results. The resulting
2D slice image is rotated to align the propagation and $z$-axis, typically
less than 1. For light sheets imaged in water, stage displacements were scaled
by $1.33$ to reflect the image plane shift imposed by the index mismatch.

###  II.4 Light sheet profile analysis

We quantified both the simulated and experimental light sheet’s radial
confinement or width, $\omega(z)$, and propagation length, $z_{L}$. The width
is characterized by the full-width at half-maximum (FWHM) of the radial
intensity and the length is the range over which
$\omega(z)\leq\sqrt{2\ln{2}}\omega_{0}$.

To process raw images into light sheet parameters, We first interpolated the
axial intensity to identify the diffraction focal plane. Next, the focal plane
intensity was fit using a 3-Gaussian intensity model (eq. 13) to account for
side lobes and more complex beam structure caused by diffraction. The width,
$\omega_{0}$ was defined using the peak lobe width and reported using the
FWHM. Next, we numerically evaluated the FWHM, including side lobes, for each
plane. After calculating the width for each z-plane, we interpolated the
resulting distribution and defined the length as the range over which the
width increased by a factor of $\sqrt{2}$. In calculating the the light sheet
length, we found a numerical approach to define the radial extents (where
$\omega(z)\geq\sqrt{2}\omega_{0}$) was more consistent for comparing Gaussian
and non-Gaussian light sheets than fitting using a Gaussian function.

$I(r)=I_{\text{offset}}+\sum_{i=0}^{i=2}I_{i}\exp{\left[-2\frac{(r-\mu_{i})^{2}}{\omega_{i}^{2}}\right]}$
(13)

### Bead imaging to characterize light sheet quality

We prepared 1μ beads embedded in 1 agar (FluoSpheres F8823). Using the
standard SPIM setup, we performed imaging experiments at cuvette offsets
positions from 10 to 35, every 5. For each cuvette position, the detection
stage was translated to focus ∼1mm from the leading surface, in the middle of
the cuvette and ∼1 from the back surface. We captured 80μ z-stacks at each
position. To quantify our results, we localized and fit the beads to a
Gaussian model [29] and then applied a median and mean filter to the resulting
axial standard deviation distribution along the propagation axis. To estimate
the propagation length of the light sheet, we numerically identified the FWHM
of the dip about the focus.

## Results

### Comparison of simulated and experimental light sheets

The model results (dashed lines) depicted in Figure 4 represent light sheets
with zero shift applied are in agreement with prior results [9, 8]. The model
results in water suggest an index mismatch is a limiting factor in obtaining
tighter light sheets, but they also exhibit longer axial profiles. A
comparison between the model results in air and in water supports prior
theoretical work, which found that the spherical aberration induced by an
index mismatch extends the propagation length compared to a Gaussian focus of
equal width [8, 12].

Figure 4: (A) Light sheet widths and lengths for simulated and experimentally
light sheets. Visual comparison of various simulated (B) and experimental (C)
light sheets. The first two columns are moderate and low NA examples focusing
in air with no offset. The last column is an example of displaced light
sheets. The focal plane (dashed) and light sheet edges (solid) are highlighted
in white.

The experimental results with no shift applied closely follow the model
expectations from eq. 8 (Figure 4). Our experimental results are consistent
with previous observations of Gaussian beam or “sinc” light sheets that
exhibit longer propagation lengths for a given $\omega_{0}$ [9, 10]. Focusing
in water, the tightest attainable light sheet is limited by aberrations. Both
the model and experimental results show that focusing through an index
mismatch is a limiting factor in achieving quality light sheets. In this
initial experiment, we did not fine tuning the relative spacing of the
excitation optics and cuvette for the RI mismatch case.

The visual comparison between the model and experimental data emphasizes our
ability to accurately capture features of the light sheets (Fig. 4B,C). While
the model replicates focal features such as side lobes effectively, as
aberrations increase, we observe small deviations for out-of-focus features.
We suspected this discrepancy arises because these features are highly
sensitive to the precise phase profile of the beam or to the objective lens
aberrations, which we are not modeling. A visual comparison between
experimental and acquisition light sheets with applied focus shows that we can
capture the aberration characteristics associated with correcting first order
defocus with an ETL.

### Remote Focus Optimization for Heterogeneous Refractive Index

Using an objective lens designed for a different RI will lead to aberrations
and impact both the excitation profile and detected image quality. In
addition, many large sample SPIM experiments have a RI mismatch between the
optics, sample immersion media, and sample itself leading to a displacement of
the focus. Traditionally, the approach to addressing this issue involves
holding the sample chamber constant and displacing the light sheet (either
physically or using remote focusing) until it is within the sample. Leveraging
our model, we explored optimal configurations for remote focusing in such
environments.

We find for a given imaging configuration, there are up to two optimal light
sheet positions where the light sheet aberrations are minimized. This occurs
because the ETL induces an optical path length difference ($\Delta\text{OPL}$)
that counterbalances the path length difference accumulated through the
remaining optics. The RI mismatch imposes an additional $\Delta\text{OPL}$,
degrading focus quality and shifting the focal plane further from the
objective ($\Delta\text{OPL}\propto\sqrt{\left(\frac{r}{d}\right)^{2}+1}-1$,
where $d$ is the distance propagated in the medium and $r$ is the distance
from the optical axis). Consequently, adjustments in the ETL lens curvature
are required to maintain the focal position within the FOV. Using the ETL
remote focus model, we predicted the optimal spacing between the objective and
the cuvette to minimize the beam waist ($\omega_{0}$) and maintain these
values when shifting the light sheet via remote focusing (Figure 5).

Figure 5: (A) The cuvette offset, $dc$, is the distance between objective and
RI mismatch. The light sheet displacement, $df$, is calculated with respect to
the cuvette surface. (B) Axial aberration results for all possible
combinations of $dc$ and $df$. Stars represent approximate position of
experimental data. (D) Experimental static light sheet PSF results with z
standard deviation, $\sigma_{z}(y)$ and focus extent (black lines). Each row
represents a different cuvette position and each column corresponds to the
same light sheet displacement.

We evaluate our predictions by imaging and quantifying the point spread
function (PSF) across the camera FOV, using diffraction limited fluorescent
micro-spheres in agar, for various light sheet shifts and objective to cuvette
spacing. The PSF $\sigma_{z}$ fit results along the propagation direction
reflect the optical sectioning attributed to the light sheet’s focal region.
Quantifying the extent over which the PSF remains confined in the light sheet
propagation direction, we find that our model well predicts the optical
sectioning and propagation length in the presence of RI mismatch induced
aberrations. The optimal position for placing the cuvette to achieve a uniform
PSF $\sigma_{z}$ within the propagation length and maximizing the remote focus
range was determined to be between 25 and 30, or approximately 70% of the
objective focal length. Furthermore, we observed that when the cuvette is
positioned at either extreme—either too close or too far from the
objective—the quality of the light sheet improves with displacements away from
these extreme positions.

## Discussion

We have studied the effect of remote focusing and refractive index mismatch on
light sheet intensity distributions for large FOV SPIM. Through numerical
modeling and experimental validation, we confirmed that RI mismatches lead to
aberrated light sheet intensity profiles. However, some recovery is possible
by jointly optimizing the applied light sheet shift and the objective to
sample chamber spacing.

Ray tracing to account for the remote focus pathway aberrations enabled us to
estimate the remote focus range and determine the optimal RI mismatch position
to maximize optical sectioning. This approach ensures that the light sheet
focus is both optimized to enhance image contrast and correctly positioned
within the sample. By incorporating remote focus pathway aberrations in the
light sheet simulation, we demonstrated that considering both diffraction
effects and non-paraxial aberrations is crucial for accurately characterizing
light sheets in remote focus and RI mismatch applications.

Adaptive frameworks based on heuristic metrics or guide stars, which account
for RI mismatches and other image-degrading aberrations, significantly improve
obtainable image quality across multiple SPIM optical configurations [30, 31,
32, 25, 33, 34, 35, 36, 37, 38]. Our results suggest that simply starting with
an optimized imaging configuration will enhance image contrast across the FOV.
This finding extends to extended FOV methods, such as ASLM, where optimizing
both the RI mismatch position and the light sheet position will further
improve image quality, independent of additional corrective measures. Future
work could extend these methods to account for invariant illumination in
reconstructing light sheet acquisitions of large samples, where the cuvette
position and remote focus parameters should be updated to maintain the best
quality light sheet. Additionally, extending the model to the full image
formation framework would enable exploration of adaptive optics informed by
modeling to dynamically assist the correction of the illumination pathway
aberrations.

Our findings on the optimal configurations for remote focusing, especially in
the presence of RI mismatches, provide valuable guidelines for improving
imaging quality in SPIM. We hope this study offers a practical strategy for
enhancing SPIM performance across a range of biological samples and imaging
configurations.

## Funding

SJS, PTB, and DPS acknowledge funding from NIH NHLBI (R01HL068702).

## Disclosures

The authors declare no conflicts of interest.

## Data availability

Example datasets for the imaging the light sheet in transmission and bead
measurements are available [39]. All imaging data is available upon request.
Code to run analysis is available on our Github repository [40].

## References

  * Voie  _et al._ [1993] A. H. Voie, D. Burns, and F. Spelman, Orthogonal-plane fluorescence optical sectioning: Three-dimensional imaging of macroscopic biological specimens, Journal of microscopy 170, 229 (1993).
  * Huisken  _et al._ [2004] J. Huisken, J. Swoger, F. Del Bene, J. Wittbrodt, and E. H. Stelzer, Optical sectioning deep inside live embryos by selective plane illumination microscopy, Science 305, 1007 (2004).
  * Power and Huisken [2017] R. M. Power and J. Huisken, A guide to light-sheet fluorescence microscopy for multiscale imaging, Nature methods 14, 360 (2017).
  * McDole  _et al._ [2018] K. McDole, L. Guignard, F. Amat, A. Berger, G. Malandain, L. A. Royer, S. C. Turaga, K. Branson, and P. J. Keller, In toto imaging and reconstruction of post-implantation mouse development at the single-cell level, Cell 175, 859 (2018).
  * Ahrens  _et al._ [2013] M. B. Ahrens, M. B. Orger, D. N. Robson, J. M. Li, and P. J. Keller, Whole-brain functional imaging at cellular resolution using light-sheet microscopy, Nature methods 10, 413 (2013).
  * Kathe  _et al._ [2022] C. Kathe, M. A. Skinnider, T. H. Hutson, N. Regazzi, M. Gautier, R. Demesmaeker, S. Komi, S. Ceto, N. D. James, N. Cho, _et al._ , The neurons that restore walking after paralysis, Nature 611, 540 (2022).
  * Djutanta  _et al._ [2023] F. Djutanta, P. T. Brown, B. Nainggolan, A. Coullomb, S. Radhakrishnan, J. Sentosa, B. Yurke, R. F. Hariadi, and D. P. Shepherd, Decoding the hydrodynamic properties of microscale helical propellers from brownian fluctuations, Proceedings of the National Academy of Sciences 120, e2220033120 (2023).
  * Remacha  _et al._ [2020] E. Remacha, L. Friedrich, J. Vermot, and F. O. Fahrbach, How to define and optimize axial resolution in light-sheet microscopy: a simulation-based approach, Biomedical optics express 11, 8 (2020).
  * Chang  _et al._ [2020] B.-J. Chang, K. M. Dean, and R. Fiolka, Systematic and quantitative comparison of lattice and gaussian light-sheets, Optics express 28, 27052 (2020).
  * Liu  _et al._ [2023a] G. Liu, X. Ruan, D. E. Milkie, F. Görlitz, M. Mueller, W. Hercule, A. Killilea, E. Betzig, and S. Upadhyayula, Characterization, comparison, and optimization of lattice light sheets, Science advances 9, eade6623 (2023a).
  * Shi  _et al._ [2022] Y. Shi, T. A. Daugird, and W. R. Legant, A quantitative analysis of various patterns applied in lattice light sheet microscopy, Nature communications 13, 4607 (2022).
  * Deng  _et al._ [2021] S. Deng, Z. Ding, D. Yuan, M. Liu, and H. Zhou, Investigation of the extended focusing capability of the spherical aberration to enlarge the field of view in light-sheet fluorescence microscopy, JOSA A 38, 19 (2021).
  * Schacht  _et al._ [2010] P. Schacht, S. B. Johnson, and P. A. Santi, Implementation of a continuous scanning procedure and a line scan camera for thin-sheet laser imaging microscopy, Biomedical optics express 1, 598 (2010).
  * Gao [2015] L. Gao, Extend the field of view of selective plane illumination microscopy by tiling the excitation light sheet, Optics Express 23, 6102 (2015).
  * Botcherby  _et al._ [2008] E. J. Botcherby, R. Juškaitis, M. J. Booth, and T. Wilson, An optical technique for remote focusing in microscopy, Optics Communications 281, 880 (2008).
  * Chmielewski  _et al._ [2015] A. K. Chmielewski, A. Kyrsting, P. Mahou, M. T. Wayland, L. Muresan, J. F. Evers, and C. F. Kaminski, Fast imaging of live organisms with sculpted light sheets, Scientific reports 5, 9385 (2015).
  * Dean  _et al._ [2015] K. M. Dean, P. Roudot, E. S. Welf, G. Danuser, and R. Fiolka, Deconvolution-free subcellular imaging with axially swept light sheet microscopy, Biophysical journal 108, 2807 (2015).
  * Polyanskiy [2024] M. N. Polyanskiy, Refractiveindex. info database of optical constants, Scientific Data 11, 94 (2024).
  * Goodman [2005] J. W. Goodman, Computational diffraction and propagation, in  _Introduction to Fourier optics_ (Roberts and Company publishers, 2005) pp. 140–143, the Exact Transfer Function Approach.
  * Olarte  _et al._ [2018] O. E. Olarte, J. Andilla, E. J. Gualda, and P. Loza-Alvarez, Light-sheet microscopy: a tutorial, Advances in Optics and Photonics 10, 111 (2018).
  * Born and Wolf [1975a] M. Born and E. Wolf,  _Principles of optics: electromagnetic theory of propagation, interference and diffraction of light_ (Pergamon Press, 1975) section 9.1.2.
  * Robert R. Shannon [1992] J. C. W. Robert R. Shannon, Applied optics and optical engineering (Elsevier, 1992) Chap. Basic Wavefront Aberration Theory for Optical Metrology, pp. 28–29.
  * Born and Wolf [1975b] M. Born and E. Wolf,  _Principles of optics: electromagnetic theory of propagation, interference and diffraction of light_ (Pergamon Press, 1975) section 9.1.3.
  * Born and Wolf [1975c] M. Born and E. Wolf,  _Principles of optics: electromagnetic theory of propagation, interference and diffraction of light_ (Pergamon Press, 1975) section 5.3.
  * Ryan  _et al._ [2017] D. P. Ryan, E. A. Gould, G. J. Seedorf, O. Masihzadeh, S. H. Abman, S. Vijayaraghavan, W. B. Macklin, D. Restrepo, and D. P. Shepherd, Automatic and adaptive heterogeneous refractive index compensation for light-sheet microscopy, Nature communications 8, 612 (2017).
  * Singh  _et al._ [2017] J. N. Singh, T. M. Nowlin, G. J. Seedorf, S. H. Abman, and D. P. Shepherd, Quantifying three-dimensional rodent retina vascular development using optical tissue clearing and light-sheet microscopy, Journal of Biomedical Optics 22, 076011 (2017).
  * Marin  _et al._ [2024] Z. Marin, X. Wang, D. W. Collison, C. McFadden, J. Lin, H. Borges, B. Chen, D. Mehra, Q. Shen, S. Galecki, S. Daetwyler, R. Fiolka, and K. M. Dean, navigate: an open-source platform for smart light-sheet microscopy, bioRxiv 10.1101/2024.02.09.579083 (2024).
  * Edelstein  _et al._ [2014] A. D. Edelstein, M. A. Tsuchida, N. Amodaj, H. Pinkard, R. D. Vale, and N. Stuurman, Advanced methods of microscope control using $\mu$manager software, Journal of biological methods 1 (2014).
  * Brown  _et al._ [2023] P. T. Brown, S. J. Sheppard, A. Coullomb, and D. P. Shepherd, localize-psf (2023), https://github.com/QI2lab/localize-psf.
  * Bourgenot  _et al._ [2012] C. Bourgenot, C. D. Saunter, J. M. Taylor, J. M. Girkin, and G. D. Love, 3d adaptive optics in a light sheet microscope, Optics express 20, 13252 (2012).
  * Tomer  _et al._ [2014] R. Tomer, L. Ye, B. Hsueh, and K. Deisseroth, Advanced clarity for rapid and high-resolution imaging of intact tissues, Nature protocols 9, 1682 (2014).
  * Royer  _et al._ [2016] L. A. Royer, W. C. Lemon, R. K. Chhetri, Y. Wan, M. Coleman, E. W. Myers, and P. J. Keller, Adaptive light-sheet microscopy for long-term, high-resolution imaging in living organisms, Nature biotechnology 34, 1267 (2016).
  * Hubert  _et al._ [2019] A. Hubert, F. Harms, R. Juvénal, P. Treimany, X. Levecq, V. Loriette, G. Farkouh, F. Rouyer, and A. Fragola, Adaptive optics light-sheet microscopy based on direct wavefront sensing without any guide star, Optics Letters 44, 2514 (2019).
  * Chen  _et al._ [2020] Y. Chen, X. Li, D. Zhang, C. Wang, R. Feng, X. Li, Y. Wen, H. Xu, X. S. Zhang, X. Yang, _et al._ , A versatile tiling light sheet microscope for imaging of cleared tissues, Cell Reports 33 (2020).
  * Rai  _et al._ [2022] M. R. Rai, C. Li, and A. Greenbaum, Quantitative analysis of illumination and detection corrections in adaptive light sheet fluorescence microscopy, Biomedical Optics Express 13, 2960 (2022).
  * Li  _et al._ [2022] C. Li, M. R. Rai, H. T. Ghashghaei, and A. Greenbaum, Illumination angle correction during image acquisition in light-sheet fluorescence microscopy using deep learning, Biomedical Optics Express 13, 888 (2022).
  * Rai  _et al._ [2023] M. R. Rai, C. Li, H. T. Ghashghaei, and A. Greenbaum, Deep learning-based adaptive optics for light sheet fluorescence microscopy, Biomedical Optics Express 14, 2905 (2023).
  * Liu  _et al._ [2023b] Y. Liu, B. Liu, J. Green, C. Duffy, M. Song, J. D. Lauderdale, and P. Kner, Volumetric light sheet imaging with adaptive optics correction, Biomedical Optics Express 14, 1757 (2023b).
  * Sheppard [2024] S. J. Sheppard, Example datasets associated with ”tools for model driven extended field of view in1 selective plane illumination microscopy” (2024), https://doi.org/10.5281/zenodo.12775387.
  * Sheppard  _et al._ [2024] S. J. Sheppard, P. T. Brown, and D. P. Shepherd, Code associated with ”tools for model driven extended field of view in1 selective plane illumination microscopy” (2024), https://doi.org/10.5281/zenodo.12752313.


#  Revisiting Supervision for
Continual Representation Learning

Daniel Marczak∗1,2, Sebastian Cygert1,3, Tomasz Trzciński1,2,4, Bartłomiej
Twardowski1,5,6
1IDEAS NCBR; 2Warsaw University of Technology; 3Gdańsk University of
Technology;
4Tooploox; 5Autonomous University of Barcelona; 6Computer Vision Center
Corresponding author, email<EMAIL_ADDRESS>
###### Abstract

In the field of continual learning, models are designed to learn tasks one
after the other. While most research has centered on supervised continual
learning, recent studies have highlighted the strengths of self-supervised
continual representation learning. The improved transferability of
representations built with self-supervised methods is often associated with
the role played by the multi-layer perceptron projector. In this work, we
depart from this observation and reexamine the role of supervision in
continual representation learning. We reckon that additional information, such
as human annotations, should not deteriorate the quality of representations.
Our findings show that supervised models when enhanced with a multi-layer
perceptron head, can outperform self-supervised models in continual
representation learning.

##  1 Introduction

Figure 1: In a two-task continual learning scenario, supervised learning (SL)
results in representations that perform well on the second task but poorly on
the first task. On the other hand, representations trained with self-
supervised learning (SSL) have higher first-task performance but they
underperform on the second task. We show that simple modifications to
supervised learning (SL+MLP) yield representations that are superior on the
first task and on par with SL on the second task. We report average over 6
different scenarios.

In continual learning (CL), the goal of the model is to learn new tasks
sequentially. A number of recent works study continual learning from a
representation learning perspective and show that unsupervised approaches
build more robust representations when trained continually: Madaan et al.
(2022) shows that self-supervised learning (SSL) methods build representations
that are more robust to forgetting than supervised learning (SL) while Davari
et al. (2022) notices that training SimCLR (Chen et al., 2020) have
advantageous properties for continual learning compared to SL. However, it is
still counter-intuitive that access to more information (labels) results in
worse representations in continual learning.

One of the potential reasons is the transferability gap between supervised and
unsupervised learning. It was believed that the superior transferability of
unsupervised learning can be attributed to a special design of contrastive
loss (Zhao et al., 2020) or lack of annotations during training (Ericsson et
al., 2020). However, recent works (Wang et al., 2021; Sariyildiz et al., 2023)
identify that a multi-layer perceptron (MLP) projector commonly used in SSL
(Chen & He, 2020; Grill et al., 2020) is a crucial component that improves
transferability of SSL models. Following that founding Wang et al. (2021);
Sariyildiz et al. (2023) use an MLP projector to improve transferability of SL
and achieve state-of-the-art transfer learning performance, surpassing
unsupervised methods.

In this work, encouraged by these advancements in improving the
transferability of supervised models, we revisit supervision for continual
representation learning. We argue that additional information (human
annotations) should not hurt the quality of representations in continual
learning, as suggested by Madaan et al. (2022). Motivated by the latest study
on transferability of representations in self-supervised and supervised
learning, we aim to improve transferability between tasks in continual
learning. We are the first to show that supervised models can continually
learn representations of higher quality than self-supervised models when
trained with a simple MLP head (see Fig. 1).

The main contributions of this paper are as follows. (1) We empirically show
that SL equipped with a simple MLP projector can learn higher-quality
representations than SSL methods in continual finetuning scenarios. (2) We
show that the use of the MLP projector can be coupled with several continual
learning methods, further improving their performance. (3) We shed light on
the reasons behind the strong performance of supervised learning with MLP
projector: better transferability, lower forgetting, and increasing diversity
of representations.

##  2 Related Work

Self-supervised learning In this work, we use BarlowTwins (Zbontar et al.,
2021) (denoted as SSL) which considers an objective function measuring the
cross-correlation matrix between the features and SimCLR (Chen et al., 2020)
which uses contrastive learning based on noise-contrastive estimation. A
number of studies (Bordes et al., 2023; Chen & He, 2020; Zbontar et al., 2021;
Jing et al., 2022) show that an MLP projector between the encoder and the loss
function is a crucial component to prevent the collapse of the representations
and improve their transferability. Transferable representations Wang et al.
(2021) found out that adding a projection network boosts the transferability
of the supervised models’ features as well. This was further explored in
Sariyildiz et al. (2023) and it was shown that it is possible to build
representations that are good for both the source and the downstream tasks. In
this work, we revisit those findings in the context of models learned on a
sequence of tasks.

Supervised Continual Learning (SCL) aims to create systems that can learn to
solve novel tasks using new annotated data while retaining the ability to
solve previously learned tasks (Parisi et al., 2019). Unsupervised Continual
Learning (UCL) aims to improve the quality of learnt represetntations
utilizing an ever-changing stream of unlabeled data. Recent works (Fini et
al., 2022; Madaan et al., 2022; Gomez-Villa et al., 2021) apply SSL in the UCL
setting and claim their superior results for continual representation
learning.

##  3 Experimental Setup

Datasets We utilize four different datasets: CIFAR10 (Krizhevsky, 2009) (C10),
CIFAR100 (Krizhevsky, 2009) (C100), SVHN (Netzer et al., 2011) and ImageNet100
(Tian et al., 2019) (IN100). We use $D/N$ to denote that dataset $D$ is split
into $N$ tasks with an equal number of classes in each task without
overlapping ones. We use $A\xrightarrow{}B$ to denote task shift meaning that
the model was trained on two tasks, the first one was dataset $A$ and the
second one was dataset $B$.

Methods We use the following supervised methods: (1) SL - backbone with linear
head with a cross-entropy loss function, (2) SL+MLP - SL with MLP projector
added between the backbone and a linear head that is discarded at test-time,
(3) t-ReX (Sariyildiz et al., 2023), and (4) SupCon (Khosla et al., 2020). For
CL strategies we use LwF (Li & Hoiem, 2018), CaSSLe (Fini et al., 2022) and
PRF (Gomez-Villa et al., 2021). We use ResNet-18 (He et al., 2016) as a
feature extractor.

Evaluation We use k-NN classifier to evaluate the quality of representations
following Fini et al. (2022); Madaan et al. (2022) and Nearest Mean Classifier
(NMC) as in Rebuffi et al. (2017); Yu et al. (2020) to evaluate the stability
of representations. We use CKA (Kornblith et al., 2019) to measure the
similarity between representations of two models. Moreover, we use forgetting
($F$) and forward transfer ($FT$) commonly used in continual learning (Lopez-
Paz & Ranzato, 2017). We also measure task exclusion difference $EXC$ (Hess et
al., 2023) to evaluate the level of retention of task-specific features. We
use subscripts to indicate the evaluation dataset, e.g. $Acc_{C10}$ means
"accuracy on C10".

##  4 Experimental results

Fig. 2 presents our main results showing that supervised models can build
stronger representations than self-supervised models under continual
finetuning, contrary to previous beliefs (Madaan et al., 2022). We recognize
that the key component to enhancing the performance of supervised models is an
additional MLP projector during training and discarded afterward. We identify
two factors contributing to superior results of SL+MLP. Firstly, we observe
that the performance of supervised models after the initial task is largely
improved by the addition of the MLP projector, resulting in accuracy close to
SSL models. In order to achieve good task-agnostic accuracy on the whole
dataset (seen and unseen classes), the model trained on a single task needs to
perform well on unseen data. Therefore, we attribute the advantage of SL+MLP
to the increased transferability of representations induced by MLP projector.
Secondly, we notice that SL+MLP is the only method able to incrementally
accumulate knowledge and consistently improve performance.

Figure 2:  SL+MLP: (1) achieves strong performance after the initial task
compared to SL which indicates that it produces representations that are
transferable to the unseen tasks; (2) is the only method that is able to
accumulate knowledge learned on a sequence of tasks. We report task-agnostic
k-NN accuracy on the whole dataset (including unseen tasks).

Method | CL strategy | C100/5 | C100/20 | IN100/5
---|---|---|---|---
Supervised Continual Learning
SL | Finetune | 38.5$\pm$0.4 | 17.2$\pm$0.3 | 35.3$\pm$1.3
| LWF | 57.4$\pm$0.2 | 45.2$\pm$1.2 | 60.5$\pm$0.3
| PFR | 57.7$\pm$0.4 | 44.4$\pm$1.3 | 58.7$\pm$0.2
SL+MLP | Finetune | 61.9$\pm$0.5 | 47.1$\pm$0.7 | 62.4$\pm$0.4
| LWF | 58.7$\pm$0.2 | 51.9$\pm$0.1 | 60.4$\pm$0.2
| PFR | 63.6$\pm$0.2 | 54.5$\pm$0.2 | 65.2$\pm$0.1
t-ReX | Finetune | 59.2$\pm$0.6 | 50.8$\pm$0.1 | 59.2$\pm$0.6
| LwF | 58.3$\pm$0.4 | 50.4$\pm$0.1 | 58.6$\pm$1.0
| PFR | 60.9$\pm$0.5 | 53.4$\pm$0.3 | 63.9$\pm$0.6
SupCon | Finetune | 49.4$\pm$0.3 | 30.0$\pm$0.7 | 57.6$\pm$0.6
| CaSSLe | 61.1$\pm$0.2 | 49.2$\pm$1.2 | 70.4$\pm$0.6
| PFR | 57.0$\pm$0.2 | 51.2$\pm$0.8 | 68.0$\pm$0.7
Unsupervised Continual Learning
BarlowTwins | Finetune | 54.1$\pm$0.3 | 40.0$\pm$0.8 | 57.0$\pm$0.4
| CaSSLe | 58.6$\pm$0.6 | 49.3$\pm$0.1 | 64.9$\pm$0.1
| PFR | 57.2$\pm$0.2 | 46.0$\pm$0.7 | 61.1$\pm$0.2
SimCLR | Finetune | 48.9$\pm$0.4 | 33.4$\pm$0.5 | 54.7$\pm$0.4
| CaSSLe | 55.9$\pm$0.5 | 48.2$\pm$0.4 | 59.3$\pm$0.5
| PFR | 53.8$\pm$0.3 | 49.4$\pm$0.1 | 57.7$\pm$0.2

Table 1: k-NN accuracy of the learnt representations. The best result in bold
and second best underlined.

Tab. 1 presents extended results including multiple SL and SSL approaches in
continual finetuning and paired with different CL methods. Firstly, we observe
that all the supervised methods equipped with the projector (SL+MLP, t-ReX,
and SupCon) significantly outperform simple SL. What is worth noting is the
fact that all these methods were trained with different supervised losses:
SL+MLP uses cross-entropy, t-ReX uses cosine softmax cross-entropy and SupCon
uses supervised contrastive loss. Secondly, we observe that the positive
effects of the MLP projector and CL strategy compound. As a result, the best
models are those (1) trained in a supervised way (2) with the use of the MLP
projector and (3) coupled with CL strategy based on temporal learnable
projection, namely CaSSLe or PFR.

We investigate the quality of representations built by supervised and self-
supervised training in Fig. 3. SL+MLP outperforms both SSL and SL in most of
the experiments. In Tab. 2 we observe high representation forgetting for SL,
significantly lower for SSL, and the lowest for SL equipped with MLP
projector. It also shows the results of task exclusion comparison ($EXC$). SL
achieves small positive $EXC$ meaning that it forgets most features specific
to the initial task. SL+MLP achieves the highest $EXC$ which shows that it is
able to successfully retain a large portion of task-specific features.
Surprisingly, SSL exhibits negative $EXC$. It means that it is better to train
SSL model from scratch on another task than to finetune the model pretrained
on the task of interest. In Tab. 2 we report CKA similarity between the models
trained on C10 and the rest of the models. We observe that usage of MLP head
in SL increases CKA between the C10 model and other models. Moreover, the
models pretrained on C10 and finetuned on another task have higher similarity
to C10 models than the models trained on another dataset from scratch, which
is not necessarily the case for SL models. SSL models have the highest CKA
scores, however, they usually underperform compared to SL+MLP suggesting that
SSL produces similar but less discriminative features.

Training | SL | SSL | SL+MLP
---|---|---|---
sequence | $Acc_{C10}\uparrow$ | $F_{C10}\downarrow$ | $EXC_{C10}\uparrow$ | $\text{CKA}_{C10}\uparrow$ | $Acc_{C10}\uparrow$ | $F_{C10}\downarrow$ | $EXC_{C10}\uparrow$ | $\text{CKA}_{C10}\uparrow$ | $Acc_{C10}\uparrow$ | $F_{C10}\downarrow$ | $EXC_{C10}\uparrow$ | $\text{CKA}_{C10}\uparrow$
C10 | 92.6$\pm$0.1 | - | - | - | 88.8$\pm$0.1 | - | - | - | 93.3$\pm$0.1 | - | - | -
C100 | 74.9$\pm$0.2 | - | - | 0.46$\pm$0.00 | 80.8$\pm$0.1 | - | - | 0.56$\pm$0.01 | 84.5$\pm$0.4 | - | - | 0.49$\pm$0.01
C10$\xrightarrow{}$C100 | 76.1$\pm$0.1 | 16.6$\pm$0.2 | 1.2$\pm$0.3 | 0.50$\pm$0.00 | 79.1$\pm$0.2 | 9.7$\pm$0.3 | -1.8$\pm$0.2 | 0.52$\pm$0.01 | 88.8$\pm$0.2 | 4.5$\pm$0.3 | 4.3$\pm$0.6 | 0.57$\pm$0.00
SVHN | 21.8$\pm$0.3 | - | - | 0.05$\pm$0.00 | 58.6$\pm$1.2 | - | - | 0.27$\pm$0.01 | 56.3$\pm$0.2 | - | - | 0.20$\pm$0.01
C10$\xrightarrow{}$SVHN | 22.6$\pm$0.5 | 70.1$\pm$0.5 | 0.8$\pm$0.4 | 0.05$\pm$0.01 | 54.9$\pm$0.7 | 33.8$\pm$0.7 | -3.7$\pm$1.9 | 0.25$\pm$0.01 | 62.7$\pm$0.8 | 30.6$\pm$0.8 | 6.4$\pm$1.0 | 0.25$\pm$0.01

Table 2:  We observe high representation forgetting for SL, significantly
lower for SSL, and the lowest for SL+MLP. SL is able to preserve a small
fraction of task-specific features while SL+MLP can retain much more, based on
their $EXC$ scores. Surprisingly, SSL achieves negative $EXC$

We present the results of the forward transfer evaluation in Tab. 3. All the
methods benefit from pretraining on CIFAR100 which is semantically close to
CIFAR10. However, pretraining on semantically distant SVHN hinders the
performance of SL but it hardly influences the performance of SSL and SL+MLP.

Figure 3: SL+MLP (blue) achieves better representations than SL (red) and SSL
(green) in most sequences of tasks.

Method | C10 | C100$\xrightarrow{}$C10 | SVHN$\xrightarrow{}$C10
---|---|---|---
$Acc_{C10}\uparrow$ | $Acc_{C10}\uparrow$ | $FT_{C10}\uparrow$ | $Acc_{C10}\uparrow$ | $FT_{C10}\uparrow$
SL | 92.6$\pm$0.1 | 94.0$\pm$0.2 | 1.3$\pm$0.3 | 91.5$\pm$0.2 | -1.1$\pm$0.3
SSL | 88.8$\pm$0.1 | 89.2$\pm$0.1 | 0.5$\pm$0.2 | 88.5$\pm$0.1 | -0.3$\pm$0.2
SL+MLP | 93.3$\pm$0.1 | 94.3$\pm$0.1 | 1.0$\pm$0.0 | 93.2$\pm$0.2 | -0.1$\pm$0.1

Table 3:  All methods benefit prom pretraining on C100 which is semantically
close to C10. However, pretraining on semantically distant SVHN hinders the
performance of SL.

Figure 4:  Representations learned with SL+MLP (right) exhibit desirable
properties from the continual learning point of view: (1) they consist of a
more diverse set of features (contrary to SL, left); (2) they improve feature
diversity when learning new tasks consistently across all the presented
settings. Vertical dashed lines denote 95% of the variance explained.

Figure 5: Task aware NMC accuracy on C10. After training on C10 (T1), both SL
and SSL models achieve high NMC performance (yellow). After training the
second task (T2), the nearest mean classification using old prototypes results
in performance degradation (green). We calculate _an upper-bound_ accuracy
after training on the second task by recalculating the prototypes using old
data and a new backbone (purple). Note that it is not possible in SCL as old
data is inaccessible. Gray dotted line marks random guess performance.

To gain further insight into the properties of continually trained
representations, we analyze the spectra of representations following the
procedure from Jing et al. (2022). We perform singular value decomposition of
the covariance matrix of the representations $C=USV^{T}$, where
$S=diag(\sigma^{k})$ and $\sigma^{k}$ is $k$-th singular value of $C$. Fig. 5
presents how singular value spectra change after each task for different
training methods and different sequences of tasks. Firstly, we observe that SL
exhibits signs of neural collapse (Papyan et al., 2020) \- a large fraction of
variance is described by a few dimensions roughly equal to the number of
classes in the training set. This is an undesirable property in continual
representation learning as the representations should be more versatile and
useful not only for current but also for past and future tasks. Adding MLP to
SL prevents neural collapse and results in features’ properties more similar
to SSL. Secondly, we observe that for SL, the diversity of the features
decreases in subsequent tasks. SSL and SL+MLP are able to consistently improve
the diversity of the representations suggesting its superiority in continual
representation learning.

We define representations as stable when they do not drift in the
representation space when the network is trained on a new task. The stability
provides a different perspective when evaluating continually trained
representations. It is irrelevant when the only objective is to continually
learn representations, however, it is a desired property when we also want to
solve downstream tasks continually (Yu et al., 2020). The results are
presented in Fig. 5. Representations of all the methods are not stable in high
distribution shift scenario C10$\xrightarrow{}$SVHN. However, in a low
distribution shift scenario, C10$\xrightarrow{}$C100, SL exhibits high
stability while SL+MLP underperforms in that regard. Note that performance
degradation can be only partially attributed to forgetting of representations
as the upper-bound performance is still high after training on the second task
for most of the methods. These results suggest that there exists a trade-off
between the stability and expressiveness of representations trained
continually as methods that build stronger representations tend to have lower
stability.

## Acknowledgments and Disclosure of Funding

Daniel Marczak is supported by National Centre of Science (NCN, Poland) Grant
No. 2021/43/O/ST6/02482. This research was partially funded by National
Science Centre, Poland, grant no 2020/39/B/ST6/01511 and grant no
2022/45/B/ST6/02817. Bartłomiej Twardowski acknowledges the grant
RYC2021-032765-I.

## References

  * Bordes et al. (2023) Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, and Pascal Vincent.  Guillotine regularization: Why removing layers is needed to improve generalization in self-supervised learning.  _Transactions on Machine Learning Research_ , 2023.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton.  A simple framework for contrastive learning of visual representations.  In _International Conference on Machine Learning (ICML)_ , 2020.
  * Chen & He (2020) Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  _Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Davari et al. (2022) Mohammad Reza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky.  Probing representation forgetting in supervised and unsupervised continual learning.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Ericsson et al. (2020) Linus Ericsson, H. Gouk, and Timothy M. Hospedales.  How well do self-supervised models transfer?  _Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Fini et al. (2022) Enrico Fini, Victor G. Turrisi da Costa, Xavier Alameda-Pineda, Elisa Ricci, Karteek Alahari, and Julien Mairal.  Self-supervised models are continual learners.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Gomez-Villa et al. (2021) Alex Gomez-Villa, Bartlomiej Twardowski, Lu Yu, Andrew D. Bagdanov, and Joost van de Weijer.  Continually learning self-supervised representations with projected functional regularization.  _Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2021.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altch’e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, B. ’. Pires, Z. Guo, M. G. Azar, Bilal Piot, K. Kavukcuoglu, R. Munos, and Michal Valko.  Bootstrap your own latent: A new approach to self-supervised learning.  _Neural Information Processing Systems (NeurIPS)_ , 2020.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Hess et al. (2023) Timm Hess, Eli Verwimp, Gido M. van de Ven, and Tinne Tuytelaars.  Knowledge accumulation in continually learned representations and the issue of feature forgetting.  _arXiv preprint arXiv: 2304.00933_ , 2023.
  * Jing et al. (2022) Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian.  Understanding dimensional collapse in contrastive self-supervised learning.  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  _Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Kornblith et al. (2019) Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton.  Similarity of neural network representations revisited.  _International Conference on Machine Learning (ICML)_ , 2019.
  * Krizhevsky (2009) Alex Krizhevsky.  Learning multiple layers of features from tiny images.  _University of Toronto_ , 2009.
  * Li & Hoiem (2018) Zhizhong Li and Derek Hoiem.  Learning without forgetting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 2018.
  * Lopez-Paz & Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Madaan et al. (2022) Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang.  Representational continuity for unsupervised continual learning.  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Netzer et al. (2011) Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * Papyan et al. (2020) Vardan Papyan, X. Y. Han, and David L. Donoho.  Prevalence of neural collapse during the terminal phase of deep learning training.  _Proceedings of the National Academy of Sciences_ , 2020.
  * Parisi et al. (2019) German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  _Neural networks_ , 2019.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert.  icarl: Incremental classifier and representation learning.  In _Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Sariyildiz et al. (2023) Mert Bulent Sariyildiz, Yannis Kalantidis, Karteek Alahari, and Diane Larlus.  No reason for no supervision: Improved generalization in supervised models.  In _International Conference on Learning Representations (ICLR)_ , 2023.
  * Tian et al. (2019) Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  _European Conference On Computer Vision (ECCV)_ , 2019.
  * Wang et al. (2021) Yizhou Wang, Shixiang Tang, Feng Zhu, Lei Bai, Rui Zhao, Donglian Qi, and Wanli Ouyang.  Revisiting the transferability of supervised pretraining: an mlp perspective.  _Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Yu et al. (2020) Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer.  Semantic drift compensation for class-incremental learning.  In _Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Zbontar et al. (2021) Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  _International Conference on Machine Learning (ICML)_ , 2021.
  * Zhao et al. (2020) Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin.  What makes instance discrimination good for transfer learning?  _International Conference on Learning Representations_ , 2020.