# Snapshot High Dynamic Range Imaging with a Polarization Camera

Mingyang Xie*, Matthew Chan*, Christopher Metzler * Equal contribution The
authors are with the Department of Computer Science, University of Maryland,
College Park, MD, 20742 (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>metzler@umd.edu).

###### Abstract

High dynamic range (HDR) images are important for a range of tasks, from
navigation to consumer photography. Accordingly, a host of specialized HDR
sensors have been developed, the most successful of which are based on
capturing variable per-pixel exposures. In essence, these methods capture an
entire exposure bracket sequence at once in a single shot. This paper presents
a straightforward but highly effective approach for turning an off-the-shelf
polarization camera into a high-performance HDR camera. By placing a linear
polarizer in front of the polarization camera, we are able to simultaneously
capture four images with varied exposures, which are determined by the
orientation of the polarizer. We develop an outlier-robust and self-
calibrating algorithm to reconstruct an HDR image (at a single polarity) from
these measurements. Finally, we demonstrate the efficacy of our approach with
extensive real-world experiments.

###### Index Terms:

computational photography, polarization, high dynamic range imaging

##  I Introduction

The dynamic range of digital cameras is constrained to a limited range—far
narrower than the human eye. As a result, when everyday scenes are captured by
a standard low-dynamic-range (LDR) camera, a large number of pixels will be
over-exposed or under-exposed and information will be lost. The most common
approach to record HDR scenes is to capture multiple images with varying
exposures [1, 2, 3]. However, such multi-shot HDR techniques are susceptible
to motion-induced ghosting artifacts.

Over the last two decades, a plethora of single-shot HDR imaging techniques
have been developed to avoid this problem. One can optically split an image so
as to capture multiple exposures on multiple sensors simultaneously [4, 5],
one can perform glare-encoding to recover bright highlights [6, 7, 8], one can
use exotic modulo sensors and solve a phase unwrapping problem to recover an
HDR scene [9, 10, 11], and one can also simply capture an over-exposed LDR
image and then use a CNN to hallucinate the missing details [12].

Arguably the most successful approach to date for single-shot HDR imaging is
to apply per-pixel exposures [13]. This approach is simpler and more robust
than optical encoding based methods and, unlike modulo sensors, does not
suffer from any brightness ambiguities at edges/discontinuities in an image.

Sony recently commercialized this technology and now sells dedicated HDR image
sensors, such as the IMX490. In this work we demonstrate how one can achieve
similar functionality using an off-the-shelf polarization camera.

Figure 1: Single-shot HDR reconstruction using a polarization camera. Our
method combines a polarization camera and linear polarizing filter (top left)
to capture 4 multi-exposure images simultaneously (top right) and reconstruct
HDR images (bottom).

###  I-A Our Contribution

We demonstrate how an off-the-shelf polarization camera can be (reversibly)
repurposed to perform snapshot HDR imaging. Conceptually, our hardware setup
works similarly to Sony’s HDR sensors—albeit with an adjustable dynamic range.

This functionality is achieved by placing a linear polarizer in front of the
camera’s lens (see Figure 1). Combining this polarizer with the polarization
camera’s Bayer-like polarization grid, we effectively capture four
exposures—one for each of the four polarizer angles
($0^{\circ},45^{\circ},90^{\circ},135^{\circ}$)—in a single-shot. Under this
construction, the residual between angle of the linear polarizer and the angle
of the polarization array determines each image’s relative exposure [14].
Images captured by approximately orthogonal polarization angles (e.g. when the
linear polarizer is rotated to $90^{\circ}$ and we read out the $0^{\circ}$
polarized image) result in images with an extremely low exposure (see Figure
3).

This paper makes the following contributions:

  * •

We develop a novel methodology for capturing high dynamic range information
with minimal hardware modification to an off-the-shelf polarization camera.

  * •

We introduce an HDR reconstruction algorithm that is robust to outliers and
light leakage caused by imperfect polarizers.

  * •

We analyze the effects that imperfect polarizers and highly polarized scenes
have on our reconstructions.

  * •

We conduct extensive experiments demonstrating that our method significantly
outperforms state-of-the-art software-only HDR reconstruction algorithms, as
well as existing polarization-camera based HDR imaging techniques [15].

##  II Related Work

###  II-A Multi-frame HDR Reconstruction

Exposure bracketing based HDR reconstruction methods [16, 17, 18] fuse well-
exposed pixels from images captured at different exposure stops to produce an
HDR image.

While effective for static scenes, these techniques are susceptible to
ghosting artifacts caused by motion blur and camera shake. Imaging in low-
light further exacerbates this issue as the shutter must remain open for
longer intervals of time to properly expose pixels.

As part of their work on the Google Pixel, Hasinoff et al. proposed HDR+ [19],
which relies on a burst of short-exposure images instead of a traditional
exposure bracket. While under-exposed regions of these images are influenced
by photon shot noise, this effect is mitigated by averaging intensities across
captures. The primary benefit of this method is that it’s inherently less
affected by ghosting artifacts since the shutter is open for a shorter
duration. Still, the HDR ratio achievable by this method is capped due to the
limited dynamic range of captured exposure stops.

Removing the need for exposure calibration altogether, Exposure Fusion [3]
fuses LDR images according to heuristics such as contrast, saturation, and
well-exposedness. We note, however, that while this method produces images
with an HDR-like appearance, the resulting images do not accurately reflect
the scene’s true intensity.

###  II-B Software-based Snapshot HDR Reconstruction

Recovering an HDR image from a single LDR image is an under-constrained
problem. Over-exposed pixels contain very little information and under-exposed
pixels are overwhelmed by noise. Deep learning based methods [12, 20] address
this problem by using learned priors. However, while these methods excel at
hallucinating plausible information in poorly exposed regions, they perform
poorly on out-of-distribution data.

###  II-C Hardware-based Snapshot HDR Reconstruction

Addressing the challenge of snapshot HDR reconstruction from a different
perspective, some methods leverage exotic imaging hardware to encode HDR
information form a single capture. Nayar et al. [13] proposed a split-pixel
sensor design capable of capturing four different stops of exposure
simultaneously. So et al. [11] and Zhao et al. [21] prevent clipping by
capturing HDR scenes with a modulo camera and phase unwrapping the result.
Typically, methods of this nature generalize well across scenes, but there are
some caveats. For instance, So et al.’s MantissaCam struggles to recover
regions with sharp discontinuities, such as windows.

More recently, Zhou et al. used a polarization camera to perform HDR imaging
[15]. Unlike our method, their approach does not use an additional polarizing
filter in front of the lens. Even without an added filter, each of the four
polarizers in the camera attenuates incoming light in a slightly different
way. Using this intensity information and leveraging polarity information,
Zhou et al. train their Pol-HDR network to recover HDR images from a snapshots
of four polarized LDR images. The lack of an additional polarizing filter
drastically reduces the range of exposures covered, resulting in lower dynamic
range reconstructions compared to our method.

Figure 2: Polarization and color encoding in polarization cameras.
Polarization cameras contain a stacked filter grid to encode information about
both polarity and color in each pixel. The top-level RGGB Bayer grid encodes
color information while the underlying polarizer grid encodes polarity.

##  III Proposed Method

###  III-A Hardware

Polarizers filter incident light such that light waves of a specific polarity
are transmitted while orthogonally polarized light waves are obstructed.
Polarization cameras use arrays of polarization filters and color filters to
encode the polarity and color of each capture pixel.

Specifically, each $4\times 4$ grid of pixels is composed of four $2\times 2$
pixel blocks, creating an RGGB Bayer pattern. Each pixel in these $2\times 2$
blocks contains its own Bayer-like pattern of polarizing filters, arranged in
$\\{\frac{\pi}{2},\frac{\pi}{4},\frac{3\pi}{4},0\\}$ order. See Figure 2.

In this work, we place an additional linear polarizing filter in front of the
lens of a polarization camera to modulate the intensity of light transmitted
onto the sensor. According to Malus’ law, the intensity of each of these
pixels is attenuated according to the square of the cosine of the angle
between the two overlapping polarizers [22]. This variable attenuation enables
the simultaneous capture of four exposure stops—rather than polarities—in a
single shot (see Figure 3).

###  III-B Problem Statement

Digital cameras measure the intensity of incident light $I_{0}$ by exposing
the sensor for some time $\Delta t$ to obtain a raw image

$\displaystyle I(\Delta t)$ $\displaystyle=f(I_{0}\Delta t)$ (1)

where $f$ is the clipping operator $\text{max}(\text{min}(x,255),0)$
corresponding to the 8-bit depth of our hardware.

Assuming that $\Delta t$ is known, reconstruction of an HDR image becomes a
straightforward process—simply compute a weighted sum of LDR images based on
exposure. The introduction of an additional polarizing filter (at an unknown
orientation) in our optical system complicates this process, however, as the
exposure of each image is no longer known.

Although HDR reconstruction algorithms that do not require explicit knowledge
of $\Delta t$ exist (e.g., [17]), these algorithms do not perform metric HDR
but rather produce physically inaccurate—albeit visually pleasing—images.
Thus, the question that we seek to answer becomes, “how can we recover the
relative exposures of the four images we capture?”

Figure 3: Malus’ law. The more that elements of the on-sensor polarizer array
are aligned with the linear polarizer, the more light they transmit. Likewise,
the more orthogonal they become, the more they attenuate incident light.
Transmission is proportional to cosine squared of the angle $\theta$ between
the polarizers.

###  III-C Exposure Calibration

With the addition of a polarizer in front of the lens, our polarization camera
simultaneously captures a set of four images
$S=\\{I(\theta_{1}),I(\theta_{2}),I(\theta_{3}),I(\theta_{4})\\}$
corresponding to polarization angles
$\\{0,\frac{\pi}{4},\frac{\pi}{2},\frac{3\pi}{4}\\}$ respectively. Each image
has a transmission rate dependent on $\theta$ that determines their exposure
$\Delta t=\text{cos}^{2}\theta_{i}$.

Assuming the camera’s raw measurements are accessible, we can ignore non-
linearities in the polarization camera’s image processing pipeline (ISP) and
express the intensities measured by each of the four pixel polarities as

$\displaystyle I(\theta_{1})$
$\displaystyle=f(I_{0}\text{cos}^{2}(\theta_{1}))$ $\displaystyle
I(\theta_{2})$ $\displaystyle=f(I_{0}\text{cos}^{2}(\theta_{2}))$
$\displaystyle I(\theta_{3})$
$\displaystyle=f(I_{0}\text{cos}^{2}(\theta_{3}))$ $\displaystyle
I(\theta_{4})$ $\displaystyle=f(I_{0}\text{cos}^{2}(\theta_{4})).$ (2)

We estimate the exposure of each image $I(\theta_{i})$ by observing that
pixels in the on-sensor polarizer array share the following relationship:

$\displaystyle\theta_{3}$ $\displaystyle=\theta_{1}+\frac{\pi}{2},$ (3)
$\displaystyle\theta_{4}$ $\displaystyle=\theta_{2}+\frac{\pi}{2}.$ (4)

For example, using Equation (3), we can express images $I(\theta_{1})$ and
$I(\theta_{3})$ in terms of $\theta_{1}$:

$\displaystyle I(\theta_{1})$ $\displaystyle=f(I_{0}\text{cos}^{2}\theta_{1})$
(5) $\displaystyle I(\theta_{3})$
$\displaystyle=f(I_{0}\text{sin}^{2}\theta_{1})$ (6)

For pixels for which both $I(\theta_{1})$ and $I(\theta_{3})$ are properly
exposed, we obtain a closed-form solution for estimating angle $\theta_{1}$

$\hat{\theta}_{1}=\text{arctan}\left(\sqrt{\frac{I(\theta_{3})}{I(\theta_{1})}}\right),$
(7)

which allows us to estimate the exposure: $\text{cos}^{2}\hat{\theta}_{1}$. We
estimate $\hat{\theta}_{2},\hat{\theta}_{3},$ and $\hat{\theta}_{4}$ using
analogous expressions.

It is important to note that angle estimates $\hat{\theta}$ are computed over
all $N$ pixels of the image. We aggregate these estimates (and reject under
and over-exposed pixels) by taking their mode to obtain our final (scalar)
estimate

$\hat{\theta}_{i}=\text{Mode}(\hat{\theta}_{i,j}),j\in[1,N].$ (8)

Figure 4: Fusing LDR into HDR images. Our polarization camera captures 4
multi-exposure images (left column). These are subsequently merged together to
produce an HDR image (right column).

Having acquired these values, reconstruction of the HDR image boils down to
the following steps: (1) scale each LDR image by its exposure, (2) mask out
overexposed and under-exposed regions of the image, and (3) compute a
normalized weighted sum over the masked and scaled LDR images:

Given a set of estimated angles $\hat{\theta}_{i}$ and corresponding LDR
images, we first rescale the intensity of each LDR image $I(\theta_{i})$ using

$I^{\prime}(\theta_{i})=\frac{I(\theta_{i})}{\text{cos}^{2}\hat{\theta_{i}}}.$
(9)

We subsequently assign each pixel in the LDR image a corresponding weight
indicating its well-exposedness. Specifically, pixels outside of the minimum
and maximum intensity range $[p_{min},p_{max}]$ are considered poorly exposed
and receive zero weight according to the indicator function

$w(x)=\begin{cases}1,p_{min}\leq x\leq p_{max}\\\
0,\text{otherwise}\end{cases}$ (10)

Finally, we reconstruct the HDR result $H$ using a normalized weighted sum
over the scaled images

$H=\frac{\sum_{i}^{4}w(I^{\prime}(\theta_{i}))\odot
I^{\prime}(\theta_{i})}{\sum_{i}^{4}w(I^{\prime}(\theta_{i}))+\epsilon}$ (11)

where $\odot$ is the Hadamard product and $\epsilon$ is a small positive value
which avoids division by zero.

Figure 5: Qualitative results. We compare our method against state-of-the-art multi-shot (HDR+ [19]), learning-based (HDR-CNN [12]) and polarization-based (Pol-HDR [15]) HDR reconstruction algorithms. Insets show zoomed in regions and $P_{det}$ heat maps [23]. Red pixels indicate high probabilities of detecting differences against the ground truth. Blue pixels indicate low detection probabilities. | HDR-CNN | HDR+ | Pol-HDR | Exposure Fusion | Ours
---|---|---|---|---|---
PSNR $\uparrow$ | 21.36 | 18.49 | 15.37 | 13.05 | 25.25
SSIM $\uparrow$ | 0.85 | 0.69 | 0.65 | 0.60 | 0.88
MS-SSIM $\uparrow$ | 0.90 | 0.83 | 0.73 | 0.77 | 0.90
Q-score $\uparrow$ | 6.12 | 5.97 | 3.25 | 5.26 | 7.40
Figure 6: Quantitative results. Image quality scores of each baseline are
averaged across all scenes in our dataset. Each metric is computed on the tone
mapped HDR results. Best scores are shown in bold.

##  IV Experiments

###  IV-A Hardware

Data in our experiments are captured on a LUCID Vision Lab PHX050S1-QC
polarization camera with a $38 off-the-shelf linear polarizer attached. All
images are shot in RAW format at $1224\times 1024$ resolution and each scene
is imaged across 11 different exposure settings (covering over 8 stops of
exposure). In total, we collect and label data from 12 natural scenes to form
our HDR dataset.

###  IV-B Setup and Baselines

We evaluate our single-shot HDR method against both single-shot [15, 17, 12]
and multi-shot [19] approaches on a variety of scenes. In each scene, we
calibrate the linear polarizer’s orientation to span the broadest dynamic
range possible and capture an exposure bracket $B=\\{S_{1},\cdots,S_{11}\\}$
consisting of 11 snapshots using different exposure times. Each snapshot
$S_{k}$ contains 4 images $I(\theta_{i})$, one image per polarity.

Using Equation (11), we reconstruct the ground truth HDR image from LDR images
$I(\theta_{i})$ in $B$ belonging to a single polarity $\theta_{i}$. Then, we
produce both our result and the Exposure Fusion [17] baseline result using
images $I(\theta_{1}),\cdots,I(\theta_{4})$ from a single snapshot $S_{k}$. We
also run HDR-CNN [12] on individual images from $S_{k}$ and use the highest
scoring reconstruction as a baseline.

To generate our HDR+ [19] baseline results, we capture a separate burst
$B_{hdrplus}$ of 10 short exposure images. We also capture a separate snapshot
$S_{polhdr}$ with the linear polarizer off to produce our Pol-HDR [15]
baseline results. All HDR reconstructions are tone mapped (where appropriate)
using Reinhard’s tone mapping operator [24] for better viewing on digital
displays. For fair comparison, we apply the same tone mapping operator to HDR
images for each scene.

Each method is benchmarked using peak signal-to-noise ratio (PSNR), structural
similarity index (SSIM) [25], multi-scale structural similarity index (MS-
SSIM) [26], Q-score, and $P_{map}$ [23]. For context, Q-score is an image
quality indicator that behaves similarly to MS-SSIM. On the other hand,
$P_{map}$ is a 2D heat map where the color of each pixel indicates the
probability of a human observer detecting differences against the ground
truth. Red pixels represent a high detection likelihood and blue pixels
represent a low detection likelihood.

###  IV-C Results

Of all the methods, Exposure Fusion [17] produces the lowest scoring results
due to its reliance on heuristics like saturation and contrast to merge
images. This causes the relative color and brightness of objects in the HDR
results to consistently differ from the ground truth. Pol-HDR [15] also scores
relatively low on our benchmark, as it runs on images captured with the linear
polarizer off, which severely limits the amount of intensity information
available. Note that their method requires images with 1:1 aspect ratio as
input, so we zero-pad their results before evaluation.

Google’s HDR+ [19] algorithm scores relatively well and produces reasonable
looking HDR images. However, its results suffer from the limited dynamic range
of the low exposure burst $B_{hdrplus}$. As a result, details in under-exposed
regions are barely discernible (see the brick pattern in the circular window
scene of Figure 5).

\begin{overpic}[width=104.07117pt]{img/leak/scene.pdf}
\put(18.0,87.0){{\centering Parallel Polarizers\@add@centering}}
\end{overpic}\begin{overpic}[width=104.07117pt]{img/leak/leak.pdf}
\put(15.0,87.0){{\centering Orthogonal Polarizers\@add@centering}}
\end{overpic}

Figure 7: Imperfect polarizers. In theory, polarizers perfectly polarize
incoming light so that no light is transmitted across two orthogonal
polarizers. However, imperfections occur during the manufacturing process such
that polarizers are never $100\%$ efficient. We demonstrate this by imaging a
light bulb (left) through orthogonal polarizers. The captured image (right)
visibly contains non-zero pixels around bright regions of the scene.

Compared to other baselines, HDR-CNN [12] was the highest performing method
despite taking only a single frame $I(\theta)$ as input. It reasonably
recovers information in under and over exposed regions of the image using
learned priors. For example, in the table lamp scene, HDR-CNN suppresses over-
exposed pixels around the incandescent bulb, making it appear more like a
street lamp.

Our method scores the highest across all metrics (see Figure 6) and achieves a
18% higher average PSNR than the next best method. Furthermore, $P_{map}$
scores indicate that our method produces images that qualitatively resemble
the ground truth. For example, both the shape of the tungsten filament in the
table lamp and the brick pattern around the circular window are well-exposed
in our result.

##  V Analysis

###  V-A Imperfect Polarizers

In theory, angle estimates $\hat{\theta}$ computed using Equation (7) recover
the exact angle $\theta$ and exposure $\text{cos}^{2}\theta$ of each image. In
practice, however, we observe errors in these estimates which stem from the
imperfections present in real polarizers [27].

Recall that polarizers filter incident light such that light parallel to the
polarizer’s axis has maximal transmission intensity $T_{max}$ and the light
orthogonal to the its axis has minimal transmission intensity $T_{min}$. Up
until now, we’ve assumed that the ratio between $T_{min}$ and $T_{max}$ (known
as the extinction ratio) is equal to 0, which is the case for perfect
polarizers. Real world polarizers, however, have a non-zero extinction ratio.
For instance, elements in the polarization array of our camera have an
extinction ratio $\rho$ around 1:500.

In Figure 7, we experimentally demonstrate the non-zero extinction ratio of
real world polarizers by setting the angle $\theta$ between two polarizers to
$\frac{\pi}{2}$. According to Malus’ law, all light should be blocked and the
recorded pixel intensity values should be zero. Instead, we visibly see white
pixels in the image around the bright tungsten filament in the light bulb.

To account for imperfect polarizers, we revisit Equation (7) and re-express
the intensity $I$ of light on the sensor in terms of the extinction ratio
$\rho$.

Irradiance $I_{0}$ incident on the camera can be described by the Jones vector
$\vec{E}_{0}$ where

$\displaystyle I_{0}$ $\displaystyle=|\vec{E}_{0}|^{2}.$ (12)

This vector expresses the polarization state of light as

$\vec{E}=\begin{bmatrix}A_{x}e^{i\delta_{x}}\\\
A_{y}e^{i\delta_{y}}\end{bmatrix}=\begin{bmatrix}X\\\ Y\end{bmatrix},$ (13)

where $A_{x},A_{y}$ denote amplitude and $\delta_{x},\delta_{y}$ denote phase
delay along axes $x,y$. For convenience, we define axes $x,y$ such that they
are aligned with the transmission and extinction axes of the first polarizer
$P_{1}$ respectively.

The extinction ratios $\rho_{1},\rho_{2}$ of polarizers $P_{1},P_{2}$ share
the following relationship with amplitudes $A_{1,x},A_{1,y}$ of waves exiting
$P_{1}$ and amplitudes $A_{2,x},A_{2,y}$ of waves exiting $P_{2}$:

$\displaystyle\rho_{1}$ $\displaystyle=\sqrt{\frac{A_{1,y}}{A_{1,x}}}$ (14)
$\displaystyle\rho_{2}$ $\displaystyle=\sqrt{\frac{A_{2,y}}{A_{2,x}}}.$ (15)

For simplicity, we assume that both polarizers have the same extinction ratio
$\rho_{1}=\rho_{2}=\rho$, but our derivation also applies to non-uniform
extinction ratios.

The first polarizer $P_{1}$ attenuates the amplitude $A_{y,0}$ of incident
light $\vec{E}_{0}$ along the $y$ axis by a factor

$\displaystyle\alpha=\sqrt{\frac{\rho}{\rho+1}}$ (16)

Assuming that $T_{max}=1$, the light $\vec{E}_{1}$ exiting $P_{1}$
subsequently becomes

$\vec{E}_{1}=\begin{bmatrix}1\\\ \alpha\end{bmatrix}\odot\vec{E}_{0}.$ (17)

The second polarizer $P_{2}$ also attenuates light by a factor $\alpha$.
However, this attenuation happens along $P_{2}$’s axes which are not
necessarily aligned with $x,y$. By definition, the two axes differ from one
another with angle $\theta$. So, we project light $\vec{E}_{1}$ after $P_{1}$
onto $P_{2}$’s axes $u,v$ before applying the attenuation factor $\alpha$.

$proj_{\textbf{u,v}}{\vec{E}_{2}}=\begin{bmatrix}1\\\
\alpha\end{bmatrix}\begin{bmatrix}\text{cos}\theta&\text{sin}\theta\\\
-\text{sin}\theta&\text{cos}\theta\end{bmatrix}\vec{E}_{1}$ (18)

Finally, we reproject the attenuated vector onto $x,y$ and obtain an
expression for the light $\vec{E}_{2}$ that hits the sensor

$\vec{E}_{2}=\begin{bmatrix}\text{cos}\theta&-\text{sin}\theta\\\
\text{sin}\theta&\text{cos}\theta\end{bmatrix}proj_{\textbf{u,v}}{\vec{E}_{2}}.$
(19)

Expanding terms, we derive the following expression of light observed by the
sensor $\vec{E}_{2}$ with respect to $\alpha$ and incident light
$\vec{E}_{0}=\begin{bmatrix}X_{0}\\\ Y_{0}\end{bmatrix}$

$\vec{E}_{2}=\begin{bmatrix}X_{0}\text{cos}^{2}\theta+\alpha
X_{0}\text{sin}^{2}\theta+(\alpha+\alpha^{2})(Y_{0}\text{sin}{\theta}\text{cos}{\theta})\\\
\alpha^{2}Y_{0}\text{cos}^{2}\theta+\alpha
Y_{0}\text{sin}^{2}\theta+(1-\alpha)(X_{0}\text{sin}{\theta}\text{cos}{\theta})\\\
\end{bmatrix}.$ (20)

Intensity measured by the camera is simply the squared amplitude of the wave
$\vec{E}_{2}$ at the sensor plane, described by

$I(\theta,\alpha)=|\vec{E}_{2}|^{2}.$ (21)

Removing our prior assumption of perfect polarizers, we reformulate Equation
(7) to take an additional parameter $\alpha$ related to the extinction ratio
by Equation (18).

$\hat{\theta}_{1}=\text{arctan}\left(\sqrt{\frac{I(\theta_{3},\alpha)}{I(\theta_{1},\alpha)}}\right)$
(22)

Figure 8: Extinction ratio vs. $\hat{\theta}$ estimate error. As the
polarizers’ extinction ratio $\rho$ deviates from 0, errors in our estimate of
angle $\theta$ grow logarithmically. The extinction ratio of our linear
polarizer is drawn in red for reference.

In Figure 8, we visualize the effect that imperfect polarizers have on our
estimates by plotting estimate error as a function of the extinction ratio
$\rho$ (assuming the incident light is vertically polarized). Interestingly,
the graph reveals an approximately logarithmic relationship between the error
and extinction ratio. For example, on our imaging hardware, which has an
extinction ratio of 1:500, our method estimates ${\theta}_{1}$ with an error
of 2.93%.

###  V-B Highly Polarized Scenes

Our optical setup measures an HDR scene at a single polarity; this produces
measurements distinct from those obtained from an exposure bracket captured by
a standard DSLR camera. When dealing with highly polarized scenes (e.g., light
reflected off the surface of a lake), the angle $\phi$ of the initial linear
polarizer $P_{1}$ noticeably affects the resulting image. To analyze this, we
conduct the following experiment:

We position our polarization camera at approximately Brewster’s angle relative
to the reflective surface of a creek. With the linear polarizer off, we
capture an multi-shot exposure bracket and generate 4 HDR images (one image
per polarity). Reattaching the linear polarizer, we recapture the scene
twice—once with the polarizer nearly vertical ($\phi\approx\pi/2$) and once
with it nearly horizontal ($\phi\approx 0$).

In Figure 9, we compare (a) a vertically-polarized, multi-shot HDR reference
image against three images: (b) the horizontally-polarized multi-shot HDR
image, (c) our method’s snapshot HDR image with $\phi=-13^{\circ}$, and (d)
our method’s snapshot HDR image with $\phi=77^{\circ}$. Images captured with
the polarizer at a vertical angle filter out specular reflections—allowing us
to see through the water’s surface—whereas those with the polarizer at a
horizontal angle (b, c) do not.

Although our method captures light at a single polarity, we emphasize that
this property is not strictly negative. Filtration of light by polarity is
useful for numerous applications such as material classification and
reflection removal. Consider the example in Figure 9. Without the use of a
polarizer, specular reflections on the creek completely obstruct our view of
the rock and sediment beneath the water’s surface. Inclusion of a vertical
polarizer allows us to filter out these reflections and capture a clear image
of the riverbed.

Figure 9: Sensitivity to polarized incident light. Imaging through a vertical
polarizer (left column) filters out specular highlights reflecting off of the
water’s surface. Rotating the polarizer horizontally (right column) allows us
to image reflections on the water. Overlays show Q-scores and $P_{det}$ heat
maps against the reference image (a).

##  VI Discussion and Limitations

Our technique has a few notable limitations, the most significant of which is
light efficiency. After passing through the polarizers, only a fraction of the
light incident on the camera’s initial polarizer makes it to the sensor. Thus
our system is best suited for imaging in well-lit environments.

Another limitation associated with our method is its spatial resolution.
Similar to mosaiced HDR sensors, our polarization camera trades off spatial
resolution to capture multi-exposure information. Advanced demosaicking
algorithms could help recover this information [28].

###  VI-A Mosaiced HDR Sensors

Mosaiced HDR sensors are powerful tools for single-shot HDR; yet, they are
specialized sensors that cost upwards of $500. By contrast, our proposed
system costs as little as $40 for individuals that own a polarization camera.
In addition, our system offers the ability to measure polarity simply by
removing the polarizer, which is useful for surface normal estimation,
reflection removal, material classification, and more.

Figure 10: Additional results.

##  VII Conclusion

In this work, we propose a straightforward and reversible procedure to turn a
polarization camera into a snapshot HDR camera. Moreover, we introduce a
simple, yet effective, self-calibrating HDR reconstruction algorithm for our
system and experimentally demonstrate that our algorithm outperforms existing
single-shot and multi-shot HDR reconstruction methods. Finally, we analyze the
effects that both imperfect polarizers and highly polarized light have on our
results.

Unlike dedicated HDR sensors, our system is dual purpose—the polarizer can be
added for HDR imaging or removed for polarization imaging. Our approach thus
provides a convenient way for polarization camera users to perform high-
quality snapshot HDR imaging at a minimal cost.

## References

  * [1] Mann, Picard, S. Mann, and R. W. Picard, “On being ‘undigital’ with digital cameras: Extending dynamic range by combining differently exposed pictures,” in _Proceedings of IS &T_, 1995, pp. 442–448.
  * [2] P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in _Proc. SIGGRAPH_ , 1997, pp. 369–378.
  * [3] T. Mertens, J. Kautz, and F. Van Reeth, “Exposure fusion: A simple and practical alternative to high dynamic range photography,” in _Computer graphics forum (Eurographics)_ , vol. 28, no. 1. Wiley Online Library, 2009, pp. 161–171.
  * [4] M. McGuire, W. Matusik, H. Pfister, B. Chen, J. F. Hughes, and S. K. Nayar, “Optical splitting trees for high-precision monocular imaging,” _IEEE Computer Graphics and Applications_ , vol. 27, no. 2, pp. 32–42, 2007\.
  * [5] M. D. Tocci, C. Kiser, N. Tocci, and P. Sen, “A versatile hdr video production system,” _ACM Trans. Graph. (SIGGRAPH)_ , vol. 30, no. 4, pp. 41:1–41:10, 2011.
  * [6] M. Rouf, R. Mantiuk, W. Heidrich, M. Trentacoste, and C. Lau, “Glare encoding of high dynamic range images,” in _Proc. CVPR 2011_ , 2011, pp. 289–296.
  * [7] C. A. Metzler, H. Ikoma, Y. Peng, and G. Wetzstein, “Deep optics for single-shot high-dynamic-range imaging,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1375–1385.
  * [8] Q. Sun, E. Tseng, Q. Fu, W. Heidrich, and F. Heide, “Learning rank-1 diffractive optics for single-shot high dynamic range imaging,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 1386–1396.
  * [9] H. Zhao, B. Shi, C. Fernandez-Cull, S. Yeung, and R. Raskar, “Unbounded high dynamic range photography using a modulo camera,” in _Proc. ICCP_ , 2015, pp. 1–10.
  * [10] C. Zhou, H. Zhao, J. Han, C. Xu, C. Xu, T. Huang, and B. Shi, “Unmodnet: Learning to unwrap a modulo image for high dynamic range imaging,” in _Neural Information Processing Systems_ , 2020.
  * [11] H. M. So, J. N. P. Martel, P. Dudek, and G. Wetzstein, “Mantissacam: Learning snapshot high-dynamic-range imaging with perceptually-based in-pixel irradiance encoding,” _2022 IEEE International Conference on Computational Photography (ICCP)_ , pp. 1–12, 2021.
  * [12] G. Eilertsen, J. Kronander, G. Denes, R. K. Mantiuk, and J. Unger, “Hdr image reconstruction from a single exposure using deep cnns,” _ACM Transactions on Graphics (TOG)_ , vol. 36, no. 6, p. 178, 2017.
  * [13] S. K. Nayar and T. Mitsunaga, “High dynamic range imaging: Spatially varying pixel exposures,” in _Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662)_ , vol. 1. IEEE, 2000, pp. 472–479.
  * [14] J. Bennett, “Polarization — introduction,” in _Encyclopedia of Modern Optics_ , R. D. Guenther, Ed. Oxford: Elsevier, 2005, pp. 190–204. [Online]. Available: https://www.sciencedirect.com/science/article/pii/B0123693950006916
  * [15] C. Zhou, Y. Han, M. Teng, J. Han, S. Li, C. Xu, and B. Shi, “Polarization guided hdr reconstruction via pixel-wise depolarization,” _IEEE Transactions on Image Processing_ , vol. 32, pp. 1774–1787, 2023.
  * [16] P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in _ACM SIGGRAPH 2008 classes_ , 2008, pp. 1–10.
  * [17] T. Mertens, J. Kautz, and F. Van Reeth, “Exposure fusion,” in _15th Pacific Conference on Computer Graphics and Applications (PG’07)_. IEEE, 2007, pp. 382–390.
  * [18] Q. Yan, D. Gong, P. Zhang, Q. Shi, J. Sun, I. Reid, and Y. Zhang, “Multi-scale dense networks for deep high dynamic range imaging,” in _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2019, pp. 41–50.
  * [19] S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” _ACM Transactions on Graphics (TOG)_ , vol. 35, no. 6, p. 192, 2016.
  * [20] X. Chen, Y. Liu, Z. Zhang, Y. Qiao, and C. Dong, “Hdrunet: Single image hdr reconstruction with denoising and dequantization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 354–363.
  * [21] H. Zhao, B. Shi, C. Fernandez-Cull, S.-K. Yeung, and R. Raskar, “Unbounded high dynamic range photography using a modulo camera,” in _2015 IEEE International Conference on Computational Photography (ICCP)_. IEEE, 2015, pp. 1–10.
  * [22] E. Collett, _Field Guide to Polarization_ , ser. Field Guides. SPIE Press, 2005. [Online]. Available: https://books.google.com/books?id=5lJwcCsLbLsC
  * [23] R. Mantiuk, K. J. Kim, A. G. Rempel, and W. Heidrich, “Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions,” _ACM Transactions on graphics (TOG)_ , vol. 30, no. 4, pp. 1–14, 2011.
  * [24] E. Reinhard, W. Heidrich, P. Debevec, S. Pattanaik, G. Ward, and K. Myszkowski, _High dynamic range imaging: acquisition, display, and image-based lighting_. Morgan Kaufmann, 2010.
  * [25] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [26] Z. Wang, E. Simoncelli, and A. Bovik, “Multiscale structural similarity for image quality assessment,” in _The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003_, vol. 2, 2003, pp. 1398–1402 Vol.2.
  * [27] C. Huang, S. Zhao, H. Chen, and Z. Liao, “Attenuation characterization of multiple combinations of imperfect polarizers,” _JOSA A_ , vol. 27, no. 5, pp. 1060–1068, 2010.
  * [28] S. Qiu, Q. Fu, C. Wang, and W. Heidrich, “Linear polarization demosaicking for monochrome and colour polarization focal plane arrays,” in _Computer Graphics Forum_ , vol. 40, no. 6. Wiley Online Library, 2021, pp. 77–89.


NLP

    natural language processing
NLI

    natural language inference
LM

    Language Model
PLM

    Pretrained Language Model
STS

    semantic textual similarity
KG

    knowledge graph
PwC

    Papers with Code
P

    Precision
R

    Recall
MRR

    Mean Reciprocal Rank
MAP

    Mean Average Precision

# Efficient Domain Adaptation of Sentence Embeddings Using Adapters

Tim Schopf, Dennis N. Schneider, and Florian Matthes
Technical University of Munich, Department of Computer Science, Germany
<EMAIL_ADDRESS>
###### Abstract

Sentence embeddings enable us to capture the semantic similarity of short
texts. Most sentence embedding models are trained for general semantic textual
similarity tasks. Therefore, to use sentence embeddings in a particular
domain, the model must be adapted to it in order to achieve good results.
Usually, this is done by fine-tuning the entire sentence embedding model for
the domain of interest. While this approach yields state-of-the-art results,
all of the model’s weights are updated during fine-tuning, making this method
resource-intensive. Therefore, instead of fine-tuning entire sentence
embedding models for each target domain individually, we propose to train
lightweight adapters. These domain-specific adapters do not require fine-
tuning all underlying sentence embedding model parameters. Instead, we only
train a small number of additional parameters while keeping the weights of the
underlying sentence embedding model fixed. Training domain-specific adapters
allows always using the same base model and only exchanging the domain-
specific adapters to adapt sentence embeddings to a specific domain. We show
that using adapters for parameter-efficient domain adaptation of sentence
embeddings yields competitive performance within 1% of a domain-adapted,
entirely fine-tuned sentence embedding model while only training approximately
3.6% of the parameters.

##  1 Introduction

Learning sentence embeddings is an essential task in natural language
processing (NLP) and has already been extensively investigated in the
literature Kiros et al. (2015); Hill et al. (2016); Conneau et al. (2017);
Logeswaran and Lee (2018); Cer et al. (2018); Reimers and Gurevych (2019); Gao
et al. (2021); Wu et al. (2022); Schopf et al. (2023d, a). Sentence embeddings
are especially useful in information retrieval Lewis et al. (2020); Schopf et
al. (2022); Schneider et al. (2022) or unsupervised text classification
settings Schopf et al. (2021, 2023b, 2023c). Lately, the most popular approach
for sentence embedding learning is to fine-tune pretrained language models
with a contrastive learning objective Liu et al. (2021); Zhang et al. (2022);
Chuang et al. (2022); Nishikawa et al. (2022); Cao et al. (2022); Jiang et al.
(2022). While this approach provides state-of-the-art results, all of the
model’s weights are updated during fine-tuning, making this method resource-
intensive. This is a problem, particularly when domain-specific models are
needed. Then, a specialized model must be trained for each domain of interest,
resulting in resource-intensive training.

Figure 1: Sentence embedding models are usually trained to obtain state-of-
the-art sentence representations for general semantic textual similarity
tasks. By injecting domain-specific knowledge of adapters into the sentence
embedding model, we can efficiently adapt the resulting representations for
semantic textual similarity tasks in different domains.

Recently, adapters have emerged as a parameter-efficient strategy to fine-tune
Language Models. Adapters do not require fine-tuning of all parameters of the
pretrained model and instead introduce a small number of task-specific
parameters while keeping the underlying pretrained language model fixed
Pfeiffer et al. (2021a). They enable efficient parameter sharing between tasks
and domains by training many task-specific, domain-specific, and language-
specific adapters for the same model, which can be exchanged and combined
post-hoc Pfeiffer et al. (2020a). Therefore, many different adapter
architectures have been proposed for various domains and tasks Pfeiffer et al.
(2020b, 2021b); Vidoni et al. (2020); He et al. (2021); Le et al. (2021);
Parović et al. (2022); Lee et al. (2022). However, to the best of our
knowledge, no method currently exists for efficient domain adaptation of
sentence embeddings using adapters.

In this paper, we aim to bridge this gap by proposing approaches for adapter-
based domain adaptation of sentence embeddings, allowing us to train models
for many different domains efficiently. Therefore, we investigate how to adapt
general pretrained sentence embedding models to different domains using
domain-specific adapters. As shown in Figure 1, this allows always using the
same base model to adapt sentence embeddings to a specific domain and only
needing to exchange the domain-specific adapters. Accordingly, we train
lightweight adapters for each domain and avoid expensive training of entire
sentence embedding models.

##  2 Related Work

Adapters have been introduced by Houlsby et al. (2019) as a parameter-
efficient alternative for task-specific fine-tuning of language models. Since
their introduction, adapters have been used to fine-tune models for single
tasks as well as in multi-task settings Pfeiffer et al. (2021a). Usually,
adapters are used to solve tasks such as classification Lauscher et al.
(2020), machine translation Baziotis et al. (2022), question answering
Pfeiffer et al. (2022), or reasoning Pfeiffer et al. (2021a). While there
exist adapters for semantic textual similarity (STS) tasks on the AdapterHub
Pfeiffer et al. (2020a), these are trained on general STS datasets using a
task-unspecific pretrained language model as a basis. We, however, focus on
adapting pretrained sentence embedding models to specific domains using
adapters.

##  3 Method

We assume we have a base sentence embedding model from the source domain and
labeled datasets for each target domain. Instead of fine-tuning the entire
sentence embedding model for each target domain individually, we train
lightweight adapters for each domain. This domain-specific fine-tuning with
adapters involves adding a small number of new parameters to the sentence
embedding model. During training, the parameters of the sentence embedding
model are frozen, and only the weights of the adapters are updated. Formally,
we adopt the general definition for adapter-based fine-tuning of Pfeiffer et
al. (2021a) as follows:

For each of the $N$ domains, the sentence embedding model is initialized with
parameters $\Theta_{0}$. Additionally, a set of new and randomly initialized
adapter parameters $\Phi_{n}$ are introduced. The parameters $\Theta_{0}$ are
fixed and only the parameters $\Phi_{n}$ are trained. Given training data
$D_{n}$ and a loss function $L$, the objective for each domain $n\in{1,...,N}$
is of the form:

$\Phi_{n}\leftarrow\mathop{\mathrm{argmin}}\limits_{\Phi}L(D_{n};\Theta_{0},\Phi)$
(1)

Usually, the adapter parameters $\Phi_{n}$ are significantly less than the
parameters $\Theta_{0}$ of the base model Pfeiffer et al. (2021a), e.g., only
3.6% of the parameters of the pretrained model in Houlsby et al. (2019).

##  4 Experiments

In this section, we describe the used adapter architectures, loss functions,
and datasets. In all experiments, we use $\textrm{SimCSE}_{sup-bert-base}$ Gao
et al. (2021) as the base sentence embedding model. It is trained on natural
language inference (NLI) datasets Bowman et al. (2015); Williams et al. (2018)
for STS tasks in the general domain. We fine-tune all models and adapters for
five epochs using a learning rate of $1e^{-5}$.

###  4.1 Adapter Architectures

We investigate how different adapter architectures affect the domain
adaptability of sentence embedding models.

#### Houlsby-Adapter

This adapter, introduced by Houlsby et al. (2019), uses a bottleneck
architecture. The adapter modules are added after both the multi-head
attention and feed-forward block in each transformer layer Vaswani et al.
(2017) of the base model. The adapter layers transform their input into a very
low-dimensional representation and upsample it again to the same dimension in
the output. This generates a parameter-efficient lower-dimensional
representation while most information is kept.

Figure 2: Houlsby-Adapter architecture as introduced by Houlsby et al. (2019).
On the left side, the adapter is illustrated to be added twice to each
transformer layer. Once after the multi-head attention and once after the
feed-forward layer. On the right side, the bottleneck architecture of the
adapter is presented.

#### Pfeiffer-Adapter

This adapter, introduced by Pfeiffer et al. (2021a), also uses a bottleneck
architecture. However, the adapter modules are added only after the feed-
forward block in each transformer layer of the base model. This architecture
allows merging multiple adapters trained on different tasks. In this work,
however, this multitask learning capability is not needed, and we only use the
single-task mode.

Figure 3: Pfeiffer-Adapter architecture as introduced by Pfeiffer et al.
(2021a). Unlike the Houlsby-Adapter, a single Pfeiffer-Adapter is added in
each transformer block only after the forward layer.

#### K-Adapter

This adapter, introduced by Wang et al. (2021), works as outside plug-in for
the base model. Each adapter model consists of $K$ adapter layers containing
$N$ transformer layers and two projection layers across which a skip
connection is applied. The adapter layers combine the output of an
intermediate transformer layer in the base model with the output of a previous
adapter layer. To generate the final output, the last hidden states of the
adapter are concatenated with the last hidden states of the base model and
transformed into the correct output dimension with a simple dense layer.

Figure 4: K-Adapter architecture as introduced by Wang et al. (2021). The
adapter layer (left) consists of two projection layers, $N=2$ transformer
layers, and a skip connection between two projection layers. The adapter
layers are plugged among different transformer layers of the base model. The
final output consists of the concatenated last hidden states of the adapter
and the base model.

For reference, Table 1 shows the number of parameters per adapter model
compared to commonly used base models, highlighting the efficient nature of
adapters.

| BERT-base | RoBERTa-large
---|---|---
| No. of Parameters
---
Base Model
110M | 355M
| No. of Parameters
---
Houlsby-Adapter
4M | 6M
| No. of Parameters
---
Pfeiffer-Adapter
10M | 12M
| No. of Parameters
---
K-Adapter
47M | 47M

Table 1: Number of trainable Parameters for different base models and adapter
architectures.

Datasets $\rightarrow$ AskUbuntu SciDocs Average Models $\downarrow$ Cite CC
CR CV Out-of-the-box SimCSE (lower bound) 60.3 79.3 82.10 76.87 78.36 75.39
$\ell_{1}$
Houlsby-Adapter 64.0 88.2 88.69 82.42 83.99 81.46 Pfeiffer-Adapter 63.8 87.8
88.73 81.65 83.27 81.05 K-Adapter 62.5 85.6 87.70 80.09 82.85 79.75 In-domain
supervised SimCSE (upper bound) 65.3 88.0 87.74 84.15 83.32 81.70 $\ell_{2}$
Houlsby-Adapter 64.5 87.3 89.01 82.41 84.42 81.53 Pfeiffer-Adapter 64.2 87.0
88.63 81.98 84.41 81.24 K-Adapter 62.8 85.3 87.92 80.05 83.29 79.87 In-domain
supervised SimCSE (upper bound) 65.2 88.3 88.11 84.46 83.63 81.94

Table 2: Evaluation results of the adapter-based domain adaptation using the
different loss functions $\ell_{1}$ and $\ell_{2}$. The evaluation metric is
MAP. We show the performance of the SimCSE model without domain-specific fine-
tuning as a lower bound. Additionally, we show the performance of SimCSE
models using traditional fine-tuning with the respective loss functions as
upper bounds. For the upper bounds, all model weights have been updated during
training. In contrast, only the adapter weights were updated during adapter
training while the base model parameters were frozen. In bold, we highlight
the best adapter performance overall and underline the best adapter results
per loss function.

###  4.2 Loss Functions

We investigate two different loss functions that are proven to teach models to
learn a notion of STS from triplets of examples. We assume a set of triplets
$\mathcal{D}=\\{(x_{i},x_{i}^{+},x_{i}^{-})\\}$, where $x_{i}$ is an anchor
sentence, $x_{i}^{+}$ is a positive sample and $x_{i}^{-}$ is a negative
sample. With $h_{i}$, $h_{i}^{+}$, and $h_{i}^{-}$ as representations of
$x_{i}$, $x_{i}^{+}$, and $x_{i}^{-}$, we use the triplet margin loss function
of Cohan et al. (2020) as follows:

$\ell_{1}=\max\\{(d(h_{i},h_{i}^{+})-d(h_{i},h_{i}^{-})+m),0\\}$

(2)

where $d$ is the L2 norm distance function and $m$ is the loss margin
hyperparameter set to 1.

Additionally, we use the contrastive objective of Gao et al. (2021) as
follows:

$\ell_{2}=-\log\frac{e^{sim(h_{i},h_{i}^{+})/\tau}}{\sum_{j=1}^{N}(e^{sim(h_{i},h_{j}^{+})/\tau}+e^{sim(h_{i},h_{j}^{-})/\tau})}$

(3)

with a mini-batch of $N$ triplets, a temperature hyperparameter $\tau$, which
is empirically set to 0.05, and $sim(h_{1},h_{2})$ as the cosine similarity
$\frac{h_{1}\cdot h_{2}}{||h_{1}||\cdot||h_{2}||}$.

###  4.3 Data

We use datasets from two different domains to evaluate the domain adaptation
abilities of our approach. We randomly split both domain-specific datasets
into 90% training and 10% test datasets.

#### SciDocs

The SciDocs dataset Cohan et al. (2020) consists of scientific papers and
their citation information. As model input, we concatenate the titles and
abstracts of papers with the [SEP] token. Since our model has a maximum input
length of $512$ tokens, the input is cut off after this threshold. A positive
sample is defined as a directly referenced paper for each anchor sample. A
negative sample is a paper referenced by the positive sample but not by the
anchor sample itself. This approach ensures that all samples address the same
topic, but the positive sample is more related to the anchor sample than the
negative one.

#### AskUbuntu

The AskUbuntu dataset Lei et al. (2016) consists of user posts from the
technical forum AskUbuntu. It already includes sentence pairs that are deemed
similar. Therefore, anchor- and positive samples are easily found. Since the
dataset inherently consists of sentences about a similar topic, the operating
system Ubuntu, negative sentences can easily be retrieved by sampling
different sentences. The dataset originates from a technical domain and is
quite different from the scientific domain of SciDocs.

##  5 Evaluation

Table 2 shows the results obtained when adapting sentence embedding models to
different domains with adapters. To put the adapter results into perspective,
we also evaluate the performance of the SimCSE base model, which is not
adapted to the specific domains, as a lower bound. Furthermore, we use
traditional domain-specific fine-tuning by training all parameters of the
SimCSE base model with the respective loss functions as upper bounds.

The evaluation reveals that adapter-based domain adaptation yields competitive
results compared to fine-tuning the entire base model. In particular, the
Houlsby and Pfeiffer adapters perform very well with both loss functions, even
though they use only a fraction of the parameters of the upper bounds. The
slightly larger K-Adapter, however, performs considerably worse than the other
adapters investigated. We conclude that the bottleneck architecture is more
suitable than the external plug-in architecture for domain adaptation of
sentence embedding models. In particular, the Houlsby adapter, although the
smallest among the adapters investigated, yields the best results for both
loss functions. Using the out-of-the-box SimCSE model without domain
adaptation results in considerably worse performance, indicating the overall
importance of domain-specific fine-tuning for sentence embedding models.

Furthermore, the contrastive loss function $\ell_{2}$ performs consistently
better than $\ell_{1}$. Our results align with the observations of Gao et al.
(2021) who conclude that the contrastive objective ensures a distribution of
embeddings around the entire embedding space. In contrast, $\ell_{1}$ may
yield learned representations occupying a narrow vector space cone, which
severely limits their expressiveness.

From the obtained results, we conclude that using the Houlsby-Adapter
architecture together with the contrastive objective $\ell_{2}$ is most
suitable for parameter-efficient domain adaptation of sentence embedding
models. This adapter approach shows performance that is within 1% of the
supervised, entirely fine-tuned SimCSE model, while only training
approximately 3.6% of the parameters.

##  6 Conclusion

In this work, we proposed the use of adapters for parameter-efficient domain
adaptation of sentence embedding models. In contrast to fine-tuning the entire
sentence embedding model for a particular domain, adapters add a small number
of new parameters that are updated during training while the weights of the
sentence embedding model are fixed. We showed that adapter-based domain
adaptation of sentence embedding models yields competitive results compared to
fine-tuning the entire model, although only a fraction of the parameters are
trained. In particular, we show that using the Houlsby-Adapter architecture
together with a contrastive objective yields promising results for parameter-
efficient domain adaptation of sentence embedding models.

## References

  * Baziotis et al. (2022) Christos Baziotis, Mikel Artetxe, James Cross, and Shruti Bhosale. 2022.  Multilingual machine translation with hyper-adapters.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 1170–1185, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Cao et al. (2022) Rui Cao, Yihao Wang, Yuxin Liang, Ling Gao, Jie Zheng, Jie Ren, and Zheng Wang. 2022\.  Exploring the impact of negative samples of contrastive learning: A case study of sentence embedding.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3138–3152, Dublin, Ireland. Association for Computational Linguistics.
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018.  Universal sentence encoder for English.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 169–174, Brussels, Belgium. Association for Computational Linguistics.
  * Chuang et al. (2022) Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, and James Glass. 2022.  DiffCSE: Difference-based contrastive learning for sentence embeddings.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4207–4218, Seattle, United States. Association for Computational Linguistics.
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020.  SPECTER: Document-level representation learning using citation-informed transformers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2270–2282, Online. Association for Computational Linguistics.
  * Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017.  Supervised learning of universal sentence representations from natural language inference data.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * He et al. (2021) Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, and Luo Si. 2021.  On the effectiveness of adapter-based tuning for pretrained language model adaptation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2208–2222, Online. Association for Computational Linguistics.
  * Hill et al. (2016) Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016.  Learning distributed representations of sentences from unlabelled data.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1367–1377, San Diego, California. Association for Computational Linguistics.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR.
  * Jiang et al. (2022) Yuxin Jiang, Linhan Zhang, and Wei Wang. 2022.  Improved universal sentence embeddings with prompt-based contrastive learning and energy-based learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 3021–3035, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Skip-thought vectors.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.
  * Lauscher et al. (2020) Anne Lauscher, Olga Majewska, Leonardo F. R. Ribeiro, Iryna Gurevych, Nikolai Rozanov, and Goran Glavaš. 2020.  Common sense or world knowledge? investigating adapter-based knowledge injection into pretrained transformers.  In _Proceedings of Deep Learning Inside Out (DeeLIO): The First Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pages 43–49, Online. Association for Computational Linguistics.
  * Le et al. (2021) Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, and Laurent Besacier. 2021.  Lightweight adapter tuning for multilingual speech translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 817–824, Online. Association for Computational Linguistics.
  * Lee et al. (2022) Jaeseong Lee, Seung-won Hwang, and Taesup Kim. 2022.  FAD-X: Fusing adapters for cross-lingual transfer to low-resource languages.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 57–64, Online only. Association for Computational Linguistics.
  * Lei et al. (2016) Tao Lei, Hrishikesh Joshi, Regina Barzilay, Tommi Jaakkola, Kateryna Tymoshenko, Alessandro Moschitti, and Lluís Màrquez. 2016.  Semi-supervised question retrieval with gated convolutions.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1279–1289, San Diego, California. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 9459–9474. Curran Associates, Inc.
  * Liu et al. (2021) Che Liu, Rui Wang, Jinghua Liu, Jian Sun, Fei Huang, and Luo Si. 2021.  DialogueCSE: Dialogue-based contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2396–2406, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Logeswaran and Lee (2018) Lajanugen Logeswaran and Honglak Lee. 2018.  An efficient framework for learning sentence representations.  In _International Conference on Learning Representations_.
  * Nishikawa et al. (2022) Sosuke Nishikawa, Ryokan Ri, Ikuya Yamada, Yoshimasa Tsuruoka, and Isao Echizen. 2022.  EASE: Entity-aware contrastive learning of sentence embedding.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3870–3885, Seattle, United States. Association for Computational Linguistics.
  * Parović et al. (2022) Marinela Parović, Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2022\.  BAD-X: Bilingual adapters improve zero-shot cross-lingual transfer.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1791–1799, Seattle, United States. Association for Computational Linguistics.
  * Pfeiffer et al. (2022) Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. 2022.  xGQA: Cross-lingual visual question answering.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2497–2511, Dublin, Ireland. Association for Computational Linguistics.
  * Pfeiffer et al. (2021a) Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021a.  AdapterFusion: Non-destructive task composition for transfer learning.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 487–503, Online. Association for Computational Linguistics.
  * Pfeiffer et al. (2020a) Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a.  AdapterHub: A framework for adapting transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 46–54, Online. Association for Computational Linguistics.
  * Pfeiffer et al. (2020b) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020b.  MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7654–7673, Online. Association for Computational Linguistics.
  * Pfeiffer et al. (2021b) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2021b.  UNKs everywhere: Adapting multilingual language models to new scripts.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10186–10203, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Schneider et al. (2022) Phillip Schneider, Tim Schopf, Juraj Vladika, Mikhail Galkin, Elena Simperl, and Florian Matthes. 2022.  A decade of knowledge graphs in natural language processing: A survey.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 601–614, Online only. Association for Computational Linguistics.
  * Schopf et al. (2023a) Tim Schopf, Karim Arabi, and Florian Matthes. 2023a.  Exploring the landscape of natural language processing research.
  * Schopf et al. (2021) Tim Schopf, Daniel Braun, and Florian Matthes. 2021.  Lbl2vec: An embedding-based approach for unsupervised document retrieval on predefined topics.  In _Proceedings of the 17th International Conference on Web Information Systems and Technologies - WEBIST_ , pages 124–132. INSTICC, SciTePress.
  * Schopf et al. (2023b) Tim Schopf, Daniel Braun, and Florian Matthes. 2023b.  Evaluating unsupervised text classification: Zero-shot and similarity-based approaches.  In _Proceedings of the 2022 6th International Conference on Natural Language Processing and Information Retrieval_ , NLPIR ’22, page 6–15, New York, NY, USA. Association for Computing Machinery.
  * Schopf et al. (2023c) Tim Schopf, Daniel Braun, and Florian Matthes. 2023c.  Semantic label representations with lbl2vec: A similarity-based approach for unsupervised text classification.  In _Web Information Systems and Technologies_ , pages 59–73, Cham. Springer International Publishing.
  * Schopf et al. (2023d) Tim Schopf, Emanuel Gerber, Malte Ostendorff, and Florian Matthes. 2023d.  Aspectcse: Sentence embeddings for aspect-based semantic textual similarity using contrastive learning and structured knowledge.
  * Schopf et al. (2022) Tim Schopf, Simon Klimek, and Florian Matthes. 2022.  Patternrank: Leveraging pretrained language models and part of speech for unsupervised keyphrase extraction.  In _Proceedings of the 14th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2022) - KDIR_ , pages 243–248. INSTICC, SciTePress.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Vidoni et al. (2020) Marko Vidoni, Ivan Vulic, and Goran Glavas. 2020.  Orthogonal language and task adapters in zero-shot cross-lingual transfer.  _CoRR_ , abs/2012.06460.
  * Wang et al. (2021) Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021.  K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 1405–1418, Online. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wu et al. (2022) Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2022.  ESimCSE: Enhanced sample building method for contrastive learning of unsupervised sentence embedding.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 3898–3907, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Zhang et al. (2022) Miaoran Zhang, Marius Mosbach, David Adelani, Michael Hedderich, and Dietrich Klakow. 2022.  MCSE: Multimodal contrastive learning of sentence embeddings.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5959–5969, Seattle, United States. Association for Computational Linguistics.

  *[NLP]: natural language processing
  *[STS]: semantic textual similarity
  *[NLI]: natural language inference
  *[MAP]: Mean Average Precision