# A Dark Flash Normal Camera

Zhihao Xia1 Jason Lawrence2 Supreeth Achar2
1Washington University in St. Louis 2Google Research
Work done while Zhihao Xia was an intern at Google.

###### Abstract

Casual photography is often performed in uncontrolled lighting that can result
in low quality images and degrade the performance of downstream processing. We
consider the problem of estimating surface normal and reflectance maps of
scenes depicting people despite these conditions by supplementing the
available visible illumination with a single near infrared (NIR) light source
and camera, a so-called “dark flash image”. Our method takes as input a single
color image captured under arbitrary visible lighting and a single dark flash
image captured under controlled front-lit NIR lighting at the same viewpoint,
and computes a normal map, a diffuse albedo map, and a specular intensity map
of the scene. Since ground truth normal and reflectance maps of faces are
difficult to capture, we propose a novel training technique that combines
information from two readily available and complementary sources: a stereo
depth signal and photometric shading cues. We evaluate our method over a range
of subjects and lighting conditions and describe two applications: optimizing
stereo geometry and filling the shadows in an image.

##  1 Introduction

In casual mobile photography, images are often captured under poor lighting
conditions. Controlling the visible lighting or supplementing it with a flash
is often too difficult or too disruptive to be practical. On the other hand,
the near infrared (NIR) lighting in a scene can be much more easily controlled
and is invisible to the user. In this paper, we demonstrate how a single “dark
flash” NIR image and a single visible image taken under uncontrolled lighting
can be used to recover high quality maps of the surface normals, diffuse
albedos, and specular intensities in the scene. We collectively refer to the
albedo and specular intensity estimates as a “reflectance map”. Exposing these
signals within a photography pipeline opens up a range of applications from
refining independent depth estimates to digitally manipulating the lighting in
the scene. Although our method is applicable to many types of objects, we
focus on faces - the most common photography subject at the short ranges over
which active illumination is effective.

RGB Input

NIR Input

Normal Map

Figure 1: Estimating surface geometry from a single RGB image is challenging.
We augment this input with a single NIR “dark flash” image captured at the
same time, and present a network that can estimate high quality normal maps
and reflectance maps (not shown) under a wide range of visible lighting
conditions.

Our use of controlled NIR lighting provides a number of benefits. First, the
ambient NIR light in a scene is often weak or completely absent in indoor
environments and is significantly attenuated by atmospheric absorption
outdoors, making it practical to control this aspect of a scene. Second, it
results in a more tractable estimation problem in contrast to single-image
“shape from shading” and intrinsic image decomposition techniques that must
simultaneously reason about shape, material properties, and lighting. Third,
by locating the NIR light source near the camera we can minimize shadows in
the scene which would otherwise complicate this estimation problem. This setup
also produces specular highlights along surfaces that are nearly perpendicular
to the viewing direction, giving a useful cue for determining surface
orientations.

We present a deep neural network that takes as input one RGB image captured
under uncontrolled visible lighting and one monochrome NIR image captured from
the same viewpoint, but under controlled lighting provided by a single source
located near the camera. The network generates a surface normal and
reflectance estimate (diffuse albedo + specular intensity) at each pixel. We
train this network by combining two imperfect but complementary cues: a stereo
depth map that provides a reliable estimate of the low-frequency components of
the scene’s 3d shape along with photometric cues that convey higher-frequency
geometric details. These measurements are far easier to obtain than ground
truth geometry and appearance measurements. We explicitly model the specular
reflectance of human skin in a photometric loss term that guides our training
along with a prior on the albedo map that favors piecewise constant variation
[3].

We compare our technique to a baseline learning approach that uses only a
single RGB image as input and state-of-the-art methods for single image
intrinsic image decomposition [33] and relighting [28]. We are able to produce
overall more stable and more accurate outputs even in very challenging visible
light conditions. We also present two applications of integrating our
technique in a mobile photography pipeline. In all, this paper makes the
following contributions:

  * •

A new network architecture for estimating dense normal and reflectance maps
from a single RGB+NIR image pair.

  * •

A new training strategy that combines two independent and complementary
signals: one from stereo triangulation and the other from photometric cues in
RGB and NIR, along with a hardware setup for collecting this data. Notably,
our training is guided by a physically-based image formation model that
reproduces both diffuse and surface reflectance.

  * •

We demonstrate two applications of our method in a modern photography
pipeline: optimizing depths computed by an independent stereo technique and
reducing shadows in an image post-capture.

##  2 Related Work

Intrinsic imaging and shape from shading. Decomposing a single image into its
underlying shape and reflectance is a classical under-constrained problem in
computer vision [4, 19]. Barron et al. [1, 3] propose a solution, SIRFS, that
applies carefully engineered priors to disambiguate these components.
Learning-based methods have been proposed more recently that train deep neural
networks to perform this task using rendered datasets where ground truth is
available [34, 25] or sparse human annotations [6]. Ye et al. [50] apply
multiview stereo to images of buildings taken at different times and under
different lighting conditions as a source of training data. Whereas some
learning approaches function as “black boxes” [34], others apply a physically-
based image formation model to the network outputs to form a reconstruction
loss [5, 33] and others directly incorporate a rendering layer into the
network [25, 39]. In work concurrent with ours, Qiu et al. [30] explore the
use of a single visible flash image and a similar neural network architecture.
Cheng et al. [12] use an RGB+NIR image pair, although they don’t attempt to
control the NIR lighting in the scene as we do. Yoon et al. [49] train a
network that accepts a single NIR image using photometric normals computed
from a multi-point-NIR-light dataset. In contrast, our network considers both
a visible and front-lit NIR image and we use two independent signals to
supervise the training.

A number of methods are specifically designed to work on images of faces. This
includes 3D morphable models [8], which are commonly used as a prior on
reflectance and geometry in learning-based approaches [40, 35, 33]. Sanyal et
al. [32] estimate the shape of a face within a single image in the form of
blending weights over a parametric face model. Similar to our approach, other
techniques estimate dense normal or displacement maps [51] including for faces
partially hidden by occluders [41, 14]. However these methods do not attempt
to disentangle reflectance data from shading. Some portrait relighting
techniques estimate surface normal and albedo maps as intermediate signals
that guide a relighting process [28].

Distinct from these prior techniques is our use of a single front-lit NIR
image in addition to a color input image, enabling our technique to perform
well even in very harsh and low visible light conditions. We compare to a
baseline network that uses only a single RGB image as input in Section 5.1.

Figure 2: Our network learns to estimate shape and reflectance from a single
front-lit NIR image, a single RGB image under arbitrary lighting, and a
semantic segmentation map computed from the RGB image (inputs are enclosed by
the red line). During training we also use a stereo depth map and replace the
RGB image under arbitrary lighting with 4 RGB+NIR image pairs captured under
calibrated point lights (the training inputs are inside the blue dashed line).

Monocular depth estimation. Our work is also related to the growing number of
techniques that estimate depth from a single image [7]. Deep neural networks
have proven effective at performing this task when trained using ground truth
depth data [15, 23, 16, 45]. More recent work has focused on self-supervised
training methods to overcome the difficulty of obtaining ground truth data
[54, 48, 52, 17]. Sun et al. [36] explored the use of a monocular infrared
image for depth estimation and Li et al. [24] combined color and infrared
images for depth estimation in the context of autonomous driving. Despite the
progress in this field, state-of-the-art monocular depth estimation methods do
not yet achieve the same level of accuracy as shape estimation methods that
use some form of active lighting and constrain the input scene.

Fusing depth and normals. Depth estimated from methods like stereo
triangulation and normals estimated from shading cues are complementary
measurements for shape recovery. Nehab et al. [27] describe a technique that
seeks to combine the more accurate low-frequency information provided by
direct depth measurement techniques with the higher-frequency geometric
details provided by photometric measurements. We use their technique to
evaluate how our approach could be used to improve a stereo pipeline (Section
5.2). More recent work poses this as an optimization problem that seeks a
surface that best agrees with these different signals [2, 13, 48, 10, 26]. Our
training method is similar to these approaches in that we also combine a
stereo and photometric loss term.

Face relighting. Most single image face relighting methods include some
representation of shape and reflectance as intermediate components. Our
network architecture (Section 3) is similar to the one proposed by Nestmeyer
et al. [28] for simulating lighting changes in a single image assumed to have
been captured under a single directional light. Zhou et al. [53] present a
dataset of relit portrait images generated using single-image normal and
illumination estimates and a Lambertian reflectance model and then train a
network with an adversarial loss to close the appearance gap between these
synthetic images and real face images. Although surface geometry is
fundamental to relighting, it is also possible to train an end-to-end network
that does not explicitly reason about shape [37]. We similarly use multiple
images of a scene captured under varying controlled lighting to train our
network in order to enable a much simpler set of inputs for inference.

Combining infrared and color imagery. A NIR (and/or ultraviolet) dark flash
image can be used to denoise a color image captured in low visible light
conditions [22], or serve as a guide for correcting motion blur [47].
Techniques have also been developed that employ controlled NIR lighting to
simulate better visible lighting in real-time video communication systems [43,
18]. We see these as compelling potential applications of this work.

Figure 3: Illustration of our network and training strategy. We estimate
network weights that minimize a photometric loss, computed between images
rendered from our network outputs and ground truth images captured under known
lighting, and a stereo loss, driven by differences between the output normals
and those estimated using an independent stereo technique.

##  3 Network Design and Training

Our goal is to estimate a normal map and a reflectance map from a single RGB
image and a front-lit “dark flash” NIR image. We train a deep neural network
to perform this task. As an auxiliary input, we use a 6-class semantic
segmentation map computed from the RGB image (background, head, hair, body,
upper arm and lower arm) [11]. We found this segmentation map was a useful cue
for helping the network reason about shape and reflectance. An example set of
inputs are shown in Figure 2 (red line).

Our training procedure is driven in part by a physically-based image formation
model that connects the outputs of our network to images of a scene taken
under known point lighting. This image formation model combines a standard
Lambertian diffuse term with the Blinn-Phong BRDF [9], which has been used to
model the specular reflectance of human skin [44]. Specifically, we introduce
a reflectance function $f$ that gives the ratio of reflected light to incident
light for a particular unit-length light vector $\mathbf{l}$, view vector
$\mathbf{v}$, surface normal $\mathbf{n}$, four-channel (RGB+NIR) albedo
$\boldsymbol{\alpha}$, scalar specular intensity $\rho$, and specular exponent
$m$:

$f(\mathbf{l},\mathbf{v},\mathbf{n})=\boldsymbol{\alpha}+\rho\frac{m+2}{2\pi}(\mathbf{n}\cdot\mathbf{h})^{m},$
(1)

where
$\mathbf{h}=(\mathbf{n}+\mathbf{l})/\mathinner{\\!\left\lVert\mathbf{n}+\mathbf{l}\right\rVert}$.
The observed intensity at a pixel due to a point light is given by

$I(\cdot)=f(\mathbf{l},\mathbf{v},\mathbf{n})(\mathbf{n}\cdot\mathbf{l})L,$
(2)

the product of the reflectance, cosine term, and light intensity $L$. We do
not observe the reflected intensity from enough unique light directions at
each pixel to estimate all of the parameters in Equation 1. We therefore fix
the specular exponent to $m=30$ based on prior measurements of human skin [44]
and our own observations, and estimate only $\mathbf{n}$,
$\boldsymbol{\alpha}$, and $\rho$. The geometric quantities $\mathbf{l}$ and
$\mathbf{v}$, and light intensity $L$ are determined by the calibration
procedures described in Section 4.

Illustrated in Figure 3, we use a standard UNet with skip connections [31].
The encoder and decoder each consist of 5 blocks with 3 convolutional layers
per block. The bottleneck has 256 channels. The output of this UNet is
forwarded to two separate networks: a geometry branch that predicts a normal
map $\tilde{\mathbf{n}}$, and a reflectance branch that predicts an albedo map
$\tilde{\boldsymbol{\alpha}}$ and log-scale specular intensity map,
$log(\tilde{\rho})$. Both branches have 3 convolutional layers with 32
channels and one final output layer.

We do not rely on ground truth normals or reflectance data to supervise
training. Instead we combine a stereo loss and a photometric loss derived from
data that is far easier to obtain: four one-light-at-a-time (OLAT) images in
both RGB and NIR of the same subject, in the same exact pose, illuminated by a
set of calibrated lights activated individually in rapid succession, and a
stereo depth map (blue dashed line in Figure 2). These images are only used at
training time.

A stereo loss encourages our estimated normals $\tilde{\mathbf{n}}$ to agree
with the gradients of a smoothed version of the stereo depth map
$\mathbf{n}_{s}$. Specifically, it combines a L1 vector loss and angular loss:

$\mathcal{L}_{s}(\tilde{\mathbf{n}})=\|\tilde{\mathbf{n}}-\mathbf{n}_{s}\|_{1}-(\tilde{\mathbf{n}}\cdot\mathbf{n}_{s}).$
(3)

A photometric loss is computed between each of the OLAT images and an image
rendered according to Equation 2 and our network outputs for the corresponding
lighting condition:

$\mathcal{L}_{p}^{j}(\tilde{\mathbf{n}},\tilde{\boldsymbol{\alpha}},\tilde{\rho})=\mathinner{\\!\left\lVert
S_{j}\odot\left(I(\mathbf{l}_{j},\mathbf{v},\tilde{\mathbf{n}},\tilde{\boldsymbol{\alpha}},\tilde{\rho})-\hat{I}_{j}\right)\right\rVert}_{1},$
(4)

where $\hat{I}_{j}$ is the per-pixel color observed in the $j^{\mathrm{th}}$
OLAT image, and $S_{j}$ is a binary shadow map, computed using the stereo
depth and calibrated light position (Section 4). We also apply a prior to the
albedo map that encourages piecewise constant variation [3]:

$\mathcal{L}_{c}(\tilde{\boldsymbol{\alpha}})=\sum_{i}\sum_{j\in\mathcal{N}(i)}\lVert\tilde{\boldsymbol{\alpha}}_{i}-\tilde{\boldsymbol{\alpha}}_{j}\rVert_{1},$
(5)

where $\mathcal{N}(i)$ is the $5\times 5$ neighborhood centered at pixel $i$.
We apply this prior only to clothing pixels, those labeled as either body or
arms in the segmentation mask. We found that other regions in the scene did
not benefit from this regularization.

Our total loss function is a weighted sum of these terms:

$\mathcal{L}(\tilde{\mathbf{n}},\tilde{\boldsymbol{\alpha}},\tilde{\rho})=\\\
\mathcal{L}_{s}(\tilde{\mathbf{n}})+\lambda_{p}\sum_{j}{\mathcal{L}_{p}^{j}(\tilde{\mathbf{n}},\tilde{\boldsymbol{\alpha}},\tilde{\rho})}+\lambda_{c}\mathcal{L}_{c}(\tilde{\boldsymbol{\alpha}}).$
(6)

We set the weight $\lambda_{p}$ to $10$ and $\lambda_{c}$ to $50$.

#### Data Augmentation and Training.

To improve the robustness of our network, we apply a series of data
augmentations to our captured OLATs to simulate a variety of different visible
light conditions. Specifically, our training uses a combination of: evenly-lit
RGB inputs obtained by adding together all of the OLAT images; inputs with
strong shadows by selecting exactly one of the OLAT images; a mixture of two
lights with different temperatures by applying randomly chosen color vectors
to two randomly chosen OLAT images; low-light environments by adding Gaussian
noise to a single OLAT; and saturated exposures by scaling and clipping a
single OLAT. We sample evenly from these 5 lighting conditions during
training. Further details on how these lighting conditions are simulated are
provided in the supplementary material.

We train the network using the Adam optimizer [21] for $30$K iterations, with
a learning rate of $10^{-3}$ and a batch size of $8$. Training takes around
$12$ hours on a machine with 4 Tesla V100 GPUs.

##  4 Hardware Setup and Data Collection

Shown in Figure 4, our setup combines a $7.0$MP RGB camera that operates at
$66.67$ fps with a stereo pair of $2.8$MP NIR cameras that operate at $150$
fps. The RGB camera and one of the NIR cameras are co-located using a plate
beamsplitter and a light trap. The RGB and NIR cameras have a linear
photometric response and we downsample all of the images by a factor of 2 in
each dimension and take a central crop that covers the face at a resolution of
$960\times 768$.

Visible spectrum lighting is provided by 4 wide-angle LED spotlights placed at
the corners of a roughly $1.5m\times 0.8m$ (width x height) rectangle
surrounding the cameras located approximately $1.1$m from the subject. NIR
lighting is provided by 5 NIR spotlights, one adjacent to each of the visible
lights, and a flash LED light located near the reference NIR camera to produce
the “dark flash” input. These NIR light sources are temporally interleaved
with projectors that emit NIR dot speckle patterns to assist stereo matching
[29]. A microcontroller orchestrates triggering the lights and cameras to
ensure that at any time only one visible light source and one NIR light source
is active. All light sources are calibrated for position and intensity and
treated geometrically as point light sources. The light intensity term $L$ in
Equation 2 accounts for these calibrated colors. Note that the NIR and visible
light sources are not colocated and so slightly different values of
$\mathbf{l}$ are used in Equation 2 between those two conditions.

Figure 4: Our hardware setup consists of controllable NIR and visible spectrum light sources, an RGB camera, a stereo pair of NIR cameras, and two NIR dot projectors. One of the NIR cameras and the RGB camera are aligned with a beamsplitter and all of these components are triggered electronically to record the types of images shown in Figure 2. Ablation | Well lit | Shadows | Mixed colors | Overexposure | Low light
---|---|---|---|---|---
Loss | No Stereo Loss | 12.80 | 12.78 | 12.78 | 12.82 | 12.81
No NIR Photometric Loss | 12.64 | 12.66 | 12.64 | 12.69 | 12.75
No Photometric Loss | 12.77 | 12.77 | 12.81 | 12.79 | 12.77
Reflectance Model | No Specular Component | 12.44 | 12.43 | 12.44 | 12.51 | 12.47
Inputs | No RGB Input | 12.54 | 12.54 | 12.54 | 12.54 | 12.54
No NIR Input | 13.13 | 15.19 | 16.43 | 19.82 | 19.39
Ours (Full Method) | 12.08 | 12.06 | 12.06 | 12.14 | 12.10
Table 1: Mean absolute angular error in degrees of normal maps computed with modified versions of our full network. Results are reported for the five lighting conditions described in Section 5. |  |  |  |
---|---|---|---|---
RGB Input | Shading | Shading | Albedos | Albedos
| (w/o Photometric Loss) | (w/ Photometric Loss) | (w/o Blinn-Phong) | (w/ Blinn-Phong)
Figure 5: Impact of the photometric loss term in our training procedure and
the Blinn-Phong BRDF in our image formation model, respectively. When trained
without photometric loss, our network learns to output the stereo normals,
which lack fine-scale details. This has a fairly small effect on the error
measures in Table 1, but is perceptually significant as seen in these “n dot
l” shading renderings. Our full image formation model, which includes a Blinn-
Phong specular term, produces more accurate albedos across the face than using
a Lambertian model alone.

The image acquisition rate is limited by the RGB camera’s framerate and the
total light output, but is fast enough for us to record video sequences of
people who are gesturing and moving slowly. We compute optical flow [46]
between consecutive frames captured under the same lighting condition to
correct for the small amount of scene motion that occurs within a single round
of exposures. Since the RGB and reference stereo NIR camera are co-located, we
can generate pixel-aligned RGB, NIR, and depth images using scene-independent
precomputed image warps.

Each recording in our dataset is $10$ seconds long and contains $166$ sets of
frames. We recorded $9$ unique subjects, with between $5$ and $10$ sessions
per subject, for a total of $61$ recordings. We used recordings of 6 of the
subjects for training and tested on recordings of the other 3.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
RGB Input | Normals | Albedos | NIR Input | Normals | Albedos
| (RGB Only) | (RGB Only) |  | (Ours) | (Ours)
Figure 6: Comparison of our network to a modified version that takes only a
single RGB image (“RGB Only”) as input. Example results for three common
challenging lighting conditions. Top to bottom: low light / noisy inputs;
mixed light colors; harsh directional lighting with saturated intensities. The
“RGB only” network struggles to produce stable normal and reflectance
estimates from these inputs in contrast to our method.

##  5 Evaluation

We evaluate our method through several ablation studies that measure the
impact of key design decisions, including an in-depth comparison to a modified
network that takes only a single RGB image as input. We also present two
applications of our technique. Please see our supplemental video for results
on image sequences and our supplemental document for additional comparisons to
SfSNet [33] and the face relighting network of Nestmemery et al. [28].

In our evaluations we consider five different visible lighting conditions:
harsh lighting that produces strong cast shadows; a mixture of lights with
different color temperatures; saturated/overexposed intensities; low-light
conditions that produce noisy inputs; and a “well lit” condition that achieves
largely shadow-free and well exposed inputs. Our process for synthesizing
these different lighting conditions from the OLAT training images is detailed
in our supplemental document. All of our results were produced using these
five lighting conditions with the exception of Figure 1, which shows the
performance of our method in lighting conditions that do not lie in the span
of our OLAT images. That sequence was captured by casually moving a handheld
light source around the scene. Please see our supplemental video for an
expanded version of that result. None of the subjects shown in any of our
results were included in our training set.

In lieu of ground truth geometry for quantitative assessments, we construct a
baseline using the technique of Nehab et al. [27] to refine our stereo depth
maps according to normals computed by applying Lambertian photometric stereo
to the RGB OLAT training images.

###  5.1 Ablation Studies

Table 1 reports the mean absolute angular errors with respect to our baseline
for normal maps computed with variants of our network that have different loss
terms, image formation models, and inputs. Figures 5 and 6 show examples of
the perceptual impact of some of these design decisions.

Loss terms. As expected, using both stereo and photometric loss terms during
training outperforms using either one alone. We consider two types of
photometric loss - one computed on only the RGB training images (“No NIR
Photometric Loss” in Table 1) and the second computed on both the NIR and RGB
training images (“Full Method”). As illustrated in the shading images in
Figure 5, including the photometric loss enables estimating fine geometric
details that are not captured in the stereo depth maps.

Image formation model. Including the Blinn-Phong BRDF in our image formation
model improves the accuracy of the normals and diffuse albedo maps. It results
in a modest improvement in the quantitative errors in Table 1, and it produces
more uniform diffuse albedo maps with fewer artifacts (Figure 5). We attribute
this to the fact that this richer image formation model is better able to
explain the observed intensities. We also found that including this BRDF in
our model enables reconstructing the glossy appearance of skin (Section 5.3).

Network inputs. Including the NIR input image improves accuracy across the
board, especially in poor visible lighting conditions (Table 1). The benefit
of the RGB input is comparatively smaller, but making it available to the
network enables estimating visible spectrum reflectance data, which is a
requirement for many downstream applications such as lighting adjustment
(Section 5.3). Figure 6 illustrates the perceptual impact of including the NIR
input in different lighting conditions. For these comparisons we modified our
network to take only a single RGB image as input (“RGB Only”). The network
architecture was otherwise unchanged, and we applied the same training
procedure described in Section 3. Note how the performance of this “RGB Only”
network significantly degrades in challenging conditions, while our method is
far more robust to these conditions due to the more stable NIR input. It’s
particularly noteworthy how well our method is able to reconstruct plausible
diffuse albedos even for highly saturated RGB input images (bottom row of
Figure 6).

Please see our supplemental document for an expanded version of Figure 6 and
our supplemental video that includes comparisons for animated image sequences.

###  5.2 Application: Stereo Refinement

|
---|---
|
Smoothed Stereo | Refined Stereo (Ours)
Figure 7: Stereo methods often struggle to recover fine-scale surface details.
_Left:_ Applying a guided bilateral filter to raw stereo depths yields a
smoother surface but with distorted features (e.g. the nose is reduced and
skin wrinkles are missing). _Right:_ We use the method of Nehab et al. [27] to
compute a refined surface according to normals estimated with our method. Note
how details are better preserved around the eyes, nose, and mouth, along with
fine wrinkles and creases.

Stereo methods excel at measuring coarse geometry, but often struggle to
recover fine-scale surface details. This can be overcome by refining stereo
depths according to accurate high-resolution normals typically estimated with
a photometric approach [27]. We evaluate using the normals produced by our
method to refine depth measurements produced by an NIR space-time stereo
algorithm [29] (Figure 7). In comparison to using a standard bilateral filter
to smooth the stereo depths, refining them using our normals gives much higher
quality reconstructions, most notably around the mouth, nose, and eyes and
better recovery of fine wrinkles and creases in the skin. As our method works
with a single NIR image it would be straightforward to integrate it into many
existing stereo pipelines.

###  5.3 Application: Lighting Adjustment

|  |
---|---|---
|  |
RGB Input | Relit | Ground Truth
Figure 8: Our method can be used to simulate adding lights to a scene to fill
in shadows.

We also explored using our approach to digitally improve the lighting in a
portrait. Specifically, we evaluated adding a virtual fill light to brighten
shadowed parts of the face (Figure 8). We used normal and reflectance maps
estimated by our method to render the contribution of a virtual point light
located within view of the shadowed region, and then combined this with the
original RGB image. Our model enables a convincing effect, even producing
realistic specular highlights along the nasolabial folds and the tip of the
nose.

##  6 Discussion and Future Work

We observed that the specular intensity maps produced by our method are not
accurate across the entire face due to the sparse nature of our training
images – the specular component of the material appearance is simply not
observed in many parts of the scene. We believe this could be overcome by
including additional unique light positions in our training data or by
introducing priors on this component.

We record co-axial color and NIR images using two cameras aligned with a
beamsplitter. Future work could explore more practical hardware setups that
potentially use integrated RGB+NIR image sensors [38, 20] or RGB+NIR stereo
setups that would allow placing the NIR and color cameras next to each other
at a small baseline [42]. This line of work could take advantage of the
growing availability of NIR light sources and NIR sensors in modern
smartphones.

Our approach assumes a single light located near the camera is the only source
of NIR light in the scene. Although this is a safe assumption in many indoor
environments, it is not always true, especially outdoors. It would be
interesting to explore overcoming ambient NIR light, possibly through the use
of flash/no-flash image pairs.

Another area of future work is improving our method’s performance on hair and
clothing. An effective strategy would be to expand the training data to
include a larger number of subjects and more light positions in the OLAT
training sequences. Finally, there is room to improve the temporal stability
of our method by having the network explicitly consider consecutive frames.
Please see the supplemental video that includes results for image sequences.

##  7 Conclusion

We have presented a dark flash normal camera that is capable of estimating
high-quality normal and reflectance maps from a single RGB+NIR input image
that can be recorded in a single exposure without distracting the subject. A
key benefit of our method over prior work is its robustness. It performs well
even in challenging lighting conditions that are commonly encountered in
casual photography such as harsh shadows, saturated pixels, and in very low
light environments. Our method could easily be integrated into existing
smartphone camera hardware designs and software pipelines to enable a range of
applications from refining the output of an auxiliary depth camera to
improving the lighting of faces in still images and streaming video.

## References

  * [1] Jonathan T Barron and Jitendra Malik.  Shape, albedo, and illumination from a single image of an unknown object.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 334–341. IEEE, 2012.
  * [2] Jonathan T. Barron and Jitendra Malik.  Intrinsic scene properties from a single rgb-d image.  CVPR, 2013.
  * [3] Jonathan T. Barron and Jitendra Malik.  Shape, illumination, and reflectance from shading.  TPAMI, 2015.
  * [4] H. Barrow and J. M. Tenenbaum.  Recovering intrinsic scene characteristics from images.  1978\.
  * [5] Anil S. Baslamisli, Hoang-An Le, and Theo Gevers.  Cnn based learning using reflection and retinex models for intrinsic image decomposition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [6] Sean Bell, Kavita Bala, and Noah Snavely.  Intrinsic images in the wild.  ACM Transactions on Graphics (SIGGRAPH), 33(4), 2014.
  * [7] Amlaan Bhoi.  Monocular depth estimation: A survey.  arXiv preprint arXiv:1901.09402, 2019.
  * [8] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’99, page 187–194, USA, 1999\. ACM Press/Addison-Wesley Publishing Co.
  * [9] James F. Blinn.  Models of light reflection for computer synthesized pictures.  In Proceedings of the 4th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’77, page 192–198, New York, NY, USA, 1977\. Association for Computing Machinery.
  * [10] Xu Cao, Michael Waechter, Boxin Shi, Ye Gao, Bo Zheng, and Yasuyuki Matsushita.  Stereoscopic flash and no-flash photography for shape and albedo recovery.  pages 3427–3436, 06 2020.
  * [11] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  CoRR, abs/1706.05587, 2017.
  * [12] Ziang Cheng, Yinqiang Zheng, Shaodi You, and Imari Sato.  Non-local intrinsic decomposition with near-infrared priors.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2521–2530, 2019.
  * [13] Gyeongmin Choe, Jaesik Park, Yu-Wing Tai, and In Kweon.  Exploiting shading cues in kinect ir images for geometry refinement.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3922–3929, 2014.
  * [14] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [15] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  In Advances in neural information processing systems, pages 2366–2374, 2014.
  * [16] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao.  Deep ordinal regression network for monocular depth estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2002–2011, 2018.
  * [17] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow.  Digging into self-supervised monocular depth estimation.  In Proceedings of the IEEE international conference on computer vision, pages 3828–3838, 2019.
  * [18] Prabath Gunawardane, Tom Malzbender, Ramin Samadani, Alan McReynolds, Dan Gelb, and James Davis.  Invisible light: Using infrared for video conference relighting.  In 2010 IEEE International Conference on Image Processing, pages 4005–4008. IEEE, 2010.
  * [19] Berthold K. P. Horn.  Obtaining Shape from Shading Information, page 123–171.  MIT Press, Cambridge, MA, USA, 1989.
  * [20] X. Hu, F. Heide, Q. Dai, and G. Wetzstein.  Convolutional sparse coding for rgb+nir imaging.  IEEE Transactions on Image Processing, 27(4):1611–1625, 2018.
  * [21] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [22] D. Krishnan and R. Fergus.  Dark flash photography.  ACM Transactions on Graphics, SIGGRAPH 2009 Conference Proceedings, 28:To appear, 2009.
  * [23] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab.  Deeper depth prediction with fully convolutional residual networks.  In International Conference on 3D Vision (3DV), pages 239–248, 2016\.
  * [24] Yuqi Li, Haitao Zhao, Zhengwei Hu, Qianqian Wang, and Yuru Chen.  IVFuseNet: Fusion of infrared and visible light images for depth prediction.  Information Fusion, 58:1 – 12, 2020.
  * [25] Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker.  Learning to reconstruct shape and spatially-varying reflectance from a single image.  ACM Trans. Graph., 37(6), Dec. 2018.
  * [26] Zhe Liang, Chao Xu, Jing Hu, Yushi Li, and Zhaopeng Meng.  Better together: Shading cues and multi-view stereo for reconstruction depth optimization.  IEEE Access, PP:1–1, 06 2020.
  * [27] Diego Nehab, Szymon Rusinkiewicz, James Davis, and Ravi Ramamoorthi.  Efficiently combining positions and normals for precise 3d geometry.  In ACM SIGGRAPH 2005 Papers, SIGGRAPH ’05, page 536–543, New York, NY, USA, 2005. Association for Computing Machinery.
  * [28] Thomas Nestmeyer, Jean-François Lalonde, Iain Matthews, and Andreas M Lehrmann.  Learning physics-guided face relighting under directional light.  In Conference on Computer Vision and Pattern Recognition, pages 5123–5132. IEEE/CVF, June 2020.
  * [29] Harris Nover, Supreeth Achar, and Dan B Goldman.  ESPReSSo: Efficient slanted patchmatch for real-time spacetime stereo.  In Proceedings of the International Conference on 3D Vision (3DV), 2018.
  * [30] Di Qiu, Jin Zeng, Zhanghan Ke, Wenxiu Sun, and Chengxi Yang.  Towards geometry guided neural relighting with flash photography.  arXiv preprint arXiv:2008.05157, 2020.
  * [31] O. Ronneberger, P.Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention (MICCAI), volume 9351 of LNCS, pages 234–241. Springer, 2015.  (available on arXiv:1505.04597 [cs.CV]).
  * [32] Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael Black.  Learning to regress 3d face shape and expression from an image without 3d supervision.  pages 7755–7764, 06 2019.
  * [33] Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, and David W. Jacobs.  Sfsnet: Learning shape, refectance and illuminance of faces in the wild.  In Computer Vision and Pattern Regognition (CVPR), 2018.
  * [34] Jian Shi, Yue Dong, Hao Su, and Stella X. Yu.  Learning non-lambertian object intrinsics across shapenet categories.  CoRR, abs/1612.08510, 2016.
  * [35] Z. Shu, E. Yumer, S. Hadap, K. Sunkavalli, E. Shechtman, and D. Samaras.  Neural face editing with intrinsic image disentangling.  In Computer Vision and Pattern Recognition, 2017. CVPR 2017. IEEE Conference on, pages –. IEEE, 2017.
  * [36] Shaoyuan Sun, Linna Li, and Lin Xi.  Depth estimation from monocular infrared images based on bp neural network model.  In International Conference on Computer Vision in Remote Sensing, pages 237–241, 2012.
  * [37] Tiancheng Sun, Jonathan T. Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul Debevec, and Ravi Ramamoorthi.  Single image portrait relighting.  ACM Trans. Graph., 38(4), July 2019.
  * [38] Huixuan Tang, Xiaopeng Zhang, Shaojie Zhuo, Feng Chen, Kiriakos N Kutulakos, and Liang Shen.  High resolution photography with an rgb-infrared camera.  In 2015 IEEE International Conference on Computational Photography (ICCP), pages 1–10. IEEE, 2015.
  * [39] Tatsunori Taniai and Takanori Maehara.  Neural inverse rendering for general reflectance photometric stereo.  In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4864–4873. PMLR, 2018\.
  * [40] Ayush Tewari, Michael Zollöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Theobalt Christian.  MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction.  In The IEEE International Conference on Computer Vision (ICCV), 2017\.
  * [41] Anh Tran, Tal Hassner, Iacopo Masi, Eran Paz, Yuval Nirkin, and Gerard Medioni.  Extreme 3d face reconstruction: Seeing through occlusions.  pages 3935–3944, 06 2018.
  * [42] Jian Wang, Tianfan Xue, Jonathan T Barron, and Jiawen Chen.  Stereoscopic dark flash for low-light photography.  In 2019 IEEE International Conference on Computational Photography (ICCP), pages 1–10. IEEE, 2019.
  * [43] Oliver Wang, James Davis, Erika Chuang, Ian Rickard, Krystle De Mesa, and Chirag Dave.  Video relighting using infrared illumination.  In Computer Graphics Forum, volume 27, pages 271–279. Wiley Online Library, 2008.
  * [44] Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho Lee, Addy Ngan, Henrik Wann Jensen, and Markus Gross.  Analysis of human faces using a measurement-based skin reflectance model.  ACM Trans. on Graphics (Proc. SIGGRAPH 2006), 25(3):1013–1024, 2006\.
  * [45] Zhihao Xia, Patrick Sullivan, and Ayan Chakrabarti.  Generating and exploiting probabilistic monocular depth estimates.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 65–74, 2020.
  * [46] Jiangjian Xiao, Hui Cheng, Harpreet Sawhney, Cen Rao, and Michael Isnardi.  Bilateral filtering-based optical flow estimation with occlusion detection.  In Ales Leonardis, Horst Bischof, and Axel Pinz, editors, Computer Vision – ECCV 2006, pages 211–224, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
  * [47] H. Yamashita, D. Sugimura, and T. Hamamoto.  Rgb-nir imaging with exposure bracketing for joint denoising and deblurring of low-light color images.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6055–6059, 2017.
  * [48] Zhenheng Yang, Peng Wang, Wei Xu, Liang Zhao, and Ramakant Nevatia.  Unsupervised learning of geometry with edge-aware depth-normal consistency.  arXiv preprint arXiv:1711.03665, 2017.
  * [49] Youngjin Yoon, Gyeongmin Choe, Namil Kim, Joon-Young Lee, and In So Kweon.  Fine-scale surface normal estimation using a single NIR image.  In European Conference on Computer Vision, pages 486–500. Springer, 2016.
  * [50] Ye Yu and William A. P. Smith.  InverseRenderNet: Learning single image inverse rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [51] Xiaoxing Zeng, Xiaojiang Peng, and Yu Qiao.  DF2Net: A dense-fine-finer network for detailed 3d face reconstruction.  In International Conference on Computer Vision, ICCV, pages 2315–2324. IEEE, 2019.
  * [52] Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian Reid.  Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 340–349, 2018.
  * [53] Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David W. Jacobs.  Deep single-image portrait relighting.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [54] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe.  Unsupervised learning of depth and ego-motion from video.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1851–1858, 2017.

Supplementary Material

##  Appendix A Visible Lighting Conditions Used in Our Evaluation

We evaluate our method using five different visible lighting conditions that
range from favorable (“well lit”) to challenging (“shadows”, “mixed colors”,
“overexposure”, and “low light”). Figure 9 illustrates and details how these
different lighting conditions are simulated from the RGB OLAT training images.

|  |  |  |
---|---|---|---|---
(a) Well lit | (b) Shadows | (c) Mixed colors | (d) Overexposure | (e) Low light
Figure 9: The five visible lighting conditions used in our evaluations. (a) A
well lit image is created by averaging equally the four OLAT RGB images, using
weights that avoid any under- or over-saturated pixels. (b) Picking a single
OLAT image at random produces an RGB input with strong cast shadows. (c) We
simulate mixing different colored lights by picking two OLAT images at random
and remapping the color of each to a random color temperature in the range
$[1900K,2900K]$ and $[7000K,20000K]$, respectively, and then averaging them
together. (d) Scaling the intensities of one of the OLAT images with a random
scale factor in the range $[1.8,2.3]$ and then clipping the result yields an
image with harsh lighting and saturated intensities. (e) Adding Gaussian white
noise to each pixel ($\sigma=25$ 8-bit gray levels) simulates images captured
under low light conditions.

##  Appendix B Additional Ablation Studies

The graphs in Figure 10 expand on the results shown in Figure 6 in the paper.
They plot the mean angular error of the estimated normals against the baseline
for three of our five lighting scenarios, and for three different techniques:
our network modified to take only a single NIR input image (“NIR Only”), our
network modified to take only a single RGB input image (“RGB Only”), and our
full network, which considers both. In each graph the magnitude of the
lighting issue increases from left to right.

Variations in illuminant color have a relatively small effect on the normals
estimated by the “RGB only” network, but overexposed pixels and image noise
cause large errors. As one would expect the “NIR only” network is invariant to
these lighting changes (flat line in these graphs), and our proposed approach
consistently outperforms both.

|  |
---|---|---
Figure 10: Mean absolute angular errors in degrees of normal maps estimated by
three different techniques in three challenging lighting conditions as the
magnitude of each lighting issue increases from left to right. A similar
network that uses only a single RGB image to estimate normals (RGB Only) and
one that uses only a single NIR image (NIR Only) both perform worse that our
method (RGB + NIR), with the RGB Only network deteriorating rapidly alongside
the lighting issue. _Left:_ harsh lighting that produces cast shadows with
increasing image exposures. _Center:_ mixing lights with different color
temperatures. _Right:_ increasing levels of noise, which occurs in low-light
conditions.

##  Appendix C Comparisons to Prior Single (RGB) Image Normal Estimation
Methods

To the best of our knowledge, our method is the first technique for estimating
surface normals and RGB albedos from a _single_ RGB+NIR image. While the “RGB
Only” baseline in the main paper demonstrates the effectiveness of
supplementing the available visible illumination with a dark flash NIR image,
we now provide a comparison to state-of-the-art single RGB image normal
estimation methods from the literature. We compare our method to two recent
techniques:

#### SfSNet [33].

For a fair comparison, we retrain SfSNet on our dataset. We use the same basic
network architecture as our method, but with only a single RGB image as input
and with an additional lighting estimation branch. This lighting estimation
branch takes the output of the encoder network and generates an estimate of
the lighting in the input RGB image. SFSNet uses second order spherical
harmonics to represent the scene lighting, which isn’t well suited for
approximating the type of point lighting in our dataset. Therefore we instead
represent the lighting as a blending weight vector over the four visible
OLATs. The lighting estimation branch is supervised using the ground truth
OLAT mixing weights of the input RGB image. The normal branch of SfSNet is
supervised using the same stereo normals and same image reconstruction loss
that we use for our method.

#### Directional Face Relighting Network [28].

We also compare our method to the intrinsic component estimation stage of
Nestmeyer et al. [28], which uses a UNet to predict a normal and albedo map
from a single RGB image. Since we assume that the lighting conditions in the
input RGB image are unknown, we do not provide the source lighting direction
to the network. We retrain their network on our dataset by supervising the
normal estimation network path with the same stereo normals used to train our
method, and by using one of the RGB OLAT images chosen at random as the
relighting target image. Note that their instrinsic component estimation stage
does not consider cast shadows.

Table 2 reports quantitative errors in the normals estimated using all three
techniques for the different lighting conditions described in Section A. Our
method outperforms both techniques even in the well lit condition, which we
attribute to our novel training strategy that combines shape information from
complementary stereo and photometric signals, and the additional information
provided by the NIR input. In challenging lighting conditions, the benefit of
our method becomes more significant. Note that SfSNet [33] uses a self-
reconstruction loss that we found could not handle inputs with mixed color
casts, saturated intensities, or a significant amount of noise and so it fails
to produce plausible outputs in these cases (omitted from Table 2).

Method | Well lit | Shadows | Mixed colors | Overexposure | Low light
---|---|---|---|---|---
SfSNet [33] | 14.10 | 18.32 | - | - | -
Nestmeyer et al.[28] | 14.82 | 17.52 | 15.87 | 21.85 | 25.56
Ours | 12.29 | 12.27 | 12.28 | 12.30 | 12.33
Table 2: Mean absolute angular errors in degrees for three normal estimation
methods and five different lighting scenarios. Results for SfSNet [33] are
omitted in some cases where they fail to generate plausible normals.

##  Appendix D Expanded Stereo Refinement Results

Figure 11 expands on Figure 7 in the paper. Specifically it shows the input
images and the normal map estimated by our method, along with the raw stereo
depths prior to any smoothing or refinement.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
RGB Input | NIR Input | Normals | Raw Stereo | Smoothed | Refined (ours)
Figure 11: Stereo methods often struggle to recover fine-scale surface
details. Applying a guided bilateral filter to raw stereo depths yields a
smoother surface but with distorted features (e.g. the nose appears pinched
and reduced). We use the method of Nehab et al. [27] to compute a refined
surface according to normals estimated with our method. Note how this better
preserves details around the eyes, nose, and mouth, along with fine wrinkles
and creases.

##  Appendix E Expanded Lighting Adjustment Results

Figure 12 expands on Figure 8 in the paper. Specifically it shows the
supplemental rendered image that is combined with the input RGB image in order
to brighten shadowed regions along the face. We also compare the difference
between using a strictly Lambertian image formation model and our full
Lambertian-plus-specular model in generating the supplemental image that is
combined with the input. Note that our full model does a better job at
reproducing specular highlights along the cheek and tip of the nose.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
RGB Input | NIR Input | Virtual Light | Virtual Light | Relit | Ground Truth
|  | (w/o Blinn-Phong) | (w/ Blinn-Phong) |  |
Figure 12: Our method can be used to simulate adding lights to a scene to fill
in shadows. We show this virtual light image generated using a Lambertian
reflectance model alone and using our full Lambertian + Blinn-Phong model,
which produces more realistic highlights. When combined with the input RGB
image (Relit) this approach compares favorably to ground truth.

##  Appendix F Additional Video Results

Please see our supplemental video for results and comparisons on image
sequences.


# Sparse Optimization for Unsupervised Extractive Summarization of Long
Documents with the Frank-Wolfe Algorithm

Alicia Y. Tsai
University of California, Berkeley
<EMAIL_ADDRESS>
&Laurent El Ghaoui
University of California, Berkeley
<EMAIL_ADDRESS>

###### Abstract

We address the problem of unsupervised extractive document summarization,
especially for long documents. We model the unsupervised problem as a sparse
auto-regression one and approximate the resulting combinatorial problem via a
convex, norm-constrained problem. We solve it using a dedicated Frank-Wolfe
algorithm. To generate a summary with $k$ sentences, the algorithm only needs
to execute $\approx k$ iterations, making it very efficient. We explain how to
avoid explicit calculation of the full gradient and how to include sentence
embedding information. We evaluate our approach against two other unsupervised
methods using both lexical (standard) ROUGE scores, as well as semantic
(embedding-based) ones. Our method achieves better results with both datasets
and works especially well when combined with embeddings for highly paraphrased
summaries.

##  1 Introduction

With the overwhelming increase of digital information, automatic text
summarization has become important for many applications such as financial
reviews, medical articles, etc. Manually summarizing this amount of
information takes a considerable amount of time and effort. This has motivated
the study of efficient and reliable automatic text summarization methods. The
task of automatic summarization is the process of generating a condensed
version of a text that best describes the original one Hahn and Mani (2000);
Luhn (1958). The two mainstream approaches in the field of automatic
summarization are extractive and abstractive.

Extractive approaches generate summaries by selecting a subset of informative
words, phrases, or sentences directly from the source text. In contrast,
abstractive approaches use linguistic methods to decompose and build a
semantic representation of the text and use natural language generation
techniques to generate a summary Chopra et al. (2016); Nallapati et al.
(2016); Zeng et al. (2016); Rush et al. (2015). In recent years, neural
network architectures have made abstractive summarization popular. However,
abstractive approaches are generally harder to develop as they require high
performing natural language generation techniques, which is also an active
research field. Besides these two categories, mixed strategies that combine
both extractive and abstractive approaches have also been proposed in recent
literature Peng et al. (2019); Cao et al. (2018); See et al. (2017). Previous
work in extractive approaches to summarization include statistical Saggion and
Poibeau (2012); Das and Martins (2007); Goldstein et al. (1999); Kupiec et al.
(1995); Paice (1990), graph-based and optimization-based ones. The graph-based
approaches treat text as a network instead of as a simple bag of words and use
graph-based ranking methods to generate a summary Erkan and Radev (2011);
Ouyang et al. (2009); Mihalcea and Tarau (2004). Optimization-based methods
use techniques such as sparse optimization Yao et al. (2015); Elhamifar and
Vidal (2013), integer linear programming (ILP) Qian and Liu (2013); Berg-
Kirkpatrick et al. (2011); Woodsend and Lapata (2011); Gillick and Favre
(2009) and constraint optimization Durrett et al. (2016); McDonald (2007) to
reconstruct the summary.

In this work, we focus on extractive summarization for long documents.
Performing automatic text summarization for long documents is especially
challenging as obtaining high quality human summaries for long documents is
often quite costly and time consuming. Recent works on extractive
summarization have been focusing on neural network architectures Nallapati et
al. (2017); Cheng and Lapata (2016). Although these methods are successful in
generating summaries for short documents, they often have difficulties with
long input sequences Shao et al. (2017).

Most recent works have started to investigate neural extractive summarization
methods for long documents Xiao and Carenini (2019); Wang et al. (2017).
However, these methods are supervised and require high quality training data
in order to train the neural network models. This creates challenges for
domains that do not have massive training datasets. Kedzie et al. (2018)
compared recent neural extractive summarization models across different
domains including news, personal stories, and medical articles. They found
that many sophisticated neural extractive summarizers do not have better
performance than those consisting of simpler models, and that word embedding
averaging performs equally or better than RNNs or CNNs for sentence embedding.
This suggests that a simpler model combined with pre-trained word embeddings
show promise for summarizing long documents in domains that have few or no
training data.

In this work, we propose an unsupervised method for extracting long documents
based on a sparse optimization framework and solve it using a dedicated Frank-
Wolfe algorithm, which can be combined with pre-trained word embeddings to
construct a distributive input representation. Our work is based on the
previous work of Cheng et al. (2018) but designed specifically for the
summarization task. The proposed framework is an unsupervised model that is
efficient and does not require a training corpora, as typical supervised
solutions would require. We test our method on two datasets that contain long
documents, 2019 Financial Outlooks and Classical Literature, and compare it
against two baselines: sparse subspace clustering (SSC) and TextRank. The
experimental results show that our method gives a higher ROUGE score than our
baseline for both datasets. In particular, when combined with sentence
embedding, our method gives a higher semantic ROUGE score when evaluated on
paraphrased summaries. Moreover, we show that our method is computationally
more efficient compared to others.

##  2 Methodologies

#### Notation

We denote $X_{(i)}$ and $X^{(i)}$ as the $i$-th row and $i$-th column of a
matrix $X$ respectively. The matrix $X_{t}$ denotes the value of $X$ at
iteration $t$ while $[X_{t}]_{(i)}$ and $[X_{t}]^{(i)}$ denotes the $i$-th row
and $i$-th column of $X_{t}$. The sum $\sum_{i=1}^{n}\|X_{(i)}\|_{2}$ is the
$L_{2}$ norm group LASSO constraint. The norm $\|\cdot\|_{F}$ is the Forbenium
norm.

###  2.1 Sparse auto-regressive problem

Extractive summarization aims at finding a minimal set of representative
sentences of the original document that effectively summarizes the entire
document. Let $A\in\mathbb{R}^{d\times n}$ be the data matrix that represents
the document where each column of $A$ represents a sentence in the source
document. Here, $d$ is the number of features for each sentence, and $n$ is
the number of sentences in the source document. The source document $A$ is
written as

$\displaystyle A\triangleq\begin{bmatrix}a_{1}&a_{2}\cdots&a_{n}\end{bmatrix}$

where the column vector $a_{i}$ is a sentence in the source document. Finding
the set of representative sentences assumes that the source document $A$ can
be approximated by a sparse combination of sentences in the document:

$\displaystyle A\approx a_{1}x_{1}^{T}+a_{2}x_{2}^{T}+\cdots+a_{n}x_{n}^{T}$

The column vector $x_{i}$ is a decision variable to be learned. Our goal is to
select $k$ sentences whose corresponding decision variable $x_{i}$ is non-
zero. If we write it in a matrix form with $x_{i}^{T}$ being the row of the
matrix variable $X$, we can formulate the above problem as an auto-regressive
problem of the form:

$\displaystyle\min_{X}$ $\displaystyle\|AX-A\|_{F}^{2}$ (1) s.t.
$\displaystyle\|v\|_{0}\leq k$ $\displaystyle
v_{i}=\|X_{(i)}\|_{2},\;\;\forall i\in[n]$ $\displaystyle X\geq 0$

where $X$ is row-sparse. Here, $X_{(i)}$ represents the $i$-th row of the
matrix variable $X$, $v_{i}$ is its norm, and the constraint is written in
terms of the $L_{0}$-norm (cardinality, or number of non-zero entries) of $v$,
effectively forcing at least $n-k$ entire rows of $X$ to be zero, thereby
singling out a short list of at most $k$ sentences that well represent the
whole data set.

The above problem is non-convex and hard to solve but can be well approximated
by the so-called $L_{1}$-norm heuristic, leading to a convex approximation:

$\displaystyle\min_{X}$ $\displaystyle\|AX-A\|_{F}^{2}$ (2) s.t.
$\displaystyle\|v\|_{1}\leq\beta$ $\displaystyle
v_{i}=\|X_{(i)}\|_{2},\;\;\forall i\in[n]$ $\displaystyle X\geq 0$

where $\beta$ is a hyper-parameter and indirectly controls the row-sparsity
(number of non-zero rows). Note that the model simply uses the $L_{1}$-norm of
vector $v$ in (1) to approximate the cardinality constraint on $v$. If $X^{*}$
is the solution of problem (2), then columns in the data matrix $A^{(j)}$ that
correspond to the non-zero rows of $X^{*}$, $X_{(j)}\neq 0$, are the selected
sentences.

###  2.2 Frank-Wolfe unsupervised extractive summarization

The Frank-Wolfe (FW) or conditional gradient algorithm is an iterative first-
order optimization algorithm for constrained convex optimization Frank and
Wolfe (1956). Although the algorithm was introduced over half a century ago,
it has experienced a revival in recent years due to its projection-free
iterations and broad applications in machine learning Jaggi (2013).

The FW algorithm solves a general constrained optimization problem of the form
$\min_{x\in\mathcal{D}}f(x)$, where the convex function $f$ is differentiable
and $L$-Lipschitz and the domain $\mathcal{D}$ is a convex compact set. At
each iteration, the FW algorithm requires solving a linear approximation to
the objective function over the domain, often referred to as a linear
minimization oracle (LMO), and then updates the solution accordingly. At each
iteration, we first calculate the gradient, solve the LMO problem to find a
descent direction, calculate the step size by line search or by
$\frac{2}{t+2}$, and update the estimate. Algorithm 1 summarizes the FW
process.

Algorithm 1 Frank-Wolfe algorithm

1:Let $t\leftarrow 0$ and $\boldsymbol{x}_{0}\in\mathcal{D}$

2:for $t=0,1,\dots,$ do

3:
$\boldsymbol{s}_{t}=\operatorname*{arg\,min}\limits_{\boldsymbol{s}\in\mathcal{D}}\langle\boldsymbol{s},\nabla
f(x_{t})\rangle$

4: Set step size $\boldsymbol{r}_{t}\leftarrow\frac{2}{t+2}$ or

5:
$\boldsymbol{r}_{t}\leftarrow\operatorname*{arg\,min}\limits_{\boldsymbol{r}\in[0,1]}f(\boldsymbol{x}_{t}+\boldsymbol{r}(\boldsymbol{s}_{t}-\boldsymbol{x}_{t}))$

6: Update
$\boldsymbol{x}_{t+1}=\boldsymbol{x}_{t}+\boldsymbol{r}_{t}(\boldsymbol{s}_{t}-\boldsymbol{x}_{t})$

7:return $\boldsymbol{x}_{t}$

Unlike other descent methods for constrained optimization that require a
projection step at each iteration, the FW algorithm is a projection-free
algorithm and only needs to solve the LMO. Applying the FW algorithm to our
sparse constrained optimization problem (2) results in an unsupervised method
for extractive summarization. Algorithm 1 is written in terms of vector
variable $x$; however, it is straightforward to extend it to the matrix
variable $X$. The algorithm starts with $X_{0}\leftarrow 0$, meaning no
sentence is selected at first. Then the algorithm greedily selects one
sentence at each iteration. The algorithm terminates once $k$ rows of $X_{t}$
are non-zero (dense) or when the algorithm converges to a row-sparse solution
with $k^{*}<k$ non-zero rows. The complete algorithm of Frank-Wolfe
unsupervised extractive summarization is outlined in algorithm 2. Next, we
explain the details of the algorithm and provide an efficient gradient
calculation scheme.

#### Linear minimization oracle

The algorithm requires solving the LMO at each iteration. The solution of the
LMO $S_{t}$ specifies the direction of descent at each step.

$\displaystyle S_{t}=\operatorname*{arg\,min}_{S\in\mathcal{D}}\langle
S,\nabla f(X_{t})\rangle$

Because of the group LASSO constraint in (2), the solution matrix $S_{t}$ is a
rank-1 matrix. The non-zero row of $S_{t}$ is chosen based on the maximum
$L_{2}$ norm of the gradient’s rows:

$\displaystyle j=\operatorname*{arg\,max}_{i}\|[\nabla f(X_{t})]_{(i)}\|_{2}$

We denote the non-zero row of $S_{t}$ at row $j$ as $[S_{t}]_{(j)}$. The
magnitude of $[S_{t}]_{(j)}$ is $\beta$ and the direction is chosen to
minimize the inner product:

$\displaystyle\big{[}S_{t}\big{]}_{(j)}=-\beta\frac{\big{[}\nabla
f(X_{t})\big{]}_{(j)}}{\big{\|}\big{[}\nabla
f(X_{t})\big{]}_{(j)}\big{\|}_{2}}$

The algorithm produces sparse and low-rank iterates since at most one extra
row of $X$ becomes non-zero in each step by the addition of
$\boldsymbol{r}_{t}S_{t}$.

#### Efficient gradient calculation

The algorithm requires calculating the gradient at each iteration. The
gradient of the objective function in (2) is:

$\displaystyle\nabla f(X)=2(A^{T}AX-A^{T}A)=2(KX-K)$

The matrix $K=A^{T}A$ can be calculated once and used throughout the
algorithm. Explicitly calculating the gradient is expensive due to the matrix-
matrix product $(\text{naively }\mathcal{O}(n^{3}))$. However, the structure
of the problem allows us to efficiently calculate $KX$ at each iteration. From
line 6 in Algorithm 1, we know that $X_{t}$ is a weighted average of $X_{t-1}$
and a rank-1 matrix $S_{t-1}$:

$\displaystyle X_{t}$
$\displaystyle=(1-\boldsymbol{r}_{t-1})X_{t-1}+\boldsymbol{r}_{t-1}S_{t-1}$

This suggests that $(KX)_{t}$ is a weighted average of $(KX)_{t-1}$ and
$KS_{t-1}=K^{(j)}[S_{t-1}]_{(j)}$. $K^{(j)}$ is the $j$-th column
corresponding to the $j$-th non-zero row of $S_{t-1}$. Since $(KX)_{t-1}$ is
known at iteration $t$, we are only required to calculate
$K^{(j)}[S_{t-1}]_{(j)}$, which is extremely fast $(\text{in
}\mathcal{O}(n))$.

#### Stopping criteria

The algorithm terminates once $k$ rows of $X_{t}$ are non-zero (dense) or when
$X_{t}$ converges to a row-sparse solution such that $-\big{\langle}\nabla
f(X_{t}),S_{t}-X_{t}\big{\rangle}<\epsilon$. Once the algorithm terminates, it
returns the $k$ sentences that correspond to the non-zero rows of $X_{t}$ by
GetSummary$(X_{t},k)$.

Algorithm 2 Frank-Wolfe unsupervised extractive summarization

1:input $\beta,k,\epsilon$

2:initialize $X_{0},(KX)_{0},\boldsymbol{r}_{0},t\leftarrow 0,0,0,1$

3:compute $K=A^{T}A$ or $\Phi(A)$

4:for $t=1,2,\dots$ do

5:
$(KX)_{t}=(1-\boldsymbol{r}_{t-1})(KX)_{t-1}+\boldsymbol{r}_{t-1}K^{(j)}[S_{t-1}]_{(j)}$

6: $\nabla f(X_{t})=2\big{(}(KX)_{t}-K\big{)}$

7: $j=\operatorname*{arg\,max}_{j}\big{\|}\big{[}\nabla
f(X_{t})\big{]}_{(j)}\big{\|}_{2}$

8: $S_{t}=0$

9: $\big{[}S_{t}\big{]}_{(j)}=-\beta\frac{\big{[}\nabla
f(X_{t})\big{]}_{(j)}}{\big{\|}\big{[}\nabla
f(X_{t})\big{]}_{(j)}\big{\|}_{2}}$

10: $\boldsymbol{r}_{t}=\frac{2}{t+2}$ or
$\operatorname*{arg\,min}\limits_{\boldsymbol{r}\in[0,1]}f(X_{t}+\boldsymbol{r}(S_{t}-X_{t}))$

11: $X_{t+1}=X_{t}+\boldsymbol{r}_{t}(S_{t}-X_{t})$

12: if NumSent$(X_{t+1})=k$ or $-\big{\langle}\nabla
f(X_{t}),S_{t}-X_{t}\big{\rangle}<\epsilon$ then

13: break $\triangleright$ $k$ rows are non-zero or $X_{t}$ converges

14: $t=t+1$

15:return GetSummary($X_{t+1},k$)

#### Sentence similarity measure

We note that the gradient of (2) depends only on the kernel (or, “Gram”)
matrix $K=A^{T}A$ and not on $A$. This matrix is akin to a similarity matrix,
with $K_{ij}$ measuring the similarity between sentences $i$ and $j$. If the
matrix $A$ is normalized, $K_{ij}$’s are cosine similarities. As a result, we
may replace the matrix $K$ with any matrix $\Phi(A)$ that offers a good
similarity measure between sentences. This allow us to incorporate various
sentence scoring functions $\Phi(\cdot)$. In this paper, we experimented with
two such similarity measures: 1) TF-IDF-like, and 2) sentence embedding.

For TF-IDF-like similarity measure, we use Okapi BM25 Robertson and Zaragoza
(2009) to construct the kernel matrix $K$. BM25 and its variants represent the
state-of-the-art TF-IDF-like sentence scoring functions. Similarly, any
sentence embedding technique can be used to embed the document matrix $A$ in a
much lower dimensional space; that is, we can set
$K_{ij}=\phi(a_{i})^{T}\phi(a_{j})$, with $\phi(a)$ the (low-dimensional)
vector representing the sentence $a$. In this work, we use a simple yet
effective sentence embedding method called smooth inverse frequency (SIF)
Arora et al. (2017) to measure the sentence similarities. In Arora et al.
(2017), the authors show that SIF, a simple weighted average of word vectors
modified by SVD, outperforms complex methods such as RNNs and LSTMs. More
sophisticated sentence embedding techniques such as neural architectures can
also be used here; however, once should also consider the cost of computing
the kernel matrix $K$ with such a technique.

In the following, the acronyms FWSum-BM25 and FWSum-SIF are used to refer to
the corresponding Frank-Wolfe unsupervised extractive summarization method
used in conjunction with the BM25 and SIF similarity kernels.

##  3 Experiments

###  3.1 Datasets

Dernoncourt et al. (2018) surveyed the current large-scale dataset for
summarization. Most of them are relatively short; usually less than 2 pages.
To experiment on long documents, we used the recently open-sourced 2019
Financial Outlooks and Classical Literature
dataset111https://github.com/SumUpAnalytics/goldsum, which contain much longer
documents than those surveyed in Dernoncourt et al. (2018).

#### 2019 Financial Outlooks

This corpus contains 10 publicly available reports on finance from a number of
large financial institutions. Each report ranges from 10 to 144 pages, with a
median length of 33 pages. There are no Gold summaries per se since the data
is not annotated by a human. Hence, we chose to define the gold summaries as
the collection of sentences or parts of sentences that appear in bold in the
content, or any sentences that are highlighted as an insert within the
content. This is a reasonable heuristic as these parts are generally prepared
by the authors to highlight the takeaway of the content.

#### Classical Literature

The corpus contains summaries of books that have been summarized by human
writers. The corpus contains 11 English-language classical books ranging from
53 to 1139 pages, with a median length of 198 pages. The Gold summaries for
each chapter of the book are retrieved from WikiSummary
222http://wikisum.com/w/Main_Page.

###  3.2 Baselines

We compared our method with two other unsupervised extractive approaches; one
uses a sparse optimization-based method and the other uses a graph-based
method.

#### Sparse subspace clustering (SSC)

Sparse subspace clustering (SSC) solves a sparse optimization program on the
auto-regressive problem similar to (1), called the _self-expressiveness
property_ of the data Elhamifar and Vidal (2013). This property assumes that
each data point can be efficiently reconstructed by a combination of other
points in the data and that there exists a sparse representation of the data
point. The authors consider a convex relaxation as we did in (2) since solving
the original sparse optimization is in general NP-hard. SSC uses the
Alternating Direction Method of Multipliers (ADMM) for solving the sparse
optimization problem. In our work, we employ the Frank-Wolfe algorithm on the
problem, which is more efficient compared to SSC. Subsequent work tried to
speed up SSC You et al. (2016) but with an expense of removing the group LASSO
constraint that is crucial for our summarization problem. In our work, we are
able to preserve the group LASSO constraint and obtain a faster run-time. In
our experiment, we used the implementation of Elhamifar and Vidal (2013),
which can be found on their
website333http://www.ccs.neu.edu/home/eelhami/codes.htm.

#### TextRank

TextRank Mihalcea and Tarau (2004) is a commonly used graph-based unsupervised
extractive summarization method. It is also very efficient when extracting
summaries from a long document. TextRank employs the similar idea of PageRank
where vertices in the graph are sentences in the document and edges between
two sentences are measured as a function of their content overlap.

###  3.3 Lexical and semantic ROUGE scores

We evaluate the systems using the ROUGE-1, ROUGE-2, and ROUGE-L Lin (2004) so
as to account for different summary lengths. The raw ROUGE score only measures
the lexical overlaps between the generated summaries and the reference
summaries. We refer to the raw ROUGE score defined in Lin (2004) as the
lexical ROUGE and used the implementation of the Python rouge
library444https://pypi.org/project/rouge/. When summarizing a long document,
humans tend to paraphrase the source document in order to condense and
synthesize the information. However, the lexical ROUGE scores are unable to
measure the quality of paraphrasing. To address this shortcoming of lexical
ROUGE when the summaries are paraphrased, word embedding ROUGE scores Ng and
Abrecht (2015) are also used to evaluate the quality of the generated
summaries. The word embedding ROUGE scores are more capable of measuring
semantic similarity of the words instead of only lexical overlaps. Ng and
Abrecht (2015) showed that the embedding ROUGE achieved better correlations
with human assessments compared to lexical ROUGE when measured with the
Spearman and Kendall rank coefficients on the TAC AESOP summarization dataset.
We refer to the word embedding ROUGE scores as the semantic ROUGE in our
evaluation.

##  4 Results and Analysis

Lexical ROUGE |  SSC |  TextRank |  FWSum-BM25 |  FWSum-SIF
---|---|---|---|---
Financial Outlook |  ROUGE-L F1 |  11.88 |  14.43 |  13.92 |  14.99
ROUGE-2 F1 |  2.15 |  3.76 |  5.17 |  3.05
ROUGE-1 F1 |  14.6 |  19.94 |  19.9 |  18
Classical Literature |  ROUGE-L F1 |  7.48 |  16.27 |  18.7 |  13.18
ROUGE-2 F1 |  0.38 |  2.54 |  3.23 |  1.25
ROUGE-1 F1 |  9.97 |  19.61 |  20.2 |  16.58
Semantic ROUGE |  SSC |  TextRank |  FWSum-BM25 |  FWSum-SIF
Financial Outlook |  ROUGE-L F1 |  30.01 |  26.2 |  22.97 |  34.56
ROUGE-2 F1 |  55.56 |  61.43 |  61.82 |  58.92
ROUGE-1 F1 |  43.4 |  48.28 |  49.97 |  47.73
Classical Literature |  ROUGE-L F1 |  31.92 |  39.15 |  39.1 |  46.6
ROUGE-2 F1 |  47.73 |  53.35 |  54.1 |  60.53
ROUGE-1 F1 |  38.72 |  44.15 |  42.43 |  48.18
Table 1: Lexical and semantic ROUGE performance for Financial Outlook and
Classical Literature data. Results that are statistically better are bold
faced and results that are statistically indistinguishable are colored as
gray. An additional experimental results can be found in appendix A.

In our experiment, we set the number of selected sentences $k$ to be the same
as the length of reference summary for all methods. The performance of all
methods on Financial Outlook and Classical Literature are shown in Table 1. As
shown in the table, FWSum-BM25 has a similar performance with TextRank
although slightly better. This may be explained by the sentence scoring
functions used by TextRank and FWSum-BM25. TextRank uses lexical overlaps
between two sentences while FWSum-BM25 uses the TF-IDF-like scoring function,
which are similar in nature.

FWSum-BW25 performs especially well when evaluated with lexical ROUGE,
highlighting its capabilities of capturing lexical information (measured by
unigram and bigram). When evaluated on the Financial Outlook data, FWSum-BW25
and TextRank generally outperform FWSum-SIF, with FWSum-BM25 being the best
performing method. Presumably, this is due to the fact that the Gold summaries
of the Financial Outlook data are taken directly from the source document
without much paraphrasing, favoring sentence scoring functions that directly
measure the content overlaps.

However, when evaluated by the semantic ROUGE on the Classical Literature
data, FWSum-SIF start to show promises. The Gold summaries of the Classical
Literature data are written by human writers and are highly paraphrased and
condensed. As a result, semantic ROUGE is a better measurement for this
dataset. As shown in the table, FWSum-SIF starts to outperform other methods
by a significant amount. The improvement over the other methods suggests that
using embedding in the sentence scoring function allows for comparisons based
on the semantics of words sequences.

This results show that different sentence scoring functions may be used based
on the nature of the summary. For summaries that are mostly taken from the
source document without much paraphrasing, a lexical overlap or TF-IDF-like
kernel matrix may be used. For summaries that are highly paraphrased, an
embedding-like kernel matrix may be more suitable. Our method is able to work
with both.

#### Computational complexity

Our method requires an up-front cost of calculating the kernel matrix $K$.
Each subsequent iteration requires mostly the LMO and gradient calculation as
detailed in section 2. By exploiting the structure of the problem, we are able
to avoid explicitly calculating the full gradient. Furthermore, due to the
greedy nature of the algorithm, it terminates when $k$ sentences are selected
or the solution converges with $k^{*}<k$ sentences. This means that the
algorithm only needs to execute $\approx k$ iterations; each iteration has a
cost linear in problem size. Figure 1 compares the algorithm run-time of our
method (FWSum-BM25), TextRank and SSC. As shown in the figure, our method is
the most efficient among the three, showing its potential for summarizing long
documents.

$0$$5$$10$$15$$20$$25$$0$$20$$40$$60$$80$$100$Length of summary as a
percentage of source document length (%)MinutesFWSum-BM25TextRankSSC Figure 1:
Algorithm run-time for FWSum-BM25, TextRank and SSC on the Financial Outlook
data. The $x$-axis shows the length of the generated summary (_i.e._ $k$) as a
percentage of the source document length (number of sentences in the source
document).

##  5 Conclusion

Unsupervised document summarization has been a challenging task, especially on
long documents. In this work, we propose an efficient unsupervised extractive
summarization model that is suitable for long documents by employing a
dedicated Frank-Wolfe algorithm. Our method allows one to incorporate sentence
embedding or any sentence scoring functions that is best suited for the
dataset or the application. We evaluate our method and compare it with two
other unsupervised extractive summarization methods on two datasets that are
much longer than other summarization corpora used in the past. We evaluate the
methods on both lexical and semantic ROUGE in order to overcome the
shortcoming of lexical ROUGE and to provide a better assessment of the quality
of the summaries. We observed that our methods (both FWSum-BM25 and FWSum-SIF)
achieve the best results for both datasets and that FWSum-SIF works especially
well with summaries that are paraphrased. Our results also motivate the
exploration of different kernel functions or embedding methods, which is left
as a future work.

## Acknowledgments

The authors would like to thank Richard Liou, Tanya Roosta, and Gary Cheng for
their constructive discussions on drafts of this paper and SumUp Analytics for
providing the dataset.

## References

  * Arora et al. (2017) Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017.  A simple but tough-to-beat baseline for sentence embeddings.
  * Berg-Kirkpatrick et al. (2011) Taylor Berg-Kirkpatrick, Dan Gillick, and Dan Klein. 2011.  Jointly learning to extract and compress.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1_ , HLT ’11, pages 481–490, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Cao et al. (2018) Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018.  Retrieve, rerank and rewrite: Soft template based neural summarization.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 152–161, Melbourne, Australia. Association for Computational Linguistics.
  * Cheng et al. (2018) Gary Cheng, Armin Askari, Laurent El Ghaoui, and Kannan Ramchandran. 2018.  Frank-wolfe algorithm for exemplar selection.  _CoRR_ , abs/1811.02702.
  * Cheng and Lapata (2016) Jianpeng Cheng and Mirella Lapata. 2016.  Neural Summarization by Extracting Sentences and Words.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 484–494, Berlin, Germany. Association for Computational Linguistics.
  * Chopra et al. (2016) Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016.  Abstractive sentence summarization with attentive recurrent neural networks.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 93–98, San Diego, California. Association for Computational Linguistics.
  * Das and Martins (2007) Dipanjan Das and André F. T. Martins. 2007.  A survey on automatic text summarization.
  * Dernoncourt et al. (2018) Franck Dernoncourt, Mohammad Ghassemi, and Walter Chang. 2018.  A Repository of Corpora for Summarization.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Durrett et al. (2016) Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein. 2016.  Learning-based single-document summarization with compression and anaphoricity constraints.  _CoRR_ , abs/1603.08887.
  * Elhamifar and Vidal (2013) Ehsan Elhamifar and Rene Vidal. 2013.  Sparse Subspace Clustering: Algorithm, Theory, and Applications.  _arXiv:1203.1005 [cs, math, stat]_.  ArXiv: 1203.1005.
  * Erkan and Radev (2011) Günes Erkan and Dragomir R. Radev. 2011.  Lexrank: Graph-based lexical centrality as salience in text summarization.  _CoRR_ , abs/1109.2128.
  * Frank and Wolfe (1956) Marguerite Frank and Philip Wolfe. 1956.  An algorithm for quadratic programming.  _Naval Research Logistics Quarterly_ , 3(1‐2):95–110.
  * Gillick and Favre (2009) Dan Gillick and Benoit Favre. 2009.  A scalable global model for summarization.  In _Proceedings of the Workshop on Integer Linear Programming for Natural Language Processing_ , pages 10–18, Boulder, Colorado. Association for Computational Linguistics.
  * Goldstein et al. (1999) Jade Goldstein, Mark Kantrowitz, Vibhu Mittal, and Jaime Carbonell. 1999.  Summarizing text documents: Sentence selection and evaluation metrics.  In _Proceedings of the 22Nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’99, pages 121–128, New York, NY, USA. ACM.
  * Hahn and Mani (2000) Udo Hahn and Inderjeet Mani. 2000.  The challenges of automatic summarization.  _Computer_ , 33(11):29–36.
  * Jaggi (2013) Martin Jaggi. 2013.  Revisiting frank-wolfe: Projection-free sparse convex optimization.  In _ICML (1)_ , pages 427–435.
  * Kedzie et al. (2018) Chris Kedzie, Kathleen McKeown, and Hal Daumé III. 2018.  Content Selection in Deep Learning Models of Summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1818–1828, Brussels, Belgium. Association for Computational Linguistics.
  * Kupiec et al. (1995) Julian Kupiec, Jan Pedersen, and Francine Chen. 1995.  A trainable document summarizer.  In _Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’95, pages 68–73, New York, NY, USA. ACM.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out: Proceedings of the ACL-04 Workshop_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Luhn (1958) H. P. Luhn. 1958.  The automatic creation of literature abstracts.  _IBM J. Res. Dev._ , 2(2):159–165.
  * McDonald (2007) Ryan McDonald. 2007.  A study of global inference algorithms in multi-document summarization.  In _Proceedings of the 29th European Conference on IR Research_ , ECIR’07, pages 557–564, Berlin, Heidelberg. Springer-Verlag.
  * Mihalcea and Tarau (2004) Rada Mihalcea and Paul Tarau. 2004.  TextRank: Bringing order into text.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 404–411, Barcelona, Spain. Association for Computational Linguistics.
  * Nallapati et al. (2016) Ramesh Nallapati, Bing Xiang, and Bowen Zhou. 2016.  Sequence-to-sequence rnns for text summarization.  _CoRR_ , abs/1602.06023.
  * Nallapati et al. (2017) Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017.  SummaRuNNer: a recurrent neural network based sequence model for extractive summarization of documents.  In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ , AAAI’17, pages 3075–3081, San Francisco, California, USA. AAAI Press.
  * Ng and Abrecht (2015) Jun-Ping Ng and Viktoria Abrecht. 2015.  Better summarization evaluation with word embeddings for ROUGE.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1925–1930, Lisbon, Portugal. Association for Computational Linguistics.
  * Ouyang et al. (2009) You Ouyang, Wenjie Li, Furu Wei, and Qin Lu. 2009.  Learning similarity functions in graph-based document summarization.  In _Computer Processing of Oriental Languages. Language Technology for the Knowledge-based Economy_ , pages 189–200, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Paice (1990) C. D. Paice. 1990.  Constructing literature abstracts by computer: Techniques and prospects.  _Inf. Process. Manage._ , 26(1):171–186.
  * Peng et al. (2019) Hao Peng, Ankur P. Parikh, Manaal Faruqui, Bhuwan Dhingra, and Dipanjan Das. 2019\.  Text generation with exemplar-based adaptive decoding.  _CoRR_ , abs/1904.04428.
  * Qian and Liu (2013) Xian Qian and Yang Liu. 2013.  Fast joint compression and summarization via graph cuts.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1492–1502, Seattle, Washington, USA. Association for Computational Linguistics.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Found. Trends Inf. Retr._ , 3(4):333–389.
  * Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 379–389, Lisbon, Portugal. Association for Computational Linguistics.
  * Saggion and Poibeau (2012) Horacio Saggion and Thierry Poibeau. 2012.  Automatic Text Summarization: Past, Present and Future.  In R. Yangarber T. Poibeau; H. Saggion. J. Piskorski, editor, _Multi-source, Multilingual Information Extraction and Summarization_ , Theory and Applications of Natural Language Processing, pages 3–13. Springer.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  * Shao et al. (2017) Louis Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2017.  Generating High-Quality and Informative Conversation Responses with Sequence-to-Sequence Models.  _arXiv:1701.03185 [cs]_.  ArXiv: 1701.03185.
  * Wang et al. (2017) Shuai Wang, Xiang Zhao, Bo Li, Bin Ge, and Daquan Tang. 2017.  Integrating Extractive and Abstractive Models for Long Text Summarization.  In _2017 IEEE International Congress on Big Data (BigData Congress)_ , pages 305–312.
  * Woodsend and Lapata (2011) Kristian Woodsend and Mirella Lapata. 2011.  Learning to simplify sentences with quasi-synchronous grammar and integer programming.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 409–420, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  * Xiao and Carenini (2019) Wen Xiao and Giuseppe Carenini. 2019.  Extractive Summarization of Long Documents by Combining Global and Local Context.  _arXiv:1909.08089 [cs]_.  ArXiv: 1909.08089.
  * Yao et al. (2015) Jin-ge Yao, Xiaojun Wan, and Jianguo Xiao. 2015.  Compressive document summarization via sparse optimization.  In _Proceedings of the 24th International Conference on Artificial Intelligence_ , IJCAI’15, pages 1376–1382. AAAI Press.
  * You et al. (2016) Chong You, Daniel P. Robinson, and Rene Vidal. 2016.  Scalable Sparse Subspace Clustering by Orthogonal Matching Pursuit.  _arXiv:1507.01238 [cs, stat]_.  ArXiv: 1507.01238.
  * Zeng et al. (2016) Wenyuan Zeng, Wenjie Luo, Sanja Fidler, and Raquel Urtasun. 2016.  Efficient summarization with read-again and copy mechanism.  _CoRR_ , abs/1611.03382.

##  Appendix A Additional Experimental Results

2019 Financial Outlook
---
Lexical ROUGE |  SMRS |  TextRank |  FWSR-BM25 |  FWSR-SIF
R-L F1 |  [11.77, 11.99] |  [14.28, 14.57] |  [13.79, 14.05] |  [14.87, 15.11]
R-L Precision |  [11.06, 11.28] |  [13.05, 13.31] |  [12.69, 12.92] |  [14.63, 14.85]
R-L Recall |  [18.69, 19.04] |  [29.06, 29.51] |  [33.58, 34.09] |  [22.50, 22.97]
R-2 F1 |  [02.07, 02.23] |  [03.65, 03.87] |  [05.04, 05.29] |  [02.96, 03.13]
R-2 Precision |  [01.63, 01.75] |  [02.62, 02.78] |  [03.44, 03.62] |  [02.37, 02.51]
R-2 Recall |  [03.01, 03.23] |  [06.65, 07.02] |  [10.38, 10.92] |  [04.88, 05.25]
R-1 F1 |  [14.47, 14.73] |  [19.78, 20.10] |  [19.75, 20.04] |  [17.87, 18.13]
R-1 Precision |  [11.91, 12.15] |  [14.77, 15.05] |  [13.95, 14.19] |  [15.56, 15.79]
R-1 Recall |  [20.01, 20.38] |  [32.36, 32.81] |  [36.63, 37.10] |  [24.02, 24.50]
Classical Literature
Lexical ROUGE |  SMRS |  TextRank |  FWSR-BM25 |  FWSR-SIF
R-L F1 |  [07.39, 07.56] |  [16.15, 16.38] |  [18.59, 18.81] |  [13.10, 13.26]
R-L Precision |  [16.96, 17.23] |  [14.03, 14.22] |  [15.82, 16.00] |  [18.04, 18.27]
R-L Recall |  [06.85, 07.00] |  [19.02, 19.31] |  [22.53, 22.81] |  [12.95, 13.15]
R-2 F1 |  [00.37, 00.39] |  [02.49, 02.58] |  [03.16, 03.29] |  [01.22, 01.27]
R-2 Precision |  [00.68, 00.73] |  [01.97, 02.04] |  [02.87, 02.98] |  [01.64, 01.71]
R-2 Recall |  [00.27, 00.29] |  [03.39, 03.51] |  [03.51, 03.66] |  [01.10, 01.15]
R-1 F1 |  [09.87, 10.07] |  [19.49, 19.72] |  [20.10, 20.30] |  [16.48, 16.68]
R-1 Precision |  [17.82, 18.10] |  [18.43, 18.65] |  [16.69, 16.88] |  [20.33, 20.58]
R-1 Recall |  [07.26, 07.42] |  [22.10, 22.44] |  [26.67, 26.97] |  [14.83, 15.08]
Table 2: Lexical ROUGE performance for 2019 Financial Outlook and Classical
Literature dataset. We reported the 95% confidence interval for the results.
Results that are statistically better are bold faced and results that are
statistically similar are colored as gray. 2019 Financial Outlook
---
Semantic ROUGE |  SMRS |  TextRank |  FWSR-BM25 |  FWSR-SIF
R-L F1 |  [29.84, 30.18] |  [26.02, 26.37] |  [22.80, 23.13] |  [34.36, 34.75]
R-L Precision |  [32.70, 34.01] |  [23.78, 24.09] |  [20.97, 21.25] |  [37.83, 38.97]
R-L Recall |  [46.01, 46.33] |  [52.45, 52.76] |  [55.00, 55.33] |  [48.66, 48.95]
R-2 F1 |  [55.45, 55.67] |  [61.33, 61.53] |  [61.72, 61.91] |  [58.84, 59.00]
R-2 Precision |  [52.59, 52.80] |  [57.14, 57.33] |  [56.27, 56.46] |  [56.17, 56.34]
R-2 Recall |  [59.16, 59.45] |  [66.46, 66.74] |  [68.73, 69.00] |  [62.22, 62.49]
R-1 F1 |  [43.27, 43.52] |  [48.16, 48.40] |  [49.85, 50.09] |  [45.54, 45.73]
R-1 Precision |  [41.31, 41.55] |  [44.80, 45.03] |  [45.92, 46.14] |  [43.23, 43.41]
R-1 Recall |  [46.02, 46.33] |  [52.45, 52.76] |  [55.00, 55.33] |  [48.66, 48.95]
Classical Literature
Semantic ROUGE |  SMRS |  TextRank |  FWSR-BM25 |  FWSR-SIF
R-L F1 |  [31.83, 32.04] |  [38.97, 39.33] |  [38.89, 39.30] |  [46.29, 46.91]
R-L Precision |  [21.38, 21.94] |  [37.69, 38.16] |  [45.74, 46.49] |  [48.37, 49.44]
R-L Recall |  [33.25, 33.50] |  [40.35, 40.58] |  [42.19, 42.42] |  [44.38, 44.62]
R-2 F1 |  [47.62, 47.84] |  [52.87, 53.83] |  [54.00, 54.20] |  [60.41, 60.65]
R-2 Precision |  [57.92, 58.19] |  [56.11, 56.32] |  [55.50, 55.71] |  [66.39, 66.62]
R-2 Recall |  [40.78, 41.01] |  [49.98, 50.19] |  [52.84, 53.07] |  [55.42, 55.66]
R-1 F1 |  [38.62, 38.82] |  [44.05, 44.24] |  [42.33, 42.52] |  [48.06, 48.29]
R-1 Precision |  [50.06, 50.38] |  [43.99, 44.17] |  [42.72, 42.92] |  [59.80, 60.04]
R-1 Recall |  [31.83, 32.04] |  [44.38, 44.62] |  [42.19, 42.42] |  [40.35, 40.58]
Table 3: Sematic ROUGE performance for 2019 Financial Outlook and Classical
Literature dataset. We reported the 95% confidence interval for the results.
Results that are statistically better are bold faced.