# Attention Head Masking for Inference Time Content Selection
in Abstractive Summarization

Shuyang Cao  Lu Wang
Computer Science and Engineering
University of Michigan
Ann Arbor, MI
{caoshuy<EMAIL_ADDRESS>

###### Abstract

How can we effectively inform content selection in Transformer-based
abstractive summarization models? In this work, we present a simple-yet-
effective attention head masking technique, which is applied on encoder-
decoder attentions to pinpoint salient content at inference time. Using
attention head masking, we are able to reveal the relation between encoder-
decoder attentions and content selection behaviors of summarization models. We
then demonstrate its effectiveness on three document summarization datasets
based on both in-domain and cross-domain settings. Importantly, our models
outperform prior state-of-the-art models on CNN/Daily Mail and New York Times
datasets. Moreover, our inference-time masking technique is also data-
efficient, requiring less than 20% of the training samples to outperform BART
fine-tuned on the full CNN/DailyMail dataset.

##  1 Introduction

Large pre-trained Transformers have achieved state-of-the-art results on
various summarization datasets with a fine-tuning phase to streamline the
summarization pipeline Lewis et al. (2020); Yan et al. (2020). Yet, it is
still unclear how one can use large models more effectively for abstractive
summarization . For example, prior work shows that informing content selection
via attention weight updating in recurrent neural networks can further boost
summarizer performance Gehrmann et al. (2018). However, with multi-heads
attentions at all layers in Transformers Vaswani et al. (2017), highlighting
salient content becomes non-trivial.

In this work, we propose an inference-time attention head masking mechanism
that works on encoder-decoder attentions to underscore salient content from
the source and improve the quality of abstractive summaries. Based on this
mechanism, we first demonstrate the relation between encoder-decoder
attentions and content selection behaviors, on three summarization datasets of
CNN/DailyMail (CNN/DM), New York Times (NYT), and XSum. Second, we study
whether multiple heads at the same layer collectively guide the summarization.
Partial masking is found to be most effective, indicating a strong
collaborative effect and the importance of head selection.

Based on these observations, we evaluate attention head masking on
summarization benchmarks with salience labels provided by externally trained
content selectors. On all three datasets, our model consistently outperforms
fine-tuned BART Lewis et al. (2020) and several top performing Transformer-
based abstractive summarization models Zhang et al. (2019b); Yan et al.
(2020). Summaries generated by our model are also considered to have better
informativeness by human judges. Moreover, we illustrate that attention head
masking is data-efficient: on CNN/DM, BART fine-tuned on less than $20\%$ of
the training data outperforms a version trained on the full set. Finally, we
show that our method is effective under a cross-domain setting. With a content
selector trained on NYT, BART fine-tuned on CNN/DM gains more than three
points of ROUGE scores when tested on NYT articles.111Our code is available
at: https://shuyangcao.github.io/projects/inference_head_masking.

##  2 Related Work

Large Pre-trained Models for Summarization. Many recent advancements in text
summarization have been achieved by large pre-trained language models Zhang et
al. (2019a); Liu and Lapata (2019); Song et al. (2019); Zhang et al. (2019b).
In particular, BART has demonstrated impressive performance on summarization,
and is used as the base model in this work. Nonetheless, all prior attempts
take pre-trained models as is and conduct fine-tuning on target datasets,
without knowing if it is the most effective usage. In contrast, we bring
insights into the relation between attentions and content selection via
masking operations to further improve summarization performance.

Content Selection for Abstractive Summarization. Content selection is a
crucial step, where salient information is first detected and then summarized
into concise abstracts Chen and Bansal (2018); Xu and Durrett (2019). To
minimize the propagation of selection errors, content selection is modeled as
an extra component and learned within an end-to-end trained model Zhou et al.
(2017); Li et al. (2018); Gehrmann et al. (2018). To the best of our
knowledge, we are the first to apply masks on selected layers and attention
heads in Transformers for content selection in summarization. Moreover, our
masking mechanism is only activated during inference, without any model
modification.

Analyzing Multi-head Attentions has attracted growing interests in the NLP
community Clark et al. (2019); Kovaleva et al. (2019). Among the work that is
relevant to encoder-decoder attentions, Michel et al. (2019) and Voita et al.
(2019) observe that only a small portion of heads is relevant for translation
and encoder-decoder attentions tend to be more important than self-attentions.
Meanwhile, word alignments for machine translation are induced from encoder-
decoder attention weights Li et al. (2019); Kobayashi et al. (2020). However,
none of prior work employs attentions to improve generation quality. As far as
we are aware, this is the first work that studies the content selection
effects of encoder-decoder attentions and uses them to guide better summary
generation.

##  3 Attention Head Masking

We adopt large pre-trained sequence-to-sequence Transformer models (BART,
specifically) for abstractive summarization. Transformer is built with multi-
head attentions. Attentions are computed per step based on a query
$\mathbf{q}$ along with the key and value matrices, $\mathbf{K}$ and
$\mathbf{V}$:

$\mathrm{Attention}(\mathbf{q},\mathbf{K},\mathbf{V})=\mathrm{softmax}(\frac{\mathbf{q}\mathbf{K}^{T}}{\sqrt{d_{k}}}+\mathbf{m})\mathbf{V}$
(1)

where $d_{k}$ is a scaling factor and $\mathbf{m}$ is for padding or masking
future tokens (when the value is $-\infty$).

Masking Operation. We propose attention head masking in encoder-decoder
attentions, which blocks attentions to unimportant tokens, to better
concentrate multi-head attentions on salient input tokens. Importantly, it is
activated during inference. Concretely, we add an $\mathbf{\tilde{m}}$ inside
the softmax operator of Eq. 1, with implementation displayed in Fig. 1. The
size of $\mathbf{\tilde{m}}$ is the same as the input length. If the $i$-th
token is tagged as salient, the corresponding element in $\mathbf{\tilde{m}}$
is set to $0$ (attendable to the attention heads), and $-\infty$ otherwise
(hidden from these heads). The saliency labels can be predicted by an
externally trained content selector.

Figure 1: Illustration of attention head masking ($\mathbf{\tilde{m}}$).

##  4 Encoder-decoder Attentions and Content Selection

In this section, we first probe into the content selection behavior of each
single head (§ 4.1), and then study the synergism among heads at the same
layer (§ 4.2). In § 4.3, we analyze the attentions’ focus.

Our analysis is conducted on CNN/DM Hermann et al. (2015), NYT Consortium and
Company (2008), and XSum Narayan et al. (2018). We follow Lewis et al. (2020)
for data preprocessing and train/validation/test splits on CNN/DM and XSum,
and adopt the setups in Paulus et al. (2018) for NYT, except that we keep
entities and numbers. The number of samples in training, validation, and test
set are: $287{,}188$, $13{,}367$ and $11{,}490$ for CNN/DM; $588{,}909$,
$32{,}716$ and $32{,}703$ for NYT; $204{,}045$, $11{,}332$ and $11{,}334$ for
XSum.

For experiments in this section, we create an analysis set of $1{,}000$ random
samples from the validation split of each dataset to reduce computational
cost.

###  4.1 Content Selection Effects

First, we study the feasibility of using encoder-decoder attentions to inform
content selection and subsequently boost summary informativeness. Concretely,
we apply attention head masking based on oracle content selection labels
(henceforth oracle masking). Oracle labels are constructed by aligning a
reference summary to the source article, where we iteratively find the longest
common subsequences between the two.

Taking a fine-tuned BART model, we apply oracle masking on each head at each
layer when decoding on the analysis set. The ROUGE score obtained in this
setting is denoted as $r_{ora}$. We then apply uniform encoder-decoder
attention weights over the source to build a baseline that mimics no content
selection, inspired by Wiegreffe and Pinter (2019). This yields a ROUGE score
of $r_{uni}$. The content select effect per head can thus be calculated as the
ROUGE improvement, i.e., $r_{ora}-r_{uni}$.

Figure 2:  ROUGE-1 F1 improvement with oracle masks for each head at each
layer on the analysis set of CNN/DM. Overall, top layers see greater
improvement than bottom layers. Layer 1 is the bottom layer connected with the
word embeddings.

Overall, it is more effective to constrain attentions to salient content at
the top layers, according to the results on CNN/DM in Fig. 2. Specifically,
with oracle masking, the top layer yields the most ROUGE-1 improvement. We
observe similar trends on NYT and XSum (figures are in Appendix C). This
indicates the feasibility of leveraging attention head masking to improve
summary informativeness.

###  4.2 Synergism Analysis

Figure 3: [Left] ROUGE-1 F1 improvement by incrementally applying oracle
masking to the next head with most ROUGE improvement per layer on CNN/DM.
Dotted lines indicate that the newly masked heads do not have individual ROUGE
improvements. [Right] ROUGE-1 recall improvement by masking all heads vs. sum
of improvement by masking each head separately on CNN/DM. Best viewed in
color.

Next, we study whether masking multiple heads can further boost content
selection and whether they form synergy. On the left of Fig. 3, we show
content selection effect by gradually applying oracle masking on more heads at
each layer, with heads sorted based on individual ROUGE improvements. Notably,
the most ROUGE-1 improvement is achieved by masking 15 (out of 16) heads at
the top layer, suggesting a strong collaborative effect on content selection
by masking multiple heads.

We further compare the ROUGE score gains between oracle masking on all heads
and the sum of individual effects, illustrated on the right of Fig. 3. The
discrepancies between the two values suggest that the heads may not be
independent at pinpointing salient content. In Appendix D, we reach similar
results on NYT and XSum.

Based on the above observations, we argue that it is necessary to select
layers and heads accordingly to achieve the best content selection effect,
with more summarization results reported in § 5.

###  4.3 Attention Focus

We further provide a fine-grained study on what types of words the heads
attend to. Concretely, we consider each word generated during decoding,
denoted as $y$. Given an attention head, we follow the highest attention
weight to identify the input word $x$ (“attendee”). We study several
categories of attendee $x$: (1) word in the reference (salient); (2) content
word; (3) the first and last words in the document. For salient and content,
we further consider two subcategories: $x=y$ (copy) and $x\neq y$ (non-copy).
We then tally the occurrences of each type of attendees per head at each layer
on the analysis set.

(a) copy salient

(b) non-copy salient

(c) copy content

(d) first

Figure 4:  copy and non-copy salient attendee word percentages on the analysis
set of CNN/DM. Top layers focus on words to be “copied", while bottom layers
attend to the broader salient context.

We show the percentages of copy and non-copy salient attendees, copy content
attendees, and first attendees on CNN/DM in Fig. 4. As can be seen, top layers
tend to focus on input tokens that will be generated as is, while bottom
layers attend to salient words that are not used for current generation.
Additionally, bottom layers frequently attend to the first token of the
document, where bottom layers are more likely performing context gathering. On
NYT and XSum (figures are in Appendix E), similar trends are observed except
that the first attendees are more focused by the top layers on NYT articles,
where many of them start with all capitalized words.

##  5 Summarization Results with Attention Head Masking

In this section, we show how to leverage attention head masking and a content
selector to improve summary informativeness on three datasets. We first train
a binary sequence tagger for each dataset to label salient tokens in the
source, used for system masking for attention heads. Our sequence tagger is a
RoBERTa Liu et al. (2019) encoder followed by a double layer multilayer
perceptron (MLP) with a hyperbolic tangent activation function in between. To
obtain the probability for each token, the MLP output is further fed into a
sigmoid activation function. Details for training and decoding are in Appendix
A.

The decision boundary for the sequence tagger is selected according to the F1
score calculated between the predicted tags and the ground-truth labels on the
validation set. We search for the best decision boundary from $0.1$ to $0.4$,
with a step size of $0.01$. The final decision boundaries used for taggers
trained on CNN/DM, NYT, XSum are $0.20$, $0.24$, and $0.18$, achieving ROUGE-1
F1 of $43.70$, $44.10$, and $31.56$, respectively.

To select which heads at which layers to mask, we employ a greedy selection
strategy. On the analysis set, we gradually apply system masking on four heads
with most ROUGE improvement according to the study in § 4.1, and we select the
heads that achieve the highest sum of ROUGE-1 F1 and ROUGE-2 F1. We apply four
heads each time to reduce computational cost of hyperparameter searching.
Heads selected for each dataset are in Appendix B.

Model | R-1 | R-2 | R-L
---|---|---|---
BERTSum Liu and Lapata (2019) | 42.13 | 19.60 | 39.18
UniLM Dong et al. (2019) | 43.33 | 20.21 | 40.51
PEGASUS Zhang et al. (2019b) | 44.17 | 21.47 | 41.11
ProphetNet (Yan et al., 2020) | 44.20 | 21.17 | 41.39
BART (ours) | 44.19 | 21.20 | 40.98
\+ attention head masking (ours) | 45.54∗ | 22.24∗ | 42.44∗
(a) CNN/DailyMail

Model | R-1 | R-2 | R-L
---|---|---|---
BottomUp Gehrmann et al. (2018) | 47.38 | 31.23 | 41.81
DCA Celikyilmaz et al. (2018) | 48.08 | 31.19 | 42.33
BERTSum Liu and Lapata (2019) | 49.02 | 31.02 | 45.55
ASGARD Huang et al. (2020) | 51.29 | 34.97 | 48.26
BART (ours) | 53.00 | 36.31 | 48.90
\+ attention head masking (ours) | 53.52∗ | 36.69 | 49.24
(b) New York Times

Model | R-1 | R-2 | R-L
---|---|---|---
BERTSum Liu and Lapata (2019) | 38.81 | 16.50 | 31.27
PEGASUS | 47.21 | 24.56 | 39.25
BART (ours) | 45.36 | 22.30 | 37.11
\+ attention head masking (ours) | 45.35 | 22.31 | 37.15
(c) XSum

Table 1:  Automatic evaluation with ROUGE. $\ast$: significantly better than
BART with approximate randomization test ($p<0.005$). Our method outperforms
BART and previous models on CNN/DM and NYT.

In-domain Results. Table 1 shows that applying our attention head masking
technique on BART obtains significantly better results on CNN/DM and NYT,
compared to several top performing abstractive summarization models trained
with large Transformers. The improvement is more pronounced for CNN/DM than
the other two datasets. We believe this is due to the difference in
abstractiveness among the three datasets. CNN/DM has more extractive summaries
compared to the other datasets Grusky et al. (2018), suggesting attention head
masking is more effective on extractive datasets. Notably, PEGASUS is pre-
trained with 3.8TB of news articles, the BART model used in our work is only
pre-trained with 160GB of a combination of news, books, stories, and web text.
The large size of the pre-training data might be a big contributor to the
better performance by PEGASUS on XSum.

For human evaluation, we hire three fluent English speakers to rate $50$ pairs
of summaries generated with and without attention head masking based on BART
for informativeness and faithfulness. Informativeness measures how well the
summary captures salient content from the article, while faithfulness
indicates whether the summary correctly reflects the content in the source
article. The annotators are asked to determine if attention head masking
improves any of the two aspects. As shown in Table 2 where all ratings by
three judges are considered, summaries generated with attention head masking
are considered to have better informativeness, but no substantial improvement
on faithfulness is observed.

| w/ masking | w/o masking | Tie
---|---|---|---
Informativeness | 36.0% | 19.3% | 44.7%
Faithfulness | 10.0% | 7.3% | 82.7%
Table 2: Percentages of summaries with and without attention head masking
favored by annotators on informativeness and faithfulness. The Krippendorff’s
$\alpha$ for informativeness and faithfulness are $0.30$ and $0.47$.  Figure
5: Results on CNN/DM with different sizes of training data. Our method
consistently improves the summarizer.

Limited Training Data. Next, we study if our masking technique is still
effective if given limited training samples. We use the limited training
samples to train both the summarizer and the content selector. As can be seen
in Fig. 5, our masking technique consistently increases ROUGE scores with
varying amounts of training data. Notably, our model trained on only $30$K
samples (with attention head masking) outperforms the model trained on the
full dataset, suggesting that directly informing content selection is more
data-efficient than model fine-tuning on more summaries.

Cross-domain Results. Finally, we show results on NYT using BART fine-tuned on
CNN/DM, with system masks predicted by a tagger trained on different sizes of
NYT samples (Table 3). Using a selector trained with only $10$k of target
domain samples, we already significantly improve the performance by BART
trained on CNN/DM only.

Selector Training Data | R-1 | R-2 | R-L
---|---|---|---
No masking | 31.11 | 14.68 | 28.19
$10$K | 34.98 | 17.95 | 31.87
$100$K | 34.71 | 17.70 | 31.61
$589$K (full) | 35.13 | 18.07 | 32.03
Table 3: Results on NYT summaries generated by BART trained on CNN/DM, with
masks predicted by content selectors trained on different sizes of NYT data.

##  6 Conclusion

We propose attention head masking that constrains encoder-decoder attention
heads to attend to salient tokens, to inform content selection in abstractive
summarization. With this technique, we first demonstrate the relation between
encoder-decoder attentions and content selection behaviors. With system masks
predicted by external content selectors, we show that attention head masking
can consistently improve ROUGE scores over competitive summarization models on
three benchmarks. Summaries generated with attention head masking are also
preferred by human judges more frequently. Additional experiments demonstrate
that our method is more data-efficient and effective on both in-domain and
cross-domain settings.

## Acknowledgements

This research is supported in part by National Science Foundation through
Grant IIS-1813341, and by the Office of the Director of National Intelligence
(ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract
# FA8650-17-C-9116. The views and conclusions contained herein are those of
the authors and should not be interpreted as necessarily representing the
official policies, either expressed or implied, of ODNI, IARPA, or the U.S.
Government. The U.S. Government is authorized to reproduce and distribute
reprints for governmental purposes notwithstanding any copyright annotation
therein. We thank three anonymous reviewers for their constructive
suggestions.

## References

  * Celikyilmaz et al. (2018) Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018.  Deep communicating agents for abstractive summarization.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1662–1675, New Orleans, Louisiana. Association for Computational Linguistics.
  * Chen and Bansal (2018) Yen-Chun Chen and Mohit Bansal. 2018.  Fast abstractive summarization with reinforce-selected sentence rewriting.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 675–686, Melbourne, Australia. Association for Computational Linguistics.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019.  What does BERT look at? an analysis of BERT’s attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 276–286, Florence, Italy. Association for Computational Linguistics.
  * Consortium and Company (2008) Linguistic Data Consortium and New York Times Company. 2008.  _The New York Times Annotated Corpus_.  LDC corpora. Linguistic Data Consortium.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’ Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 13063–13075. Curran Associates, Inc.
  * Gehrmann et al. (2018) Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018.  Bottom-up abstractive summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4098–4109, Brussels, Belgium. Association for Computational Linguistics.
  * Grusky et al. (2018) Max Grusky, Mor Naaman, and Yoav Artzi. 2018.  Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 708–719, New Orleans, Louisiana. Association for Computational Linguistics.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28_ , pages 1693–1701. Curran Associates, Inc.
  * Huang et al. (2020) Luyang Huang, Lingfei Wu, and Lu Wang. 2020.  Knowledge graph-augmented abstractive summarization with semantic-driven cloze reward.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kobayashi et al. (2020) Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020.  Attention is not only a weight: Analyzing transformers with vector norms.
  * Kovaleva et al. (2019) Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019.  Revealing the dark secrets of BERT.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4365–4374, Hong Kong, China. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2018) Wei Li, Xinyan Xiao, Yajuan Lyu, and Yuanzhuo Wang. 2018.  Improving neural abstractive document summarization with explicit information selection modeling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1787–1796, Brussels, Belgium. Association for Computational Linguistics.
  * Li et al. (2019) Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019.  On the word alignment from neural machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1293–1303, Florence, Italy. Association for Computational Linguistics.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text summarization with pretrained encoders.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3730–3740, Hong Kong, China. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Michel et al. (2019) Paul Michel, Omer Levy, and Graham Neubig. 2019.  Are sixteen heads really better than one? In _Advances in Neural Information Processing Systems_ , volume 32, pages 14014–14024. Curran Associates, Inc.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of NAACL-HLT 2019: Demonstrations_.
  * Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018.  A deep reinforced model for abstractive summarization.  In _International Conference on Learning Representations_.
  * Song et al. (2019) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019.  MASS: Masked sequence to sequence pre-training for language generation.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 5926–5936, Long Beach, California, USA. PMLR.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30, pages 5998–6008. Curran Associates, Inc.
  * Voita et al. (2019) Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019.  Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5797–5808, Florence, Italy. Association for Computational Linguistics.
  * Wiegreffe and Pinter (2019) Sarah Wiegreffe and Yuval Pinter. 2019.  Attention is not not explanation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 11–20, Hong Kong, China. Association for Computational Linguistics.
  * Xu and Durrett (2019) Jiacheng Xu and Greg Durrett. 2019.  Neural extractive text summarization with syntactic compression.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3292–3303, Hong Kong, China. Association for Computational Linguistics.
  * Yan et al. (2020) Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020.  Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training.
  * Zhang et al. (2019a) Haoyu Zhang, Jingjing Cai, Jianjun Xu, and Ji Wang. 2019a.  Pretraining-based natural language generation for text summarization.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 789–797, Hong Kong, China. Association for Computational Linguistics.
  * Zhang et al. (2019b) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019b.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.
  * Zhou et al. (2017) Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2017.  Selective encoding for abstractive sentence summarization.  _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.

##  Appendix A Training and Decoding Settings

When training the sequence taggers, we minimize the average binary cross-
entropy of each token’s selection probability relative to the ground-truth
label. The parameters of the RoBERTa encoder are fixed. We set the learning
rate to $5\times 10^{-4}$ and batch size to $128$. Unless specified, all the
models in this paper are trained with Adam Kingma and Ba (2015) optimizer and
training will be stopped if there is no improvement on the validation set for
$2$ consecutive epochs.

For BART models, we follow the instructions provided by Fairseq Ott et al.
(2019) to set the training hyperparameters on CNN/DM and XSum. We use the same
hyperparameters for CNN/DM and NYT, except that we adopt a linear learning
rate decay of $30{,}000$ steps in total for NYT.

During test, we use a beam size of $5$, $5$, $6$ for CNN/DM, NYT, and XSum,
respectively. To reduce computational cost, we use beam size $1$ for our
analysis experiments on all datasets. The length penalties are $2.0$, $1.5$
and $1.0$ for CNN/DM, NYT, and XSum, following Lewis et al. (2020). We set the
minimal and maximal lengths during decoding as: $55$ and $140$ for CNN/DM, $0$
and $140$ for NYT, and $10$ and $60$ for XSum.

##  Appendix B Head Selection

For CNN/DM, we apply masking to all heads at layer 1. The ROUGE-1/2/L F1 on
the analysis set are $36.43/16.02/33.59$.

For NYT, we apply masking to 12 heads at layer 3. The indices of heads are: 3,
4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15. The ROUGE-1/2/L F1 on the analysis set
are $55.27/39.20/48.16$.

For XSum, we apply masking to 12 heads at layer 3. The indices of heads are:
1, 2, 3, 4, 6, 7, 8, 9, 11, 13, 14, 15. The ROUGE-1/2/L F1 on the analysis set
are $45.77/22.82/37.60$.

##  Appendix C Content Selection Effects on XSum and NYT

The content selection effects for BART models fine-tuned on XSum and NYT,
measured by the ROUGE improvement from the uniform attention weight setting to
the oracle masking setting, are shown in Fig. 6.

On all three datasets, it is more effective to constrain attentions to salient
content at the top layers. Especially, the top layer yields the most ROUGE-1
improvement. Moreover, the ROUGE improvement by a specific head varies among
different datasets.

(a) XSum

(b) NYT

Figure 6: ROUGE-1 improvement with oracle masks for each head at each layer on
the analysis sets of XSum and NYT.

##  Appendix D Additional Results for Synergism Analysis

We show the synergism analysis for models fine-tuned on XSum and NYT in Fig.
7. They both echo the observation on CNN/DM that multiple heads have strong
collaborative effects and heads may not be independent at pinpointing
different salient content.

(a) XSum

(b) NYT

Figure 7: [Left] ROUGE-1 F1 improvement by incrementally applying oracle
masking to the next head with most ROUGE improvement per layer on XSum and
NYT. Dotted lines indicate that the newly masked heads do not have individual
ROUGE improvements. [Right] ROUGE-1 recall improvement by masking all heads
vs. sum of improvement by masking each head separately on XSum and NYT. Better
displayed with color.

##  Appendix E Attention Focus

We show the percentages of each type of attendees on the analysis sets of
XSum, NYT, and CNN/DM in Fig. 8, Fig. 9, and Fig. 10, respectively. We find
that heads have similar focus for salient words, content words, and the last
word across different datasets. Interestingly, the attention focus for the
first word on NYT is different from other datasets. On NYT, many articles
start with all capitalized words, which might become the focus of some heads.

(a) copy salient

(b) non-copy salient

(c) copy content

(d) non-copy content

(e) first

(f) last

Figure 8:  Percentages of copy salient, non-copy salient, copy content, non-
copy content, first and last attendees for each head at each layer on the
analysis set of XSum.

(a) copy salient

(b) non-copy salient

(c) copy content

(d) non-copy content

(e) first

(f) last

Figure 9:  Percentages of copy salient, non-copy salient, copy content, non-
copy content, first and last attendees for each head at each layer on the
analysis set of NYT.

(a) non-copy content

(b) last

Figure 10:  Percentages of non-copy content and last attendees for each head
at each layer on the analysis set of CNN/DM.


THE LIEB-OXFORD BOUND AND THE OPTIMAL TRANSPORT LIMIT OF DFT
LIEB-OXFORD BOUND AND SCE

1Michael SeidlM. Seidl
2Tarik BenyahiaT. Benyahia

3Derk P. KooiD.P. Kooi

4Paola Gori-GiorgiP. Gori-Giorgi

1Department of Chemistry & Pharmaceutical Sciences and Amsterdam Institute of Molecular and Life Sciences (AIMMS), Faculty of Science, Vrije Universiteit, De Boelelaan 1083, 1081HV Amsterdam, The Netherlands
2Department of Chemistry & Pharmaceutical Sciences and Amsterdam Institute of Molecular and Life Sciences (AIMMS), Faculty of Science, Vrije Universiteit, De Boelelaan 1083, 1081HV Amsterdam, The Netherlands
3Department of Chemistry & Pharmaceutical Sciences and Amsterdam Institute of Molecular and Life Sciences (AIMMS), Faculty of Science, Vrije Universiteit, De Boelelaan 1083, 1081HV Amsterdam, The Netherlands
4Department of Chemistry & Pharmaceutical Sciences and Amsterdam Institute of Molecular and Life Sciences (AIMMS), Faculty of Science, Vrije Universiteit, De Boelelaan 1083, 1081HV Amsterdam, The Netherlands<EMAIL_ADDRESS>
We review and illustrate with several examples the connection between the search for lower bounds for the optimal constant in the Lieb-Oxford inequality and the optimal transport limit (or strictly-correlated-electrons limit) of density functional theory. We focus in particular on several contributions from Elliott Lieb which already hinted at this connection.

§ INTRODUCTION

Elliott Lieb's work has been a continuous source of inspiration for researchers working on density functional theory (DFT) and quantum chemistry in general. In this small chapter we have chosen to focus on three among Elliott's papers (denoted throughout as L79 <cit.>, LO81 <cit.> and L83 <cit.>), which had a profound influence on our work on the so-called strictly correlated electrons (SCE) limit of DFT (also known as the optimal transport or semiclassical limit of the Levy-Lieb functional), and its applications to the Lieb-Oxford inequality. We will go through a series of simple worked-out SCE examples that illustrate and support specific sentences and equations in these three papers. We hope that, despite their simplicity, Elliott will enjoy these examples that have been crafted here explicitly for him.

The chapter is organised as follows. We will first review, in Sec. <ref>, the Lieb-Oxford inequality  <cit.>, including lower bounds on the optimal constant for different particle numbers $N$ derived from SCE calculations. The theory and ideas behind SCE are illustrated and connected to Elliott's work in Sec. <ref>, with various worked-out examples, including (Sec. <ref>) a construction from LO81 <cit.>, which was essentially a SCE state (although formulated at the time in a slightly different way), discussing a simple generalization. Then in Sec. <ref> we use SCE to do a computation that was imagined by Elliott in L83 <cit.>. Finally, we conclude with what we believe is an intriguing question on the optimal constants in the LO inequality for different particle numbers $N$.

§ LIEB-OXFORD (LO) BOUND

§.§ General concepts

Any (correctly normalized and antisymmetrized) $N$-electron wave function $\Psi$ is mapped onto an electron density
$\rho(\rv)\equiv\rho_{\Psi}(\rv)$, which we write $\Psi\mapsto\rho$, by the usual definition
ρ() = N∑_σ_1,...,σ_N∫d _2 ⋯∫d _N
|Ψ(,σ_1; _2,σ_2; ...; _N,σ_N)|^2.
The density is normalized to $N$, the total number of electrons in the system,
∫d^3r ρ() = N.
In terms of the operator
V̂_ee = 1/2∑_i,j=1^N (ij)1/|_i-_j|,
the Coulomb electron-electron repulsive energy in the state $\Psi$ is the expectation
I(Ψ) = ⟨Ψ|V̂_ee|Ψ⟩,
using the notation $I(\Psi)$ of L83, see eq (5.3) there. The Hartree functional
U[ρ] = 1/2∫d^3r∫d^3r' ρ() ρ(')/|-'|,
denoted as $U[\rho]\equiv D(\rho)$ in L83, is called the direct part of $I(\Psi)$, while its indirect part is the difference
W[Ψ] = ⟨Ψ|V̂_ee|Ψ⟩ - U[ρ_Ψ].
In the notation of L83, see Eq. (5.4) there, this same definition reads
E(Ψ) = I(Ψ) - D(ρ_Ψ).
For clarity, we report in table <ref> an overview of the notation used in LO81 and our previous work (denoted SVG) on the search of optimal constants for the bound  <cit.>.

Quantity     L83     SVG
Coulomb repulsive energy                        $I(\Psi)$     $\langle\Psi|\hat{V}_{\rm ee}|\Psi\rangle$
direct   part of $I(\Psi)$ (Hartree energy)     $D(\rho)$     $U[\rho]$
indirect part of $I(\Psi)$                      $E(\Psi)$     $W[\Psi]$

Notation used in L83 <cit.> and in SVG <cit.>.

§.§ The bound

Referred to as the Lieb-Oxford (LO) bound in the literature, Eq. (6) of LO81 or Eq. (5.7) of L83 reads
⟨Ψ|V̂_ee|Ψ⟩ ≥ U[ρ_Ψ] - C∫d  ρ_Ψ()^4/3,
where the constant $C>0$ has an unknown optimum, defined as the minimum value for which the inequality holds for any possible system. Writing

C ≥ U[ρ_Ψ]-⟨Ψ|V̂_ee|Ψ⟩/∫d  ρ_Ψ()^4/3 ≡ λ_C[Ψ]
(alternatively, SVG uses $\lambda[\Psi]=A_3\,\lambda_C[\Psi]$, where $A_3=\frac34(\frac3{\pi})^{1/3}\approx0.739$), we see that this optimum is a maximum (or a supremum),
C = sup_Ψ λ_C[Ψ].
We can also restrict ourselves to wave functions $\Psi\mapsto N$ with a given particle number $N$, and define
C(N) = sup_Ψ ↦N λ_C[Ψ]        (N∈).
Lieb and Oxford have then shown in LO81 that the corresponding unknown optimal constants $C(N)$ are monotonically increasing with growing $N$,
C(N) ≤ C(N+1),        lim_N→∞C(N)=C.

The LO bound has played (and continues to play) a very important role in the development of approximate exchange-correlation functionals in DFT, as very clearly reviewed and illustrated in the Chapter by Perdew and Sun in this same volume.

We focus here on lower and upper bounds for the unknown optimal constants $C$ of Eq. (<ref>) and $C(N)$ of Eq. (<ref>), which can be obtained, respectively, by:

$\bullet$ Challenging the LO bound of Eq. (<ref>), which means:

Finding a $\Psi$ with a particularly high value $\lambda_C[\Psi]$ in Eq. (<ref>), yielding a lower bound $C\ge\lambda_C[\Psi]$.

$\bullet$ Tightening the LO bound of Eq. (<ref>), which means:

Applying general theory to find an upper bound $\overline{C}$ for $C$, yielding $C\le\overline{C}$.

This is illustrated schematically in Fig. <ref>, which shows the tightest upper and lower bounds currently known for $C$. The current best lower bound  corresponds to the $\Psi$ (or, better, the probability density $|\Psi|^2$), describing the bcc Wigner crystal <cit.>, which has been suggested several times <cit.> as the actual optimal $C$. However, this hypothesis still remains without proof.

Schematic illustration of the search for a lower bound for the optimal constant $C$ in the Lieb-Oxford inequality: each time we find (see the encircled $\Psi$ in the figure) a wavefunction (or just a probability density $|\Psi|^2$) yielding a value for $\lambda_C[\Psi]$ of Eq. (<ref>) higher than any value previously observed, we improve the lower bound for $C$. Currently, the highest value 1.4442 ever observed for $\lambda_C[\Psi]$ is provided by the bcc Wigner crystal energy <cit.>, but it is unknown whether this is the actual optimal value. The upper bounds on $C$, instead, are more difficult to derive, as they need a rigorous proof <cit.>.

The increasingly tighter lower and upper bounds for $C$ and $C(N)$ from the literature are also collected in table <ref>.

bound for $C$ and $C(N)$     reference     year
$C\le 8.52$      <cit.>      1979
$C\le 1.68$      <cit.>     1981
$C\le 1.6359$      <cit.>     1999
$C\le 1.5765$      <cit.>     2022
$C\ge C(2)\ge 1.234\;=\;\Lambda_C[\rho_{81}]$      <cit.>     1981
$C\ge C(2)\ge 1.256\;=\;\Lambda_C[\rho_{16A}]$     <cit.>      2016

$C\ge C(60)\ge 1.4119\;=\;\Lambda_C[\rho_{16B}]$     <cit.>      2016

$C\ge 1.4442$     <cit.>      2019

Upper and lower bounds for the optimal constants $C$  and $C(N)$ of Eq. (<ref>),  where $\Lambda_C[\rho]$ is the density functional of Eq. (<ref>).

The bound of Eq. (<ref>) is not convex in $\rho$ (it is not even positive) <cit.>.
Alternatively, in Eq. (5.8) of L83 Lieb defines the functional
Ĩ(ρ)   =   inf_Ψ↦ρ ⟨Ψ|V̂_ee|Ψ⟩

≡   V_ee^SCE[ρ].
In the second line, we have introduced the designation $V_{\rm ee}^{\rm SCE}[\rho]$, Eq. (21) of SVG <cit.>. There is no need to distinguish between pure states and ensembles in the definition of the functional <cit.>. Furthermore, the minimization can be done over probability densities $P$ <cit.>,

\begin{equation}
V_{\rm ee}^{\rm SCE}[\rho] = \min_{P \mapsto \rho}\int \mathrm{d} \rv_1 \cdots\int \mathrm{d} \rv_N\,
P\big(\rv_1,\,\rv_2,\,...,\,\rv_N\big)\sum_{i > j} \frac{1}{|\rv_i - \rv_j|}.
\end{equation}

The SCE functional will be considered in detail in the following section <ref>.
Instead of Eq. (<ref>), we now obtain a convex bound,
⟨Ψ|V̂_ee|Ψ⟩ ≥ Ĩ(ρ) ≡ V_ee^SCE[ρ],
see Eq. (5.8) of L83. If the RHS $\tilde{I}(\rho)\equiv V_{\rm ee}^{\rm SCE}[\rho]$ had a simple explicit dependence on the density $\rho$, the bound given by Eq. (<ref>) would be preferable to the one of Eq. (<ref>). In particular, Eqs. (<ref>) and (<ref>) can now be written <cit.> as a supremum with respect to densities $\rho$,
C = sup_ρ Λ_C[ρ],        C(N) = sup_ρ ↦N Λ_C[ρ],
where the density functional $\Lambda_C[\rho]$ is defined as

\begin{equation}
\Lambda_C[\rho]\;\equiv\;\frac{U[\rho]-V^{\rm SCE}_{\rm ee}[\rho]}{\int d^3r\,\rho(\rv)^{4/3}}. \label{Csuppho}
\end{equation}

§ STRICTLY CORRELATED ELECTRONS (SCE)

We now turn in detail to the functional $V_{\rm ee}^{\rm SCE}[\rho]$, denoted as $\tilde{I}(\rho)$ in L83 and defined in Eq. (<ref>) above.

In Ref. <cit.>, Elliott Lieb wrote:
“Any reader who is devoted to abstract density functional theory [...]
should try to guess a plausible form for $\tilde{I}(\rho)$. (Proving it is another matter.) It will quickly be seen that
$\tilde{I}(\rho)$ must be extremely complicated, and to say that it is “nonlocal” is an understatement.”

By now, we know that the functional $\tilde{I}(\rho)$ corresponds to a multimarginal optimal transport (OT) problem <cit.> and provides the semiclassical limit ($\hbar\to 0$) of the Levy-Lieb functional <cit.>.
But even before these rigorous works, the functional $\tilde{I}(\rho)$ was constructed explicitly from the physical idea of “strictly-correlated electrons” for some special cases: $N=2$ electrons in a spherically symmetric density <cit.>, a general number of electrons $N$ in one dimensional systems <cit.>, and, later, a general $N$-electron density with spherical symmetry <cit.>. The first two constructions have been later proven to be exact in Refs. <cit.> and <cit.>, respectively, while for the construction for the general spherical $N$-electron case counterexamples have been found <cit.>. Nonetheless, the spherically-symmetric construction of Ref. <cit.> still provides a valid probability density corresponding to a given $\rho$, and thus a valid $\Lambda_C[\rho]$ that can be used to get improved lower bounds for the optimal constant <cit.>.

In the following, we provide simple explicit constructions for $V_{\rm ee}^{\rm SCE}[\rho]$ for some of these special cases, and we then show that Lieb and Oxford had actually found in LO81, for $N=2$, a spherical density for which they could also construct what we call today the SCE (or Monge) solution.

§.§ $N=2$ electrons with spherical symmetry (3D)

We here consider spherically symmetric densities $\rho(\rv)=\rho(r)$ for $N=2$ electrons,
∫_0^∞dr (4πr^2) ρ(r) = N        (N=2).
We shall first give the recipe for computing the value of $V_{\rm ee}^{\rm SCE}[\rho]\equiv\tilde{I}(\rho)$ for any given $\rho$, and then explain the physics behind it. The mathematical proof that $V_{\rm ee}^{\rm SCE}[\rho]$ for these densities yields the true infimum in Eq. (<ref>) is given in Ref. <cit.>.

Recipe: (Although we consider the case $N=2$ here, we keep the symbol “$N$” in our equations to make the origin of the numerical prefactors transparent.) For a given density $\rho$, we introduce the cumulative particle number function,
N_e(r) = ∫_0^rs (4πs^2) ρ(s).
As $r\ge0$ grows, $N_{\rm e}(r)$ increases monotonically from $N_{\rm e}(0)=0$ to $N_{\rm e}(r)\to N$ for $r\to\infty$.
When $\rho$ has compact support, $\rho(r)=0$ for $r>R$, we have $N_{\rm e}(R)=N$. Generally, including the cases with $R=\infty$, we use for the inverse function the notation
N^-1_e: [0,N]→[0,R],    ν↦r=N^-1_e(ν).
Then, fixed by the density $\rho(r)$, the SCE “co-motion function” $f(r)$ (aka, the Monge map in optimal transport theory) is defined as
f(r) = N_e^-1(2 - N_e(r)),
and the value of the density functional $V_{\rm ee}^{\rm SCE}[\rho]\equiv\tilde{I}(\rho)$ is given by
V_ee^SCE[ρ] = ∫_0^∞dr (4πr^2) 1/N ρ(r)/r+f(r)        (N=2).
This is indeed a highly nonlocal functional of the density $\rho$ !

Physics: In an SCE state with a density according to Eq. (<ref>), minimum repulsion energy in
Eq. (<ref>) is achieved as follows: The two electrons (upon simultaneous measurement of their positions)
are always found at positions $\rv_1$ and $\rv_2$ which are related by
_2 = (_1),    () = -f(r) /r.
In words: The vectors $\rv_1$ and $\rv_2$ always point in opposite directions from the origin, and the distance $r_2=|\rv_2|$ is fixed by $r_1=|\rv_1|$ through the co-motion function, $r_2=f(r_1)$.
Formally, using a Dirac $\delta$ distribution, the square modulus of such an SCE state can be written as
|Ψ(_1,_2)|^2 = ρ(_1)/N δ(_2-(_1)).
Eq. (<ref>) arises, since the probability for finding one electron inside a sphere with radius $r$ must be the same as for finding the second one outside a sphere with radius $f(r)$,
N_e(r) = 2 - N_e(f(r)).
Then Eq. (<ref>) arises, since the distance $d(r)=r+f(r)$ between the two electrons only depends on the single variable $r$ (representing the distance $r=|\rv|$  of one of them from the center).
Eq. (<ref>) implies that $f(f(r))\equiv r$, meaning that $f$ must be its own inverse,
f^-1(r) = f(r).
This result is necessary for identical particles, in which case the above condition $r_2=f(r_1)$ must be equivalent to $r_1=f(r_2)$.

Example 1: For illustration, we choose the cumulative number functions
N_e(r) = Nr^n/a^n+r^n            (a,n>0,  N∈)
(later we shall set $N=2$). These functions can be inverted explicitly,
N^-1_e(ν) = a(ν/N-ν)^1/n        (a,n>0,  N∈).
In terms of the derivatives $N'_{\rm e}(r)=\frac{d}{dr}\,N_{\rm e}(r)$, the corresponding densities are (see Fig. <ref>)
ρ_n(r) ≡ N'_e(r)/4πr^2 = N/4πa^3 nx^n-3/(1+x^n)^2    (x=r/a).

(-290, 120)$\frac{a^3}N\,\rho_n(r)$ (10, 5)$r/a$
Densities $\rho_n(r)$ of Eq. (<ref>) for $n=3$ (red), 4 (yellow), ..., up to $n=7$ (violet).

Setting $N=2$, we obtain a unique co-motion function $f(r)$, valid for all $n$,
f(r) ≡ N_e^-1(N - N_e(r)) = a(N-N_e(r)/N_e(r))^1/n = a^2/r        (N=2).
Consequently, Eq. (<ref>) now yields
≡    1/N∫_0^∞dr N'_e(r)/r+f(r)        (N=2)

=         na^n∫_0^∞dr r^n/(a^2+r^2)(a^n+r^n)^2 = 1/a V_n,
with the dimensionless integrals
V_n = n∫_0^∞dx x^n/(1+x^2)(1+x^n)^2.
For spherical densities $\;\rho(r)=\frac{N'_{\rm e}(r)}{4\pi r^2}$,  we generally have
U[ρ]   ≡   (4π)^2∫_0^Rr r ρ(r)∫_0^rs s^2 ρ(s)

=   ∫_0^Rr/r N'_e(r) N_e(r),

∫d^3r ρ()^4/3   =   1/(4π)^1/3∫_0^Rr/r^2/3 N'_e(r)^4/3.
Using here the functions $N_{\rm e}(r)$ from Eq. (<ref>), we obtain (for arbitrary $N\in\N$)
U[ρ_n]   =    N^2na^n∫_0^∞dr r^2(n-1)/(a^n+r^n)^3     =  N^2/a U_n,

∫d^3r ρ_n()^4/3   =   (Nna^n)^4/3/(4π)^1/3∫_0^∞dr r^4n/3-2/(a^n+r^n)^8/3
= N^4/3/a K_n,
with another set of dimensionless integrals,
U_n = n∫_0^∞dx x^2(n-1)/(1+x^n)^3,        K_n = n^4/3/(4π)^1/3∫_0^∞dx x^4n/3-2/(1+x^n)^8/3.
For $N=2$, we now can evaluate the density functional of Eq. (<ref>),
Λ_C[ρ_n] = 2^2U_n-V_n/2^4/3K_n.
The resulting values for the densities shown in Fig. <ref> are collected in the Table below.

$n$     $\Lambda_C[\rho_n]$
3     1.252 22
4     1.234 16
5     1.205 83
6     1.175 75
7     1.146 48

Vint = n*Integrate[(x^n)/( (1 + x^2) * ((1 + x^n)^2) ), x, 0, Infinity, Assumptions->Element[n, Integers]        n>0];
Uint = n*Integrate[x^(2*(n-1))/( (1 + x^n)^3 ), x, 0, Infinity, Assumptions->Element[n, Integers]        n>0];
Kint = ( ((n^4)/(4*Pi))^(1/3) )*Integrate[x^(4*n/3 - 2)/( (1 + x^n)^(8/3) ), x, 0, Infinity, Assumptions->Element[n, Integers]        n>0];
Vnum = N[Vint];
Unum = N[Uint];
Knum = N[Kint];
LamC = (4*Unum - Vnum)/((2^(4/3))*Knum)

For many integer values of $n$, the integrals $V_n$, $U_n$, and $K_n$ can be obtained analytically.

Numerically, the maximum seems to be reached for $n\approx2.86$, and it is equal to $\Lambda_C[\rho_n]\approx1.252~62$. This value is only slightly lower than the currently best known lower bound on the Lieb-Oxford constant $C(2)$, namely 1.256 <cit.>, and it is significantly better than the original LO81 lower bound, equal to 1.234, discussed in the next section.

§.§ An early example for an SCE state in LO81

In section 3 of LO81, Lieb and Oxford introduce for $N=2$ electrons in 3D the spherically symmetric probability density $|\psi|^2[(t,e),(s,h)]$, which we denote $|\Psi_{81}|^2$ here. The authors point out that “trivially, $I(\Psi_{81})\equiv\langle\Psi_{81}|\hat{V}_{\rm ee}|\Psi_{81}\rangle=1$ since the particles are always one unit apart", to find for the present Eq. (<ref>)
C(2) ≥ λ_C[Ψ_81] = 1.234.
$|\Psi_{81}(\rv_1,\rv_2)|^2$ turns out to describe an SCE state for $N=2$ electrons in the spherical density (LO81 uses dimensionless lengths in units of our $R$ here)
ρ_81(r) = 15/πR^5 (R-r)^2(R-r).
In this case, Eq. (<ref>) implies (see derivation below) the simple SCE co-motion function,
f(r) = R-r,
enforcing in fact a fixed distance $r+f(r)=R$ between the two electrons (“the particles are always one unit apart”),
V^SCE_ee[ρ_81] = 1/R.
To show that $f(r)=R-r$ is the true co-motion function for the density $\rho_{81}(r)$, we study the set of all possible cumulative number functions $\;N_{\rm e}:[0,R]\to[0,2],\;r\mapsto N_{\rm e}(r)$,  that in Eq. (<ref>) yield $f(r)=R-r$,
N_e(R-r) = 2 - N_e(r).

$\bullet$ Specifically for the density $\rho_{81}(r)$, with
N_e(r) ≡ ∫_0^rs (4πs^2) ρ_81(s) = 2/R^5 [6r^2 - 15R r + 10R^2] r^3,
plot=Plot[2*(6*(r^2) - 15*r + 10)*(r^3), r, 0, 1];
Show[plot, GridLines->0, 0.5, 1, 0, 1, 2]
we see that condition (<ref>) is satisfied, by replacing $r$ with $R-r$ and expanding.
There is no need for explicitly constructing the inverse function $N_{\rm e}^{-1}(\nu)$ here.

$\bullet$ For the general case, we substitute $r=\frac{R}2+u$ to rewrite condition (<ref>) as
N_e(R/2-u) = 2 - N_e(R/2+u).
In words: The co-motion function $f(r)=R-r$ arises then and only then, when the graph $y=N_{\rm e}(x)$ in the $xy$-plane has inversion symmetry about the point $(x,y)=(\frac{R}2,1)$.

Using Eq. (<ref>), we can construct infinitely many densities $\rho(r)=\frac{N'_{\rm e}(r)}{4\pi r^2}$ (for $r\in[0,R]$)
that all share the simple co-motion function $f(r)=R-r$ of Eq. (<ref>), implying that
V^SCE_ee[ρ] = 1/R.

Example 2: For $-\frac32\le a\le\frac38$, all the functions
y_a(s) = a s^5 - (2a+1/2) s^3 + (a+3/2) s        (-3/2≤a≤3/8),
having inversion symmetry about the origin $(s,y)=(0,0)$, are monotonic for $s\in[-1,1]$ with $y_a(\pm1)=\pm1$ [and $y'_a(\pm1)=0$].
Consequently, Eq. (<ref>) is satisfied when we choose
N_e(r)   =   1 + y_a(2r-R/R)

=   32 a(r/R)^5-80 a(r/R)^4+ 4 (16a-1)(r/R)^3+ 2 (3-8a)(r/R)^2.
The resulting densities $\rho_a(r)=\frac{N'_{\rm e}(r)}{4\pi r^2}$ all have the simple co-motion function $f(r)=R-r$.
With $a=\frac38$, the density $\rho_{81}(r)$ of LO81, Eq. (<ref>), is included, see Eq. (<ref>).

Using Eqs. (<ref>), (<ref>), plus Eq. (<ref>), we can evaluate the functional  $\Lambda_C[\rho_a]$ of Eq. (<ref>) for different values of $a\in[-\frac32,+\frac38]$, see Fig. <ref>.
We see that the maximum $\Lambda_C\approx1.2358$ is reached at $a\approx0.27<\frac38$, slightly higher than the value $\Lambda_C=1.234$ reported in LO81 for $a=\frac38$.

(-280, 130)$\Lambda_C[\rho_a]$ (5, 13)$a$
The values $\Lambda_C[\rho_a]$ for the functions $N_{\rm e}(r)$ of Eq. (<ref>).

§ SCE FOR A DENSITY WITH TWO “BUMPS” IN 1D

Again in L83, Elliott Lieb elaborates on the extreme nonlocality of $\tilde{I}(\rho)$ and writes “consider $N=2$ and $\rho$ consisting of two “bumps” $\rho_1$ and $\rho_2$, very far apart. As long as $\int\rho_1=\int\rho_2=1$, $\tilde{I}(\rho)\approx 0$, independently of $\rho_1$ and $\rho_2$. But when $\int\rho_1>1,\;\int\rho_2<1$, then $\tilde{I}(\rho)$ depends heavily on $\rho_1$ but not on $\rho_2$. The reason is that in the former case the two electrons can be far apart in the two bumps; in the latter case the two electrons must be partly close together in the first bump.”[We have corrected here a small typo in this sentence]

Since we know now how to exactly construct $\tilde{I}(\rho)$ for one-dimensional systems <cit.>, we follow here this suggestion, and consider a density $\rho_{\de}(x)$ for $N=2$ electrons in 1D, consisting of two rectangular “bumps” with different widths $a_{\pm}=(1\pm\de)a$ (but with equal local densities $\rho_0=\frac1a$), separated center-to-center by a large distance $R\gg a$,
ρ_(x) = {

1/a     (|x+R/2|≤a_+/2)
1/a     (|x-R/2|≤a_-/2)

0          (elsewhere)
}    (a_±=(1±)a,  0≤≤1).
The resulting cumulative particle number function,
N_e(x) = ∫_-∞^xy ρ(y),
has the inverse
N^-1_e(ν) = {
-R+(1+)a/2 + a·ν    (0≤ν<1+),

+R+(1-)a/2 + a·(ν-2)     (1+<ν≤2).
In this 1D case without “radial” symmetry, $\rho(-x)\ne\rho(x)$, the analogue of Eq. (<ref>) reads
f(x) = {
N_e^-1(N_e(x)+1)     (N_e(x)<1),

N_e^-1(N_e(x)-1)     (N_e(x)>1),.
with the cumulative particle number function $N_{\rm e}(x)=\int_{-\infty}^x\di y\,\rho(y)$ and its inverse $N^{-1}_{\rm e}(\nu)$.
Eq. (<ref>) arises, as two strictly correlated electrons in 1D are always separated by a distance over which the density integrates exactly to one particle,
∫_x^f(x)x ρ(x) = { 1     (N_e(x)<1),

-1     (N_e(x)>1)..
For the density $\rho_{\de}(x)$, the function $f(x)$ is plotted in Fig. <ref> (red linear segments).

(-280, 205)$f(x)$ (10, 118)$x$
(-220, 174)$a_+$ (-220, 151)$a_-$ (-160, 141)$R$
Co-motion functions $f(x)$ (red) and  $x$ (blue)

in the case $R=2$, $a=0.5$, and $\de=0.2$ (when $a_+=0.6$ and $a_-=0.4$). Generally, $f$ is its own inverse, $f^{-1}(x)=f(x)$.

Fig. <ref> shows that the distance between these two electrons is always either $a$ or $R$,
|f(x)-x| = {
a     (x_1<x<x_2)
R     (x_2<x<x_3 )
a     (x_3<x<x_4)
R     (x_5<x<x_6)
where $x_{1,4}=-\frac{R}2\mp\frac{a_+}2$, $x_{2,3}=-\frac{R}2\mp\frac{a_-}2$, $x_{5,6}=+\frac{R}2\mp\frac{a_-}2$.
Consequently (with $N=2$),
V^SCE_ee[ρ_]   ≡   ∫_-∞^∞1/N ρ_(x) x/|f(x)-x|

=   ρ_0/N[x_2-x_1/a+x_3-x_2/R+x_4-x_3/a+x_6-x_5/R]

=    /a + 1-/R.
As was to be expected, we find $V^{\rm SCE}_{\rm ee}[\rho_{0}]=\frac1R$ for $\de=0$ (with symmetric bumps), as this is the only case when each bump always accommodates exactly one electron, implying that the distance between these strictly correlated electrons is always exacly $R$. In the opposite case $\de=1$,
when $a_+=2a$ and $a_-=0$, there is only one single bump with width $2a$, and the two electrons always have the fixed distance $a$. Then, as predicted by Lieb, if we take the limit $R\to \infty$ we see that for $\delta=0$ the functional is zero, but when $\delta > 0$ we have a dependence on $\rho_1$ via the parameter $a$.

§ WHICH DENSITIES ARE THE MOST CHALLENGING FOR THE BOUND?

We conclude this chapter with an intriguing question: since we can now turn the search for improved lower bounds for $C(N)$ from wavefunctions to densities via the functional $\Lambda_C[\rho]$ of Eq. (<ref>), which kind of densities provide higher values of $\Lambda_C[\rho]$ for a given number of electrons $N$?

In Ref. <cit.> several different density profiles for $N=2$ electrons in a spherically symmetric density were used to compute $\Lambda_C[\rho]$. We report some of them, together with a few new ones, in table <ref>. We observe that an exponential density gives a rather high value of $\Lambda_C[\rho]$ and the profile $\sqrt{r}\, e^{-r}$ even more so. Here we have also extended this investigation to $N>2$, by using the variational solution of Ref. <cit.> for spherically symmetric densities. For example,
in table <ref> we have played with spherical density profiles of various atoms and rescaled them such that they integrate to 3. We were surprised to see that, again, the exponential profile (H atom) and the profile $\sqrt{r}\, e^{-r}$ give particularly high values. The same happens for $N=4$ and $N=10$, as shown in table <ref>.

profile     $\Lambda_{C}[\rho]\; (N=2)$
$e^{-50\,(r-1)^2}$      $0.932$
$\theta(1-r)$      $1.106$
$(1+r)^{-4}$      $1.154$
$e^{-\sqrt{r}}$     $1.224$
$e^{-r^{3/2}}$     $1.254$
$e^{-r}$     $1.255$
$\sqrt{r}\,e^{-r}$     $1.256$

Values of the functional $\Lambda_{C}[\rho]$ for different spherically-symmetric density profiles for the case $N=2$. The function $\theta(x)$ is the Heaviside step function.

profile     $\Lambda_{C}[\rho]\; (N=3)$
$\theta(1-r)$      $1.145$
$\rho_{\rm B}(r)$      $1.211$
$\rho_{\rm Be}(r)$      $1.235$
$\rho_{\rm Li}(r)$      $1.265$
$\rho_{\rm H}(r)$     $1.279$
$\sqrt{r}\,e^{-r}$     $1.282$

Values of the functional $\Lambda_{C}[\rho]$ for different spherically-symmetric density profiles for the case $N=3$. As an experiment, we have taken radial densities of various atoms (B, Be, H) and rescaled them such that they integrate to $N=3$. The radial densities for B, Be and H were computed using full CI in an aug-cc-pVDZ basis using pyscf <cit.>. The function $\theta(x)$ is the Heaviside step function.

profile     $\Lambda_{C}[\rho]\; (N=4)$     $\Lambda_{C}[\rho]\; (N=10)$
$\theta(1-r)$      $1.184$      $1.261$
$\rho_{Z=N}(r)$      $1.278$      $1.340$
$e^{-r}$     $1.307$     $1.364$
$\sqrt{r}\,e^{-r}$     $1.310$     $1.368$

Values of the functional $\Lambda_{C}[\rho]$ for different spherically-symmetric density profiles for the cases $N=4$ and $N=10$, including the corresponding neutral atom densities computed using full CI in an aug-cc-pVDZ basis using pyscf <cit.>. The function $\theta(x)$ is the Heaviside step function.

This work was supported by the Netherlands Organisation for Scientific Research (NWO) under Vici grant 724.017.001. T.B. is grateful to the Vrije Universiteit for the opportunity to contribute to this paper using the University Research Fellowship.