Update README.md
Browse files
README.md
CHANGED
|
@@ -3,14 +3,14 @@ license: apache-2.0
|
|
| 3 |
---
|
| 4 |
# Skimformer
|
| 5 |
|
|
|
|
|
|
|
| 6 |
## Model description
|
| 7 |
Skimformer is a two-stage Transformer that replaces self-attention with Skim-Attention, a self-attention module that computes attention solely based on the 2D positions of tokens in the page. The model adopts a two-step approach: first, the skim-attention scores are computed once and only once using layout information alone; then, these attentions are used in every layer of a text-based Transformer encoder. For more details, please refer to our paper:
|
| 8 |
|
| 9 |
[Skim-Attention: Learning to Focus via Document Layout](https://arxiv.org/abs/2109.01078)
|
| 10 |
Laura Nguyen, Thomas Scialom, Jacopo Staiano, Benjamin Piwowarski, [EMNLP 2021](https://2021.emnlp.org/papers)
|
| 11 |
|
| 12 |
-
A collaboration between [reciTAL](https://recital.ai/en/) & [MLIA](https://mlia.lip6.fr/) (ISIR, Sorbonne Université)
|
| 13 |
-
|
| 14 |
## Citation
|
| 15 |
|
| 16 |
``` latex
|
|
|
|
| 3 |
---
|
| 4 |
# Skimformer
|
| 5 |
|
| 6 |
+
A collaboration between [reciTAL](https://recital.ai/en/) & [MLIA](https://mlia.lip6.fr/) (ISIR, Sorbonne Université)
|
| 7 |
+
|
| 8 |
## Model description
|
| 9 |
Skimformer is a two-stage Transformer that replaces self-attention with Skim-Attention, a self-attention module that computes attention solely based on the 2D positions of tokens in the page. The model adopts a two-step approach: first, the skim-attention scores are computed once and only once using layout information alone; then, these attentions are used in every layer of a text-based Transformer encoder. For more details, please refer to our paper:
|
| 10 |
|
| 11 |
[Skim-Attention: Learning to Focus via Document Layout](https://arxiv.org/abs/2109.01078)
|
| 12 |
Laura Nguyen, Thomas Scialom, Jacopo Staiano, Benjamin Piwowarski, [EMNLP 2021](https://2021.emnlp.org/papers)
|
| 13 |
|
|
|
|
|
|
|
| 14 |
## Citation
|
| 15 |
|
| 16 |
``` latex
|