|
--- |
|
license: apache-2.0 |
|
--- |
|
# Skimformer |
|
|
|
A collaboration between [reciTAL](https://recital.ai/en/) & [MLIA](https://mlia.lip6.fr/) (ISIR, Sorbonne Université) |
|
|
|
## Model description |
|
Skimformer is a two-stage Transformer that replaces self-attention with Skim-Attention, a self-attention module that computes attention solely based on the 2D positions of tokens in the page. The model adopts a two-step approach: first, the skim-attention scores are computed once and only once using layout information alone; then, these attentions are used in every layer of a text-based Transformer encoder. For more details, please refer to our paper: |
|
|
|
[Skim-Attention: Learning to Focus via Document Layout](https://arxiv.org/abs/2109.01078) |
|
Laura Nguyen, Thomas Scialom, Jacopo Staiano, Benjamin Piwowarski, [EMNLP 2021](https://2021.emnlp.org/papers) |
|
|
|
## Citation |
|
|
|
``` latex |
|
@article{nguyen2021skimattention, |
|
title={Skim-Attention: Learning to Focus via Document Layout}, |
|
author={Laura Nguyen and Thomas Scialom and Jacopo Staiano and Benjamin Piwowarski}, |
|
journal={arXiv preprint arXiv:2109.01078} |
|
year={2021}, |
|
} |
|
``` |
