<p>You can leverage from the HuggingFace Transformers library that includes the following list of Transformers that work with long texts (more than 512 tokens):</p>
<ul>
<li><a href="https://huggingface.co/transformers/model_doc/reformer.html" rel="noreferrer">Reformer</a>: that combines the modeling capacity of a Transformer with an architecture that can be executed efficiently on long sequences.</li>
<li><a href="https://huggingface.co/transformers/model_doc/longformer.html" rel="noreferrer">Longformer</a>: with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer.</li>
</ul>
<p>Eight other recently proposed efficient Transformer models include Sparse Transformers (Child et al.,2019), Linformer (Wang et al., 2020), Sinkhorn Transformers (Tay et al., 2020b), Performers (Choromanski et al., 2020b), Synthesizers (Tay et al., 2020a), Linear Transformers (Katharopoulos et al., 2020), and BigBird (Zaheeret al., 2020).</p>
<p>The <a href="https://arxiv.org/pdf/2011.04006.pdf" rel="noreferrer">paper</a> from the authors from Google Research and DeepMind tries to make a comparison between these Transformers based on Long-Range Arena &quot;aggregated metrics&quot;:</p>
<p><a href="https://i.stack.imgur.com/TMD6h.png?s=256" rel="noreferrer"><img src="https://i.stack.imgur.com/TMD6h.png?s=256" alt="Performance, speed, and memory footprint of the Transformers" /></a></p>
<p>They also suggest that <em><strong>Longformers have better performance than Reformer when it comes to the classification task</strong></em>.</p>
How to use Bert for long text classification?
<p>We know that BERT has a max length limit of tokens = 512, So if an article has a length of much bigger than 512, such as 10000 tokens in text
How can BERT be used?</p>