<p>I have recently (April 2021) published a paper regarding this topic that you can find on arXiv (<a href="https://arxiv.org/abs/2104.07225" rel="noreferrer">https://arxiv.org/abs/2104.07225</a>).</p>
<p>There, Table 1 allows to review previous approaches to the problem in question, and the whole manuscript is about long text classification and proposing a new method called Text Guide. This new method claims to improve performance over naive and semi-naive text selection methods used in the paper (<a href="https://arxiv.org/abs/1905.05583" rel="noreferrer">https://arxiv.org/abs/1905.05583</a>) that was mentioned in one of the previous answers to this question.</p>
<p>Long story short about your options:</p>
<ol>
<li><p>Low computational cost: use naive/semi naive approaches to select a part of original text instance. Examples include choosing first n tokens, or compiling a new text instance out of the beginning and end of original text instance.</p>
</li>
<li><p>Medium to high computational cost: use recent transformer models (like Longformer) that have 4096 token limit instead of 512. In some cases this will allow for covering the whole text instance and the modified attention mechanism decreases computational cost, and</p>
</li>
<li><p>High computational cost: divide the text instance into chunks that fit a model like BERT with ‘standard’ 512 limit of tokens per instance, deploy the model on each part separately, join the resulting vector representations.</p>
</li>
</ol>
<p>Now, in my recently published paper there is a new method proposed called Text Guide. Text Guide is a text selection method that allows for improved performance when compared to naive or semi-naive truncation methods. As a text selection method, Text Guide doesn’t interfere with the language model, so it can be used to improve performance of models with ‘standard’ limit of tokens (512 for transformer models) or ‘extended’ limit (4096 as for instance for the Longformer model). Summary: Text Guide is a low-computational-cost method that improves performance over naive and semi-naive truncation methods. If text instances are exceeding the limit of models deliberately developed for long text classification like Longformer (4096 tokens), it can also improve their performance.</p>
How to use Bert for long text classification?
<p>We know that BERT has a max length limit of tokens = 512, So if an article has a length of much bigger than 512, such as 10000 tokens in text
How can BERT be used?</p>