<p>I advise you to read: Beltagy, Iz, Matthew E. Peters, and Arman Cohan. &quot;Longformer: The long-document transformer.&quot; arXiv preprint arXiv:2004.05150 (2020).</p>
<p>The main goal of this paper is that it is able to receive long document sequence tokens as input and is able to process long-term cross-partition context across the document with a linear computational cost.</p>
<p>Here, the sliding window attention mechanism uses <code>n = 512</code> tokens instead of what is known in the BERT model which takes <code>N=512</code> tokens as input sequence length.</p>
<hr />
<p> Longformer: The Long-Document Transformer</p>
<p>GitHub: <a href="https://github.com/allenai/longformer" rel="nofollow noreferrer">https://github.com/allenai/longformer</a></p>
<p>Paper: <a href="https://arxiv.org/abs/2004.05150" rel="nofollow noreferrer">https://arxiv.org/abs/2004.05150</a></p>
Best approach for semantic similarity in large documents using BERT or LSTM models
<p>I am trying to build a search application for resumes which are in .pdf format. For a given search query like &quot;who is proficient in Java and worked in an MNC&quot;, the output should be the CV which is most similar. My plan is to read pdf text and find the cosine similarity between the text and the query.</p>
<p>However, BERT has a problem with long documents. It supports a sequence length of only 512 but all my CVs have more than 1000 words. I am really stuck here. Methods like truncating the documents don't suit the purpose.</p>
<p>Is there any other model that can do this?</p>
<p>I could not find the right approach with models like Longformer and XLNet for this task.</p>
<pre><code>module_url = &quot;https://tfhub.dev/google/universal-sentence-encoder/4&quot; 
model = hub.load(module_url)
print (&quot;module %s loaded&quot; % module_url)

corpus = list(documents.values())
sentence_embeddings = model(corpus)
query = &quot;who is profiecient in C++ and has Rust&quot;
query_vec = model([query.lower()])[0]

doc_names = list(documents.keys())

results = []
for i,sent in enumerate(corpus):
  sim = cosine(query_vec, model([sent])[0])
  results.append((i,sim))
  #print(&quot;Document = &quot;, doc_name[i], &quot;; similarity = &quot;, sim)

print(results)
results= sorted(results, key=lambda x: x[1], reverse=True)
print(results)

for idx, distance in results[:5]:
  print(doc_names[idx].strip(), &quot;(Cosine Score: %.4f)&quot; % (distance))
</code></pre>