<p>BERT is trained on pairs of sentences, therefore it is unlikely to generalize for much longer texts. Also, BERT requires quadratic memory with the length of the text, using too long texts might result in memory issues. In most implementations, it does not accept sequences longer than 512 subwords.</p>

<p>Making pre-trained Transformers work efficiently for long texts is an active research area, you can have a look at a paper called <a href="https://arxiv.org/pdf/1904.08398.pdf" rel="nofollow noreferrer"><em>DocBERT</em></a> to have an idea what people are trying. But it will take some time until there is a nicely packaged working solution.</p>

<p>There are also other methods for document embedding, for instance <a href="https://github.com/RaRe-Technologies/gensim" rel="nofollow noreferrer">Gensim</a> implements doc2vec. However, I would still stick with TF-IDF.</p>

<p>TF-IDF is typically very sensitive to data pre-processing. You certainly need to remove stopwords, in many languages it also pays off to do lemmatization. Given the specific domain of your texts, you can also try expanding the standard list of stop words by words that appear frequently in news stories. You can get further improvements by detecting and keeping together named entities.</p>
Using Bert and cosine similarity fo identify similar documents
<p>we have a news website where we have to match news to a particular user. </p>

<p>We have to use for the matching only the user textual information, like for example the interests of the user or a brief description about them.</p>

<p>I was thinking to threat both the user textual information and the news text as document and find document similarity. </p>

<p>In this way, I hope, that if in my profile I wrote sentences like: <em>I loved the speach of the president in Chicago last year</em>, and a news talks about: <em>Trump is going to speak in Illinois</em> I can have a match (the example is purely casual). </p>

<p>I tried, first, to embed my documents using TF-IDF and then I tried a kmeans to see if there was something that makes sense, but I don't like to much the results. </p>

<p>I think the problem derives from the poor embedding that TF-IDF gives me. </p>

<p>Thus I was thinking of using BERT embedding to retrieve the embedding of my documents and then use cosine similarity to check similarity of two document (a document about the user profile and a news). </p>

<p>Is this an approach that could make sense? Bert can be used to retrieve the embedding of sentences, but there is a way to embed an entire document?</p>

<p>What would you advice me? </p>

<p>Thank you</p>