<p>BERT provides contextual representation, i.e., a joint representation of <em>a word and the context</em>. Unlike non-contextual embeddings, it is not as clear what the closest word should mean.</p>

<p>A good approximation of close words is certainly the prediction that BERT does as a (masked) language model. It basically says what similar words could be in the same context. However, this is not in the <a href="https://bert-as-service.readthedocs.io/en/latest/source/client.html#module-client" rel="noreferrer">client API of bert-as-service</a>. You can either implement the prediction layer yourself (I think it is just multiplication of the last layer with the embedding matrix + softmax, but maybe there is some additional projection going on) or use a different implementation such as <a href="https://github.com/huggingface/transformers" rel="noreferrer">Hugginface's Transformers</a>.</p>

<p>The most theoretically correct (and computationally expensive) solution would be running BERT on a large dataset and store pairs of words and corresponding contextual representations and then use e.g., <a href="https://github.com/facebookresearch/faiss" rel="noreferrer">faiss</a> for retrieving nearest neighbors that would include also the context, similarly as in the <a href="https://arxiv.org/abs/1911.00172" rel="noreferrer">nearest neighbors language models</a>.</p>
How to find the closest word to a vector using BERT
<p>I am trying to get textual representation(or the closest word) of given word embedding using BERT. Basically I am trying to get similar functionality as in gensim:</p>

<pre><code>&gt;&gt;&gt; your_word_vector = array([-0.00449447, -0.00310097, 0.02421786, ...], dtype=float32)
&gt;&gt;&gt; model.most_similar(positive=[your_word_vector], topn=1))
</code></pre>

<p>So far, I have been able to generate contextual word embedding using <a href="https://github.com/hanxiao/bert-as-service#getting-elmo-like-contextual-word-embedding" rel="noreferrer">bert-as-service</a> but can't figure out how to get closest words to this embedding. I have used pre-trained bert model (uncased_L-12_H-768_A-12) and haven't done any fine tuning.</p>