<h3>Poincaré Embeddings</h3>
<blockquote>
<p>However, while complex symbolic datasets often exhibit a latent hierarchical structure, state-of-the-art methods typically learn embeddings in Euclidean vector spaces, which do not account for this property. For this purpose, we introduce a new approach for learning hierarchical representations of symbolic data by embedding them into hyperbolic space – or more precisely into an n-dimensional Poincaré ball.</p>
</blockquote>
<p>Poincaré embeddings allow you to create hierarchical embeddings in a non-euclidean space. The vectors on the outside of the Poincaré ball are lower in hierarchy compared to the ones in the center.</p>
<p><a href="https://i.stack.imgur.com/uZQMe.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uZQMe.png" alt="enter image description here" /></a></p>
<p>The transformation to map a Euclidean metric tensor to a Riemannian metric tensor is an open d-dimensional unit ball.</p>
<p><a href="https://i.stack.imgur.com/S2YhR.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/S2YhR.png" alt="enter image description here" /></a></p>
<p>Distances between 2 vectors in this non-euclidean space are calculated as</p>
<p><a href="https://i.stack.imgur.com/ZxoJ4.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ZxoJ4.png" alt="enter image description here" /></a></p>
<p>The <a href="https://arxiv.org/pdf/1705.08039.pdf" rel="nofollow noreferrer">research paper for Poincaré embeddings</a> is wonderfully written and you will find some wonderful implementations in popular libraries for them as well. Needless to say, they are under-rated.</p>
<p>Two implementations that you can use are found in -</p>
<ul>
<li><code>tensorflow_addons.PoincareNormalize</code></li>
<li><code>gensim.models.poincare</code></li>
</ul>
<h3>Tensorflow Addons implementation</h3>
<p>According to the documentation, for a 1D tensor, <code>tfa.layers.PoincareNormalize</code> computes the following output along axis=0.</p>
<pre><code>          (x * (1 - epsilon)) / ||x||     if ||x|| &gt; 1 - epsilon
output =
           x                              otherwise
</code></pre>
<p>For a higher dimensional tensor, it independently normalizes each 1-D slice along the <strong>dimension axis</strong>.</p>
<p>This transformation can be simply applied to an embedding of n-dims. Let's create a 5 dim embedding for each element of the time series. The dimension axis=-1 in this case, which is mapped from a euclidean space to a non-euclidean space.</p>
<pre><code>from tensorflow.keras import layers, Model, utils
import tensorflow_addons as tfa

X = np.random.random((100,10))
y = np.random.random((100,))


inp = layers.Input((10,))
x = layers.Embedding(500, 5)(inp)
x = tfa.layers.PoincareNormalize(axis=-1)(x)  #&lt;-------
x = layers.Flatten()(x)
out = layers.Dense(1)(x)

model = Model(inp, out)
model.compile(optimizer='adam', loss='binary_crossentropy')
utils.plot_model(model, show_shapes=True, show_layer_names=False)

model.fit(X, y, epochs=3)
</code></pre>
<pre><code>Epoch 1/3
4/4 [==============================] - 0s 2ms/step - loss: 7.9455
Epoch 2/3
4/4 [==============================] - 0s 2ms/step - loss: 7.5753
Epoch 3/3
4/4 [==============================] - 0s 2ms/step - loss: 7.2429
&lt;tensorflow.python.keras.callbacks.History at 0x7fbb14595310&gt;
</code></pre>
<h3>Gensim implementation</h3>
<p>Another implementation Poincare embeddings can be found in Gensim. Its very similar to what you would use when working with Word2Vec from Gensim.</p>
<p>The process would be -</p>
<ol>
<li>Train Gensim embeddings (word2vec or poincare)</li>
<li>Initialize Embedding layer in Keras with embeddings</li>
<li>Set the embedding layer as non-trainable</li>
<li>Train model for the downstream task</li>
</ol>
<pre><code>from gensim.models.poincare import PoincareModel

relations = [('kangaroo', 'marsupial'), ('kangaroo', 'mammal'), ('gib', 'cat'), ('cow', 'mammal'), ('cat','pet')]

model = PoincareModel(relations, size = 2, negative = 2)  #Change size for higher dims
model.train(epochs=10)

print('kangroo vs marsupial:',model.kv.similarity('kangaroo','marsupial'))
print('gib vs mammal:', model.kv.similarity('gib','mammal'))

print('Embedding for Cat: ', model.kv['cat'])
</code></pre>
<pre><code>kangroo vs marsupial: 0.9481239343527523
gib vs mammal: 0.5325816385250299

Embedding for Cat:  [0.22193988 0.0776986 ]
</code></pre>
<p>More details on training and saving Poincare embeddings can be found <a href="https://radimrehurek.com/gensim/models/poincare.html" rel="nofollow noreferrer">here</a>.</p>
How do I implement Poincaré Embeddings using tfa.layers.PoincareNormalize?
<p>I am trying to implement Poincaré embeddings as discussed in a paper by Facebook (<a href="https://arxiv.org/pdf/1705.08039.pdf" rel="nofollow noreferrer">Link</a>) for my hierarchical data. You may find a more accessible explanation of Poincaré embeddings <a href="https://medium.com/@srijithpoduval/explaining-poincar%C3%A9-embeddings-d7cb9e4a2bbf" rel="nofollow noreferrer">here</a>.</p>
<p>Based on the paper I have found some implementations for Tensorflow <a href="https://github.com/kousun12/tf_hyperbolic" rel="nofollow noreferrer">here</a> and <a href="https://github.com/qiangsiwei/poincare_embedding" rel="nofollow noreferrer">here</a> as well as <a href="https://www.tensorflow.org/addons/api_docs/python/tfa/layers/PoincareNormalize" rel="nofollow noreferrer">tfa.layers.PoincareNormalize</a> in Tensorflow Addons. The latter even had a link to the paper mentioned above, which makes me believe it could be a good starting point for me. However, I had no luck implementing tfa.layers.PoincareNormalize so far and also could not find any documentation except some generic information on the API page that I linked.</p>
<p>Does anyone know how this layer is supposed to be implemented to provide the embedding in hyperbolic space discussed in the paper? My starting point is an implementation with a standard Embedding layer as presented below (it is actually an entity embedding of a categorical variable)?</p>
<pre><code>input = Input(shape=(1, ))
model = Embedding(input_dim=my_input_dim, 
                    output_dim=embed_dim, name=&quot;my_feature&quot;)(input)
model = Reshape(target_shape=(embed_dim, ))(model)
model = Dense(1)(model)
model = Activation('sigmoid')(model)
</code></pre>
<p>Simply replacing the Embedding layer by tfa.layers.PoincareNormalize does not work due to different inputs. I assume that it could be placed somwhere after the embedding layer so that for the back propagation step the &quot;values&quot; are projected into hyperbolic space on each iteration, but had no luck with that so far either.</p>