<p>Stochastic Gradient Descent seems to require significant overparameterization in order to learn, here's one paper along those lines -- <a href="https://arxiv.org/abs/1301.3583" rel="nofollow noreferrer">"Big Neural Networks Waste Capacity"</a></p>
Why is embedding_lookup better than one hot encoding with a linear transformation?
<p>I think I'm missing something obvious here but would love some help figuring this out. </p>

<p>Say I have a million words and want to embed them as part of my model. 
With TF I can do an embedding lookup, though I need to provide a matrix of size [1m*space_size]. So for 50 dimensions that comes out to 50M trainable parameters. 
On the other hand I can one hot encode the a million words with a vector of dimension 20. I can embed that into a a space of dimension 50 with a [20*50] matrix for 1K parameters. Much cheaper. Since the weights of this matrix are still trainable, I'd expect to learn something about the words and if I need more capacity I can increase the size of the space. </p>

<p>That's in theory, in practice I tried and the model didn't learn anything. So my question is, why? 
Thanks</p>