<p><strong>What are embeddings?</strong></p>
<blockquote>
<p>Word embedding is the collective name for a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the <strong>vocabulary are mapped to vectors of real numbers</strong>.</p>
<p>Conceptually it involves a mathematical embedding from a space with one dimension per word to a continuous vector space with much lower dimension.</p>
</blockquote>
<p>(Source: <a href="https://en.wikipedia.org/wiki/Word_embedding" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Word_embedding</a>)</p>
<p><strong>What is Word2Vec?</strong></p>
<blockquote>
<p>Word2vec is a group of related models that are used to produce word embeddings. These <strong>models are shallow, two-layer neural networks that are trained to reconstruct linguistic contexts of words</strong>.</p>
<p>Word2vec takes as its input a large corpus of text and produces a vector space, typically of several hundred dimensions, with <strong>each unique word in the corpus being assigned a corresponding vector in the space</strong>.</p>
<p>Word vectors are positioned in the vector space such that <strong>words that share common contexts in the corpus are located in close proximity to one another in the space</strong>.</p>
</blockquote>
<p>(Source: <a href="https://en.wikipedia.org/wiki/Word2vec" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Word2vec</a>)</p>
<p><strong>What's an array?</strong></p>
<blockquote>
<p>In computer science, an array data structure, or simply an array, is a data structure consisting of a collection of elements (values or variables), each identified by at least one array index or key.</p>
<p>An array is stored so that the position of each element can be computed from its index tuple by a mathematical formula.</p>
<p><strong>The simplest type of data structure is a linear array, also called one-dimensional array.</strong></p>
</blockquote>
<p><strong>What's a vector / vector space?</strong></p>
<blockquote>
<p>A vector space (also called a linear space) is a collection of objects called vectors, which may be added together and multiplied (&quot;scaled&quot;) by numbers, called scalars.</p>
<p>Scalars are often taken to be real numbers, but there are also vector spaces with scalar multiplication by complex numbers, rational numbers, or generally any field.</p>
<p>The operations of vector addition and scalar multiplication must satisfy certain requirements, called axioms, listed below.</p>
</blockquote>
<p>(Source: <a href="https://en.wikipedia.org/wiki/Vector_space" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Vector_space</a>)</p>
<p><strong>What's the difference between vectors and arrays?</strong></p>
<p>Firstly, the vector in word embeddings is not exactly the programming language data structure (so it's not <a href="https://stackoverflow.com/questions/15079057/arrays-vs-vectors-introductory-similarities-and-differences">Arrays vs Vectors: Introductory Similarities and Differences</a>).</p>
<p>Programmatically, a word embedding vector <strong>IS</strong> some sort of an array (data structure) of real numbers (i.e. scalars)</p>
<p>Mathematically, any element with one or more dimension populated with real numbers is a <a href="https://en.wikipedia.org/wiki/Tensor" rel="nofollow noreferrer">tensor</a>. And a vector is a single dimension of scalars.</p>
<hr />
<p>To answer the OP question:</p>
<p><strong>Why are word embedding actually vectors?</strong></p>
<blockquote>
<p>By definition, word embeddings are vectors (see above)</p>
</blockquote>
<p><strong>Why do we represent words as vectors of real numbers?</strong></p>
<blockquote>
<p>To learn the differences between words, we have to quantify the difference in some manner.</p>
</blockquote>
<p>Imagine, if we assign theses &quot;smart&quot; numbers to the words:</p>
<pre><code>&gt;&gt;&gt; semnum = semantic_numbers = {'car': 5, 'vehicle': 2, 'apple': 232, 'orange': 300, 'fruit': 211, 'samsung': 1080, 'iphone': 1200}
&gt;&gt;&gt; abs(semnum['fruit'] - semnum['apple'])
21
&gt;&gt;&gt; abs(semnum['samsung'] - semnum['apple'])
848
</code></pre>
<p>We see that the distance between <code>fruit</code> and <code>apple</code> is close but <code>samsung</code> and <code>apple</code> isn't. In this case, the single numerical &quot;feature&quot; of the word is capable of capturing some information about the word meanings but not fully.</p>
<p>Imagine the we have two real number values for each word (i.e. vector):</p>
<pre><code>&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; semnum = semantic_numbers = {'car': [5, -20], 'vehicle': [2, -18], 'apple': [232, 1010], 'orange': [300, 250], 'fruit': [211, 250], 'samsung': [1080, 1002], 'iphone': [1200, 1100]}
</code></pre>
<p>To compute the difference, we could have done:</p>
<pre><code>&gt;&gt;&gt; np.array(semnum['apple']) - np.array(semnum['orange'])
array([-68, 761])

&gt;&gt;&gt; np.array(semnum['apple']) - np.array(semnum['samsung'])
array([-848,    8])
</code></pre>
<p>That's not very informative, it returns a vector and we can't get a definitive measure of distance between the words, so we can try some vectorial tricks and compute the distance between the vectors, e.g. <a href="https://stackoverflow.com/questions/1401712/how-can-the-euclidean-distance-be-calculated-with-numpy">euclidean distance</a>:</p>
<pre><code>&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; orange = np.array(semnum['orange'])
&gt;&gt;&gt; apple = np.array(semnum['apple'])
&gt;&gt;&gt; samsung = np.array(semnum['samsung'])

&gt;&gt;&gt; np.linalg.norm(apple-orange)
763.03604108849277

&gt;&gt;&gt; np.linalg.norm(apple-samsung)
848.03773500947466

&gt;&gt;&gt; np.linalg.norm(orange-samsung)
1083.4685043876448
</code></pre>
<p>Now, we can see more &quot;information&quot; that <code>apple</code> can be closer to <code>samsung</code> than <code>orange</code> to <code>samsung</code>. Possibly that's because <code>apple</code> co-occurs in the corpus more frequently with <code>samsung</code> than <code>orange</code>.</p>
<p>The big question comes, <strong>&quot;How do we get these real numbers to represent the vector of the words?&quot;</strong>. That's where the Word2Vec / embedding training algorithms (<a href="https://link.springer.com/chapter/10.1007/3-540-33486-6_6" rel="nofollow noreferrer">originally conceived by Bengio 2003</a>) comes in.</p>
<hr />
<h1>Taking a detour</h1>
<p>Since adding more real numbers to the vector representing the words is more informative then why don't we just add a lot more dimensions (i.e. numbers of columns in each word vector)?</p>
<p>Traditionally, we compute the differences between words by computing the word-by-word matrices in the field of <a href="https://en.wikipedia.org/wiki/Distributional_semantics" rel="nofollow noreferrer">distributional semantics/distributed lexical semantics</a>, but the matrices become really sparse with many zero values if the words don't co-occur with another.</p>
<p>Thus a lot of effort has been put into <a href="https://en.wikipedia.org/wiki/Dimensionality_reduction" rel="nofollow noreferrer">dimensionality reduction</a> after computing the <a href="https://stackoverflow.com/questions/24073030/what-are-co-occurance-matrixes-and-how-are-they-used-in-nlp">word co-occurrence matrix</a>. IMHO, it's like a top-down view of how global relations between words are and then compressing the matrix to get a smaller vector to represent each word.</p>
<p>So the &quot;deep learning&quot; word embedding creation comes from the another school of thought and starts with a randomly (sometimes not-so random) initialized a layer of vectors for each word and learning the parameters/weights for these vectors and optimizing these parameters/weights by minimizing some loss function based on some defined properties.</p>
<p>It sounds a little vague but concretely, if we look at the Word2Vec learning technique, it'll be clearer, see</p>
<ul>
<li><a href="https://rare-technologies.com/making-sense-of-word2vec/" rel="nofollow noreferrer">https://rare-technologies.com/making-sense-of-word2vec/</a></li>
<li><a href="http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/" rel="nofollow noreferrer">http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/</a></li>
<li><a href="https://arxiv.org/pdf/1402.3722.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1402.3722.pdf</a> (more mathematical)</li>
</ul>
<p>Here's more resources to read-up on word embeddings: <a href="https://github.com/keon/awesome-nlp#word-vectors" rel="nofollow noreferrer">https://github.com/keon/awesome-nlp#word-vectors</a></p>
Why are word embedding actually vectors?
<p>I am sorry for my naivety, but I don't understand why word embeddings that are the result of NN training process (word2vec) are actually vectors.</p>

<p>Embedding is the process of dimension reduction, during the training process NN reduces the 1/0 arrays of words into smaller size arrays, the process does nothing that applies vector arithmetic.</p>

<p>So as result we got just arrays and not the vectors. Why should I think of these arrays as vectors?</p>

<p>Even though, we got vectors, why does everyone depict them as vectors coming from the origin (0,0)?</p>

<p>Again, I am sorry if my question looks stupid.</p>