title: Turn list of tuples into binary tensors?
body: <p>I have a list of tuples in the form below. A given tuple represents a given pair of movies that a given user liked. All tuples, together, capture every combination of movie likes found in my data. </p>
  
  <pre class="lang-py prettyprint-override"><code>[(movie_a,movie_b),...(movie_a,movie_b)] 
  </code></pre>
  
  <p>My task is to create movie embeddings, similar to word embeddings. The idea is to train a single hidden layer NN to predict the most likely movie which any user might like, given a movie supplied. Much like word embeddings, the task is inconsequential; it's the weight matrix I'm after, which maps movies to vectors. </p>
  
  <p>Reference: <a href="https://arxiv.org/vc/arxiv/papers/1603/1603.04259v2.pdf" rel="nofollow noreferrer">https://arxiv.org/vc/arxiv/papers/1603/1603.04259v2.pdf</a></p>
  
  <p>In total, there are 19,000,000 tuples (training examples.) Likewise, there are 9,000 unique movie IDs in my data. My initial goal was to create an input variable, X where each row represented a unique movie_id, and each column represented a unique observation. In any given column, only one cell would be set to 1, with all other values set to 0.</p>
  
  <p>As an intermediate step, I tried creating a matrix of zeros of the right dimensions</p>
  
  <pre class="lang-py prettyprint-override"><code>X = np.zeros([9000,19000000])
  </code></pre>
  
  <p>Understandably, my computer crashed, simply trying to allocate sufficient memory to X. </p>
  
  <p>Is there a memory efficient way to pass my list of values into PyTorch, such that a binary vector is created for every training example?</p>
  
  <p>Likewise, I tried randomly sampling 500,000 observations. But similarly, passing 9000,500000 to np.zeroes() resulted in another crash.</p>
  
  <p>My university has a GPU server available, and that's my next stop. But I'd like to know if there's a memory efficient way that I should be doing this, especially since I'll be using shared resources.  </p>