title: Why does my tensorflow code run so slow? Batch size is 1
body: <p>So I have the following objective function. It only works for one training example at a time (I have my reasons for avoiding batches). As you can see it's very lightweight so I expect it to run blazingly fast. </p>
  
  <pre><code>def objective(data, lam, item_biases, latent_items, latent_users):  
  
      user = data[0]
      rated_item = data[1]
      unrated_item = data[2]
  
  
      rated_item_bias = item_biases[rated_item]
      unrated_item_bias = item_biases[unrated_item]
  
      rated_latent_item = latent_items[rated_item]
      unrated_latent_item = latent_items[unrated_item]
  
      latent_user = latent_users[user]
  
      rated_pred = rated_item_bias + tf.reduce_sum(tf.multiply(rated_latent_item, latent_user))
      unrated_pred = unrated_item_bias + tf.reduce_sum(tf.multiply(unrated_latent_item, latent_user))
  
      difference = rated_pred - unrated_pred
      obj = tf.sigmoid(difference)
  
      obj += lam * tf.reduce_sum(rated_item_bias**2)
      obj += lam * tf.reduce_sum(unrated_item_bias**2) 
      obj += lam * tf.reduce_sum(rated_latent_item**2) 
      obj += lam * tf.reduce_sum(unrated_latent_item**2)
      obj += lam * tf.reduce_sum(latent_user**2)
      return obj
  </code></pre>
  
  <p>However, when I train it as such:</p>
  
  <pre><code>sess = tf.Session()
  sess.run(tf.global_variables_initializer())
  for iteration in range(5):
      samples = sample_triple(data, valid, 100000)
      for sample in tqdm(samples):
          cvalues = sess.run([trainer, obj], feed_dict={input_data:sample})
  </code></pre>
  
  <p>it runs abysmally slowly. I have a vanilla Python SGD implementation (e.g. hand-derived gradients) and it runs at 80K iterations/ second. This tensorflow implementation runs at about 30 iterations/ second. This is clearly unreasonable. Obviously having a batch size of 1 will slow things down. But to 30 iterations/ second?</p>
  
  <p>Is it the overhead of the <code>sess.run()</code> call? Is there a better way of making sure that I only take one sample at a time (it <em>has</em> to be SGD, no mini-batches)?</p>
  
  <p>EDIT: I should add that it's reasonably fast if I don't feed it one item per batch (the objective function had to change) but that's suboptimal as I noted...</p>