title: Tensorflow LSTM - Simple model returns only zeros forever
body: <p>I build a very simple model to predict harmony for given melody. Melody is represented as one-hot vector, harmony as k-hot vector (sparse vector with up to 5 ones). One batch of harmony/melody has shape <code>(batch_size, 64, 48)</code>.</p>
  
  <p>Unfortunatelly, after few iterations (~100), this model starts to predict zeros everywhere (<code>prediction</code> tensor is full of zeros) and stays here forever. Loss decrease up to some point and it seems it just stuck here (probably because my output is full of zeros).</p>
  
  <p>After manually checking some records, dataset seems to be fine. My dataset contains ~10k records.</p>
  
  <p>Here is my model:</p>
  
  <pre><code>import itertools
  import tensorflow as tf
  from dataset import get_batch
  
  timesteps = 64
  melody_dim = 48
  harmony_dim = 48
  
  batch_size = tf.placeholder(dtype=tf.int32)
  melody_batch, harmony_batch = get_batch(['./dataset.tfrecords'], batch_size=batch_size)
  
  cell = tf.contrib.rnn.LSTMCell(128)
  output, state = tf.nn.dynamic_rnn(cell, melody_batch, dtype=tf.float32)
  logits = tf.contrib.layers.fully_connected(output, harmony_dim, activation_fn=None)
  
  loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=harmony_batch, logits=logits))
  solver = tf.train.AdamOptimizer(1e-4).minimize(loss)
  
  prediction = tf.greater(logits, 0)
  
  with tf.Session() as sess:
      sess.run(tf.global_variables_initializer())
  
      coord = tf.train.Coordinator()
      threads = tf.train.start_queue_runners(coord=coord)
  
      for i in itertools.count():
          _, curr_loss = sess.run([solver, loss], {
              batch_size: 32
          })
  
      coord.request_stop()
      coord.join(threads)
  </code></pre>
  
  <p>Anybody know where the problem could be? Should I use embedding or something similar? I suspect that the problem is because my label (harmony) vector is very sparse.</p>