title: Saving a collection of variable length tensors to a TFRecords file in TensorFlow
body: <p>I'm trying to save a list of tensors of different lengths to a TFRecords file so that they can be easily loaded later on. The tensors in question are 1-dimensional arrays of integers.</p>
  
  <p>The reason for this is that the tensors are the result of processing a large text file. This file is very large and processing it is slow, so I don't want to have to repeat that step every time I want to run my algorithms. I originally thought of loading the text file into regular Python lists or numpy arrays and then pickling those, but the conversion from those lists to tensors itself takes a very long time, so I don't want to have to wait for that every time I run my script, either. It seems that tensors cannot be pickled directly, and even if there is some workaround for this I am under the impression that TFRecords is the "correct" way to save tensor data.</p>
  
  <p>However, I am not sure how to properly save the tensors to a TFRecords file and them load them back in as tensors. I did go through the TensorFlow tutorial in which MNIST data is saved to TFRecords files and then loaded, but there are a few differences between that and my use cases.</p>
  
  <p>The following is a block of code intended to replicate the issues I'm having in a simpler case.</p>
  
  <pre><code>import tensorflow as tf
  
  def _int64_list_feature(values):
     return tf.train.Feature(int64_list=tf.train.Int64List(value=values))
  
  filename = "/Users/me/tensorflow/test.tfrecords"
  writer = tf.python_io.TFRecordWriter(filename)
  example = tf.train.Example(features=tf.train.Features(feature={'datalist': _int64_list_feature([2,3])}))
  writer.write(example.SerializeToString())
  example = tf.train.Example(features=tf.train.Features(feature={'datalist': _int64_list_feature([8,5,7])}))
  writer.write(example.SerializeToString())
  writer.close()
  </code></pre>
  
  <p>First few lines are standard. I write two 1-D tensors to a TFRecords file, one with length 2 and one with length 3.</p>
  
  <pre><code>def read_my_file(filename_queue):
      reader = tf.TFRecordReader()
      _, serialized_example = reader.read(filename_queue)
      features = tf.parse_single_example(serialized_example, features={'datalist': tf.VarLenFeature(tf.int64), })
      datalist = features['datalist']
      return datalist
  </code></pre>
  
  <p>The helper function that it seems you are supposed to use. I am not 100% sure why this is necessary, but I couldn't get it to work without writing this, and all of the examples have something like this. In my case, the data is unlabeled so I don't have a labels variable.</p>
  
  <pre><code>filename_queue = tf.train.string_input_producer([filename], 2)
  datalists = read_my_file(filename_queue)
  datalists_batch = tf.train.batch([datalists], batch_size=2)
  </code></pre>
  
  <p>More "boilerplate"-style code from the examples. Batch size is 2 because I only have 2 examples in this code.</p>
  
  <p><code>datalists_batch</code> will now be a sparse tensor that contains both my vectors, <code>[2, 3]</code> and <code>[8, 5, 7]</code>, the first on top of the second. Therefore, I want to split them back into individual tensors. At this point, I was already concerned that the runtime of this might be pretty long too, because in my real code there are over 100,000 individual tensors that will be split.</p>
  
  <pre><code>split_list = tf.sparse_split(0, 2, datalists_batch)
  sp0 = split_list[0]
  sp1 = split_list[1]
  sp0_dense = tf.sparse_tensor_to_dense(sp0)
  sp1_dense = tf.sparse_tensor_to_dense(sp1)
  sp0_dense = tf.squeeze(sp0_dense)
  sp1_dense = tf.squeeze(sp1_dense)
  </code></pre>
  
  <p><code>split_list</code> is now a list of the individual tensors, still in sparse format (and all having a length equal to the length of the longest tensor, which is in this case 3. They are also 2-dimensional with the other dimension 1, since the <code>datalists_batch</code> tensor was 2D). I must now manipulate the tensors to get them into proper format. In the real code, I would of course use a for-loop, but there are only 2 examples in this case. First, I convert them to dense tensors. However, in the case of <code>sp0</code> this fills in the last index with a <code>0</code>, since this tensor has length 3. (This issue is discussed below.) Then, I "squeeze" them so that they are actually considered tensors with length 3 instead of 1x3.</p>
  
  <p>Finally, I need to remove the trailing zero from <code>sp0</code>. This part gave me difficulty. I don't know programmatically how many trailing zeros a particular tensor has. It is equal to the length of the longest tensor minus the length of this tensor, but I don't know the "real" lengths of the tensors without looking at the sparse indices, but I cannot access that without evaluating the "temp" tensor (since the indices are themselves tensors).</p>
  
  <pre><code>indices_0 = sp0.indices
  indices_1 = sp1.indices
  indices0_size = tf.shape(indices_0)
  indices1_size = tf.shape(indices_1)
  </code></pre>
  
  <p>These are necessary for the aforementioned slicing operations.</p>
  
  <pre><code>sess = tf.Session()
  init_op = tf.initialize_all_variables()
  sess.run(init_op)
  coord = tf.train.Coordinator()
  threads = tf.train.start_queue_runners(sess=sess, coord=coord)
  </code></pre>
  
  <p>Initializations.</p>
  
  <pre><code>sp0_fixed = tf.slice(sp0_dense, [0], [sess.run(indices0_size[0])])
  sp1_fixed = tf.slice(sp1_dense, [0], [sess.run(indices1_size[0])])
  sess.run(sp0_fixed)
  sess.run(sp1_fixed)
  </code></pre>
  
  <p>This is how I would do it. The problem is, I get strange errors when running these last three commands. I surmise that the problem is that I am creating new ops after sess.run has already been called (in the <code>sp0_fixed</code> line), so the graph is being run simultaneously. I think I should only run sess.run once. However, this makes it impossible for me to figure out the proper indices at which to slice each tensor (to remove trailing zeros). Thus, I don't know what to do next.</p>
  
  <p>I have surprisingly found nothing at all helpful on how to do something like this (save and load variable length tensors to/from files) on Google, TensorFlow documentation, and StackOverflow. I am quite sure that I'm going about this the wrong way; even if there is a workaround to rewrite the last four lines so that the program behaves correctly, the code overall seems excessively complicated to perform a very basic functionality.</p>
  
  <p>I would greatly appreciate any suggestions and feedback.</p>