title: Tensorflow: Custom data reader using py_func
body: <p>I am trying to queue up data from an hdf5 file. Since Tensorflow does not support hdf5, I created a python function that reads examples from an hdf5 file and raises <code>tf.errors.OutOfRangeError</code> when it reaches the end of the file. I then wrap this python function with <code>tf.py_func</code> and use it as an enqueue op for my queue.</p>
  
  <p>This is my code:</p>
  
  <pre><code>import h5py
  import tensorflow as tf
  from tensorflow.python.framework import errors
  import numpy as np
  
  def read_from_hdf5(hdf5_file, batch_size):
      h5py_handle = h5py.File(hdf5_file)
  
      # Check shapes from the hdf5 file so that we can set the tensor shapes
      feature_shape = h5py_handle['features'].shape[1:]
      label_shape = h5py_handle['labels'].shape[1:]
  
      #generator that produces examples for training. It will be wrapped by tf.pyfunc to simulate a reader
      def example_generator(h5py_handle):
          for i in xrange(0, h5py_handle['features'].shape[0]-batch_size+1, batch_size):
              features = h5py_handle['features'][i:i+batch_size]
              labels = h5py_handle['labels'][i:i+batch_size]
              yield [features, labels]
          raise errors.OutOfRangeError(node_def=None, op=None, message='completed all examples in %s'%hdf5_file)
  
      [features_tensor, labels_tensor] = tf.py_func(
          example_generator(h5py_handle).next,
          [],
          [tf.float32, tf.float32],
          stateful=True)
  
      # Set the shape so that we can infer sizes etc in later layers.
      features_tensor.set_shape([batch_size, feature_shape[0], feature_shape[1], feature_shape[2]])
      labels_tensor.set_shape([batch_size, label_shape[0]])
  
      return features_tensor, labels_tensor
  
  
  def load_data_from_filename_list(hdf5_files, batch_size, shuffle_seed=0):
      example_list = [read_from_hdf5(hdf5_file, batch_size) for hdf5_file in hdf5_files]
      min_after_dequeue = 10000
      capacity = min_after_dequeue + (len(example_list)+1) * batch_size #min_after_dequeue + (num_threads + a small safety margin) * batch_size
      features, labels = tf.train.shuffle_batch_join(example_list, batch_size, capacity=capacity, min_after_dequeue=min_after_dequeue, seed=shuffle_seed, enqueue_many=True)
      return features, labels, metadata
  </code></pre>
  
  <p>I expected that the <code>tf.errors.OutOfRangeError</code> would be handled by the QueueRunner, however, I get the following error and the program crashes. Is it possible to do this kind of reading from a py_func, and if it is, what am I doing wrong? If not, what approach should I use instead?</p>
  
  <pre><code>Traceback (most recent call last):
    File "/users/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/script_ops.py", line 85, in __call__
      ret = func(*args)
    File "build/bdist.linux-x86_64/egg/tronn/datalayer.py", line 27, in example_generator
      raise errors.OutOfRangeError(node_def=None, op=None, message='completed all examples in %s'%hdf5_file)
  tensorflow.python.framework.errors_impl.OutOfRangeError: completed all examples
  W tensorflow/core/framework/op_kernel.cc:993] Internal: Failed to run py callback pyfunc_13: see error log.
  </code></pre>