title: Swap a TensorFlow Dataset input pipeline with a placeholder after training
body: <p>I'm working with the new <code>tf.data.Dataset</code> API and I can't seem to figure out how to perform inference. Ultimately, I want to convert my model to a TensorRT graph and run it on the TX2, and all of the <a href="https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/python_api/workflows/tf_to_tensorrt.html#Importing-the-UFF-Model-into-TensorRT-and-Building-an-Engine" rel="noreferrer">examples I have found</a> assume you have a <code>tf.placeholder</code> for the input. Here is pseudocode for how I am training. The [...] is just meant to be a placeholder since I didn't actually run the code. Let's not debate the model, as it is just suppose to give an example:</p>
  
  <pre><code>import tensorflow as tf
  
  # Setup iterator
  datain = tf.data.FixedLengthRecordDataset(datafiles, record_bytes1)
  labels = tf.data.FixedLengthRecordDataset(labelfiles, record_bytes2)
  dataset = tf.data.Dataset.zip((datain, labels))
  dataset = dataset.prefetch(batch_size)
  dataset = dataset.repeat(n_epoch)
  iterator = dataset.make_initializable_iterator()
  
  sess = tf.Session()
  sess.run(iterator.initializer)
  [batch_x, batch_y] = iterator.get_next()
  
  # Define model function (let's not debate model except as relevant to question)
  def model_fn(xin):
      x0 = tf.transpose(tf.reshape(xin, [...], name='input'))
      w = tf.Variable(tf.truncated_normal([...], stddev=0.1))
      x1 = tf.nn.conv2d(x0, w, strides=[...], padding='VALID')
      b = tf.Variable(tf.constant(0.0, shape=[...]))
      x2 = tf.nn.bias_add(x1, b)
      x3 = tf.nn.relu(x2, name='output')
      return x3
  
  # Setup training environment
  model = model_fn(batch_x)
  loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=model, labels=batch_y))
  optimizer = tf.train.AdamOptimizer(learning_rate=1e-3).minimize(loss)
  
  # Train Model
  while True:
      try:
          sess.run(optimizer)
      except tf.errors.OutOfRangeError:
          break
  
  # Save model
  saver = tf.train.Saver(name='saver')
  saver.save(sess, 'temp/path')
  </code></pre>
  
  <p>My question is how do I get this into TensorRT without having the input be a <code>tf.placeholder</code>? All of the example I can find use a <code>tf.placeholder</code> as the input. <a href="https://stackoverflow.com/questions/50061416/freezing-graph-in-tensorflow-when-using-tf-image-dataset/">This example</a> suggests that I can replace the iterator with a placeholder using the <code>SavedModel</code> class, but I cannot seem to find any documentation on how to accomplish that.</p>
  
  <p>Thanks!</p>
  
  <p><strong>EDIT: Here is my solution thanks to the help below</strong></p>
  
  <pre><code>from tensorflow.python.tools import optimize_for_inference_lib
  import uff
  
  # You can feed data to the IteratorGetNext node using feed_dict
  input_node_name = 'iterator_scope_name/IteratorGetNext'
  output_node_name = 'model_scope_name/output'
  
  # Run inference on the trained model:
  graph = tf.get_default_graph()
  batch_x = graph.get_tensor_by_name(input_node_name + ':0')
  networkout = graph.get_tensor_by_name(output_node_name + ':0')
  testdata, testlabel = custom_data_reader_fn(data_folder)
  # This will evaluate the model
  label = sess.run(networkout, feed_dict={batch_x: testdata})
  
  # Freeze model and create a UFF file:
  graph_def = graph.as_graph_def() # Convert the graph to a serialized pb
  frozen_graph_def = tf.graph_util.convert_variables_to_constants(sess,
      graph_def, [output_node_name])
  opt_graph_def = optimize_for_inference_lib.optimize_for_inference(
      frozen_graph_def, [input_node_name], [output_node_name],
      tf.float32.as_datatype_enum)
  uff.from_tensorflow(opt_graph_def, [output_node_name], quiet=False,
      output_filename='opt_model.uff')
  </code></pre>
  
  <p>that will write out a UFF file that TensorRT can utilize. The biggest issues that I encountered was:</p>
  
  <ol>
  <li>I didn't realize that the <code>optimize_for_inference_lib.optimize_for_inference</code> operation replaced the <code>iterator</code> with a <code>tf.placeholder</code></li>
  <li>I did not know what node to feed data to for evaluation: you can feed data to the <code>IteratorGetNext</code> node</li>
  </ol>