title: Tensorflow FailedPreconditionError, but all variables have been initialized
body: <p><strong>EDIT:</strong> After trying several things, I have added the following to my code:</p>
  
  <pre><code>with tf.Session(graph=self.graph) as session:
      session.run(tf.initialize_all_variables())
      try:
          session.run(tf.assert_variables_initialized())
      except tf.errors.FailedPreconditionError:
          raise RuntimeError("Not all variables initialized!")
  </code></pre>
  
  <p>Now, occasionally this fails, i.e. <code>tf.assert_variables_initialized()</code> will raise FailedPreconditionError, even though immediately before it, <code>tf.initialize_all_variables()</code> was executed. Does anyone have any idea how this can happen?</p>
  
  <hr>
  
  <p>Original question:</p>
  
  <p><strong>Background</strong></p>
  
  <p>I'm running cross-validated (CV) hyperparameter search on a basic neural net created through Tensorflow, with GradientDescentOptimizer. At seemingly random moments I'm getting a FailedPreconditionError, for different Variables. For example (full stack trace at end of post):</p>
  
  <pre><code>FailedPreconditionError: Attempting to use uninitialized value Variable_5
       [[Node: Variable_5/read = Identity[T=DT_FLOAT, _class=["loc:@Variable_5"], _device="/job:localhost/replica:0/task:0/gpu:0"](Variable_5)]]
  </code></pre>
  
  <p>Some runs fail fairly fast, others not -- one has been running for 15 hours now without problems. I'm running this in parallel on multiple GPUs - not the optimization itself, but each CV fold.</p>
  
  <p><strong>What I've checked</strong></p>
  
  <p>From <a href="https://stackoverflow.com/questions/34001922/how-to-fix-failedpreconditionerror-tensor-flow-tutorial-using-numpy-pandas-data">this</a> and <a href="https://stackoverflow.com/questions/33788989/tensorflow-using-adam-optimizer">this</a> post I understand that this error occurs when attempting to use Variables that haven't been initialized using <code>tf.initialize_all_variables()</code>. However, I am 99% certain that I'm doing this (and if not, I'd expect it to <em>always</em> fail) - I'll post code below.</p>
  
  <p>The <a href="https://www.tensorflow.org/versions/r0.8/api_docs/python/client.html#FailedPreconditionError" rel="nofollow noreferrer">API doc</a> says that </p>
  
  <blockquote>
    <p>This exception is most commonly raised when running an operation that
    reads a tf.Variable before it has been initialized.</p>
  </blockquote>
  
  <p>"Most commonly" suggests that it can also be raised in different scenarios. So, for now the main question:</p>
  
  <p><strong>Question</strong>:
  are there other scenarios under which this exception may be raised, and what are they?</p>
  
  <p><strong>Code</strong></p>
  
  <p>MLP class:</p>
  
  <pre><code>class MLP(object):
      def __init__(self, n_in, hidden_config, n_out, optimizer, f_transfer=tf.nn.tanh, f_loss=mean_squared_error,
                   f_out=tf.identity, seed=None, global_step=None, graph=None, dropout_keep_ratio=1):
  
          self.graph = tf.Graph() if graph is None else graph           
          # all variables defined below
          with self.graph.as_default():
              self.X = tf.placeholder(tf.float32, shape=(None, n_in))
              self.y = tf.placeholder(tf.float32, shape=(None, n_out))
              self._init_weights(n_in, hidden_config, n_out, seed)
              self._init_computations(f_transfer, f_loss, f_out)
              self._init_optimizer(optimizer, global_step)
  
       def fit_validate(self, X, y, val_X, val_y, val_f, iters=100, val_step=1):
              [snip]
              with tf.Session(graph=self.graph) as session:
  VAR INIT HERE--&gt;tf.initialize_all_variables().run() #&lt;-- VAR INIT HERE
                  for i in xrange(iters):
                      [snip: get minibatch here]    
                      _, l = session.run([self.optimizer, self.loss], feed_dict={self.X:X_batch, self.y:y_batch})
                      # validate
                      if i % val_step == 0:
                          val_yhat = self.validation_yhat.eval(feed_dict=val_feed_dict, session=session)
  </code></pre>
  
  <p>As you can see, <code>tf.init_all_variables().run()</code> is always called before anything else is done. The net is initialized as:</p>
  
  <pre><code>def estimator_getter(params):
      [snip]    
      graph = tf.Graph()
      with graph.as_default():
          global_step = tf.Variable(0, trainable=False)
          learning_rate = tf.train.exponential_decay(params.get('learning_rate',0.1), global_step, decay_steps, decay_rate)
      optimizer = tf.train.GradientDescentOptimizer(learning_rate)
      net = MLP(config_num_inputs[config_id], hidden, 1, optimizer, seed=params.get('seed',100), global_step=global_step, graph=graph, dropout_keep_ratio=dropout)
  </code></pre>
  
  <p><strong>Full example stack trace:</strong></p>
  
  <pre><code>FailedPreconditionError: Attempting to use uninitialized value Variable_5
       [[Node: Variable_5/read = Identity[T=DT_FLOAT, _class=["loc:@Variable_5"], _device="/job:localhost/replica:0/task:0/gpu:0"](Variable_5)]]
  Caused by op u'Variable_5/read', defined at:
    File "tf_paramsearch.py", line 373, in &lt;module&gt;
      randomized_search_params(int(sys.argv[1]))
    File "tf_paramsearch.py", line 356, in randomized_search_params
      hypersearch.fit()
    File "/home/centos/ODQ/main/python/odq/cv.py", line 430, in fit
      return self._fit(sampled_params)
    File "/home/centos/ODQ/main/python/odq/cv.py", line 190, in _fit
      for train_key, test_key in self.cv)
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 766, in __call__
      n_jobs = self._initialize_pool()
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 537, in _initialize_pool
      self._pool = MemmapingPool(n_jobs, **poolargs)
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/site-packages/sklearn/externals/joblib/pool.py", line 580, in __init__
      super(MemmapingPool, self).__init__(**poolargs)
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/site-packages/sklearn/externals/joblib/pool.py", line 418, in __init__
      super(PicklingPool, self).__init__(**poolargs)
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/multiprocessing/pool.py", line 159, in __init__
      self._repopulate_pool()
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/multiprocessing/pool.py", line 223, in _repopulate_pool
      w.start()
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/multiprocessing/process.py", line 130, in start
      self._popen = Popen(self)
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/multiprocessing/forking.py", line 126, in __init__
      code = process_obj._bootstrap()
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
      self.run()
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/multiprocessing/process.py", line 114, in run
      self._target(*self._args, **self._kwargs)
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/multiprocessing/pool.py", line 113, in worker
      result = (True, func(*args, **kwds))
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 130, in __call__
      return self.func(*args, **kwargs)
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 72, in __call__
      return [func(*args, **kwargs) for func, args, kwargs in self.items]
    File "/home/centos/ODQ/main/python/odq/cv.py", line 131, in _fold_runner
      estimator = estimator_getter(parameters)
    File "tf_paramsearch.py", line 264, in estimator_getter
      net = MLP(config_num_inputs[config_id], hidden, 1, optimizer, seed=params.get('seed',100), global_step=global_step, graph=graph, dropout_keep_ratio=dropout)
    File "tf_paramsearch.py", line 86, in __init__
      self._init_weights(n_in, hidden_config, n_out, seed)
    File "tf_paramsearch.py", line 105, in _init_weights
      self.out_weights = tf.Variable(tf.truncated_normal([hidden_config[-1], n_out], stddev=stdev))
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 206, in __init__
      dtype=dtype)
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 275, in _init_from_args
      self._snapshot = array_ops.identity(self._variable, name="read")
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 523, in identity
      return _op_def_lib.apply_op("Identity", input=input, name=name)
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/op_def_library.py", line 655, in apply_op
      op_def=op_def)
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2117, in create_op
      original_op=self._default_original_op, op_def=op_def)
    File "/home/centos/miniconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
      self._traceback = _extract_stack()
  </code></pre>