<blockquote>
  <p>How random is the Adam optimizer?</p>
</blockquote>

<p><em>The randomness in your result <code>y</code> is not something Adam brings for the fixed values of hyper-parameters. It is based on parameters <code>W</code> and biases <code>b</code> TensorFlow fills in in respect to <code>np.random.seed(0)</code> or <code>tf.set_random_seed(0)</code>.</em></p>

<p>As described in <a href="https://arxiv.org/pdf/1412.6980.pdf" rel="nofollow noreferrer">Adam</a>, it is RMSProp combined with Gradient Descent with momentum.</p>

<p>If you check out the <a href="https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam" rel="nofollow noreferrer">arguments</a>:</p>

<ul>
<li>lr: float >= 0. Learning rate.</li>
<li>beta_1: float, 0 &lt; beta &lt; 1. Generally close to 1.</li>
<li>beta_2: float, 0 &lt; beta &lt; 1. Generally close to 1.</li>
<li>epsilon: float >= 0. Fuzz factor. If None, defaults to K.epsilon().</li>
<li>decay: float >= 0. Learning rate decay over each update.</li>
<li>amsgrad: boolean. Whether to apply the AMSGrad variant of this algorithm from the paper "On the Convergence of Adam and Beyond".</li>
</ul>

<p>There are quite a few, and by default:</p>

<pre><code>__init__(
    lr=0.001,
    beta_1=0.9,
    beta_2=0.999,
    epsilon=None,
    decay=0.0,
    amsgrad=False, **kwargs
)
</code></pre>

<p>For the fixed set of the default hyper-parameters the results will be the same.</p>
How random is AdamOptimizer?
<p>Suppose:</p>

<ol>
<li>I feed the data in the same order to 10 AdamOptimizer.</li>
<li>All AdamOptimizer tries to minimize the same objective function.</li>
<li>The initial values for the variables are different for the 10 AdamOptimizer</li>
<li>Some of the variables (let call them set b) should have no effect on the minimal value of the objective function. But I don't know which variables are in set b before the minimization.</li>
<li>The objective function is deterministic.</li>
</ol>

<p>Would the variables in set b have different values for the 10 minimization?</p>

<p>I am trying to run the 10 minimization concurrently on a GPU.
The training data is large.</p>