<p><code>initial_accumulator_value</code> is indeed the \delta and it should not be initialized to <code>0</code>. A value of <code>0.01</code> is more appropriate but the default of <code>0.1</code> is fine.</p>

<p>Btw, if you are in the business of playing with optimizers, authors of Adagrad have a new optimizer <a href="https://arxiv.org/abs/1802.09568" rel="nofollow noreferrer">https://arxiv.org/abs/1802.09568</a> that performs considerably better than existing ones. It's TF implementation should be released fairly soon, Q2 2018.</p>
How Adagrad optimizer work in TensorFlow?
<ol>
<li><p>According to the initial paper, there should be a parameter named δ. But I can't find such argument in TensorFlow <em>AdagradOptimizer</em> construtor. </p></li>
<li><p>There is an argument  named <em>initial_accumulator_value</em>, it is suggested to set as 0, but TensorFlow uses 0.1 as default. It is proper for me to set it as 0?</p></li>
</ol>

<p>Thank you so much for your time!</p>

<p>Garrett</p>