title: FTRL implementation in tensorflow V.S. FTRL in Google's research paper
body: <p>I am interested in digging the details how FTRL is implemented in tensorflow. I find some information in the file <code>"gen_training_ops.py"</code> in the folder <code>/tensorflow/python/training</code>. In this file, the formula of FTRL algorithm is described as follows:</p>
  
  <pre><code>def apply_ftrl(var, accum, linear, grad, lr, l1, l2, lr_power,
  use_locking=None, name=None):
  r"""Update '*var' according to the Ftrl-proximal scheme
  accum_new = accum + grad * grad      ------ (1)
  linear += grad + (accum_new^(-lr_power) - accum^(-lr_power)) / lr * var        ------ (2)
  quadratic = 1.0 / (accum_new^(lr_power) * lr) + 2 * l2              ------ (3)
  var = (sign(linear) * l1 - linear) / quadratic if |linear| &gt; l1 else 0.0           ------(4)
  accum = accum_new        ------(5)
  </code></pre>
  
  <p>I am also reading the paper "Ad Click Prediction: a View from the Trenches" by Google in KDD'13. The formula of FTRL algorithm is given in page 2 of this paper. Comparing this two implementations, we find some connections:
  <code>var is w_{t,i} in the paper</code>; <code>l1 is lambda1 in the paper;</code> <code>linear is zi in the paper;</code> <code>lr is alpha in the paper;</code> <code>grad is gi in the paper;</code> <code>accum is ni in the paper.</code></p>
  
  <p>But also, there are some inconsistent points:
  according to the paper, the Equation (2) above should be </p>
  
  <pre><code>linear += grad - (accum_new^(-lr_power) - accum^(-lr_power)) / lr * var
  </code></pre>
  
  <p>also we can get the following equation by comparing the two implementations:</p>
  
  <pre><code>2l2 *alpha = beta + alpha * lambda2
  </code></pre>
  
  <p>For any expert who is familiar with the FTRL implementation in the tensorflow, can you help me to clarify the meaning the parameters given in tensorflow, and the connections with the FTRL code in Google's research paper "Ad Click Prediction: a View from the Trenches".</p>