title: Reuse of computed Constant Values in Tensorflow
body: <p>I have encountered the following behavior and I'm not quite sure if this is a bug or if I'm missing out on something.</p>
  
  <h2>Minimal Example</h2>
  
  <pre><code># Create constant Matrix A
  A1 = tf.convert_to_tensor(A, dtype=tf.float64)
  
  # Compute the cholesky decomposition
  sqrtA1 = tf.linalg.cholesky(tfA1)
  
  xi = tf.placeholder(tf.float64, shape=[10000])
  
  # Matrix multiplication with the chompsky matrix
  RessqrtA1 = tf.tensordot(tfsqrtA1, tfxi, [[1],[0]])
  
  # Regular Matrix multiplication
  ResA1 = tf.tensordot(tfA1, tfxi, [[1],[0]])
  </code></pre>
  
  <p>Upon Computing something like this: </p>
  
  <pre><code>with tf.Session() as sess:
      _ = sess.run(ResA1, feed_dict={xi: np.random.randn(10000)})
      _ = sess.run(ResA1, feed_dict={xi: np.random.randn(10000)})
  </code></pre>
  
  <p>One could assume that the first <code>sess.run()</code> will take more time since the Graph has to be build and optimized. And that is what happens, here are some timings just like expected:</p>
  
  <pre><code>normal matmul
  Wall time: 36.1 s -- first run 
  
  41.1 ms ± 2.15 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) 
  --second run with timit
  </code></pre>
  
  <h2>Long Calculation times despite a constant Operation</h2>
  
  <p>Now one might think that the same holds when multiplying the <code>sqrtA1</code>, but <strong>unfortunately its not</strong>.<br>
  Computing the following:</p>
  
  <pre><code>with tf.Session() as sess:
      _ = sess.run(RessqrtA1, feed_dict={xi: np.random.randn(10000)})
      _ = sess.run(RessqrtA1, feed_dict={xi: np.random.randn(10000)})
  </code></pre>
  
  <p>Here are some timings: </p>
  
  <pre><code>cholesky
  Wall time: 2min 38s
  24.7 s ± 750 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
  </code></pre>
  
  <p>I ran the TensorFlow Profiler and obtained this information: </p>
  
  <pre><code>Profile:
  node name | requested bytes | total execution time | accelerator execution time | cpu execution time
  Cholesky                     800.00MB (100.00%, 50.00%),     24.30sec (100.00%, 99.81%),             0us (0.00%, 0.00%),     24.30sec (100.00%, 99.81%)
  MatMul                          80.00KB (50.00%, 0.00%),         46.45ms (0.19%, 0.19%),             0us (0.00%, 0.00%),         46.45ms (0.19%, 0.19%)
  Reshape                               0B (0.00%, 0.00%),            15us (0.00%, 0.00%),             0us (0.00%, 0.00%),            15us (0.00%, 0.00%)
  Const                         800.00MB (50.00%, 50.00%),            10us (0.00%, 0.00%),             0us (0.00%, 0.00%),            10us (0.00%, 0.00%)
  </code></pre>
  
  <h2>Difference between TF 1.7 and TF 1.6</h2>
  
  <p>Now for the <strong>interesting Part</strong>: This is <strong>only</strong> when you use <strong>Version 1.7.0 or higher</strong><br>
  I tried this with Version 1.6.0 and the Matmuls with the cholesky matrix took not longer than a normal matmul: </p>
  
  <pre><code>Wall time: 1min 18s -- first run
  49.7 ms ± 1.02 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
  -- timeit on the second sess.run()
  </code></pre>
  
  <p>When looking into the Computation in Version 1.6 with the Profiler you get: </p>
  
  <pre><code>Profile:node name | requested bytes | total execution time | accelerator execution time | cpu execution time
  Cholesky                    800.00MB (100.00%, 100.00%),       891us (100.00%, 100.00%),             0us (0.00%, 0.00%),       891us (100.00%, 100.00%)
  ======================End of Report==========================
  </code></pre>
  
  <p>Lastely, using this code, while replacing <code>RessqrtA1 = tf.tensordot(chol, tfxi, [[1],[0]])</code>,where <code>chol</code> is another <strong>Placeholder</strong>, gives us the desired outcome again.</p>
  
  <pre><code>%time temp = sess.run(tfsqrtA1)
  %timeit _ = sess.run(tfRessqrtA1, feed_dict={tfxi: np.random.randn(10000), chol: temp})
  
  cholesky
  Wall time: 1min 40s
  42.6 ms ± 3.86 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
  </code></pre>
  
  <h2>My Thoughts:</h2>
  
  <p>Somehow since Version 1.7 for every session run the cholesky calculation happens despite being from constant origin and thus it would suffice to calculate it once and reuse the outcome.  </p>
  
  <p>I think that such a computation happens in version 1.6 during the construction of the Graph as I measured that the <strong>calculation of the Cholesky Decomposition</strong> takes about <strong>26sec in V1.6</strong> (used a Placeholder so that it needs to be calculated during runtime), yet during the run it only took 900us as shown above.  </p>
  
  <p>In v1.7 they somehow changed this behavior. </p>
  
  <h2>The Question</h2>
  
  <p>Do you think this behavior change is normal or desired? If so, what would be a better approach when you have a similar situation? </p>
  
  <p>I'm thankful for every information, maybe someone else noticed some related problems using the linear algebra class. </p>
  
  <h2>What i have tried so far:</h2>
  
  <ul>
  <li>tinkering around with the float size (i.e. <code>tf.float32</code> instead of <code>tf.float64</code>)</li>
  <li>used different Versions in seperate environments.</li>
  <li>used the profiler to investigate the times of different Ops</li>
  <li>ran on a different pc </li>
  <li>timing in Python (e.g. <code>timeit</code> or <code>pycallgraph</code>) </li>
  </ul>