title: How to interpret TensorFlow output?
body: <p>How do I interpret the TensorFlow output for building and executing computational graphs on GPGPUs?</p>
  
  <p>Given the following command that executes an arbitrary tensorflow script using the python API.</p>
  
  <blockquote>
    <blockquote>
      <p>python3 tensorflow_test.py > out</p>
    </blockquote>
  </blockquote>
  
  <p>The first part <code>stream_executor</code> seems like its loading dependencies.</p>
  
  <pre><code>I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so locally
  I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so locally
  I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so locally
  I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
  I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so locally
  </code></pre>
  
  <p>What is a <code>NUMA</code> node?</p>
  
  <pre><code>I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:900] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  </code></pre>
  
  <p>I assume this is when it finds the available GPU</p>
  
  <pre><code>I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
  name: Tesla K40c
  major: 3 minor: 5 memoryClockRate (GHz) 0.745
  pciBusID 0000:01:00.0
  Total memory: 11.25GiB
  Free memory: 11.15GiB
  </code></pre>
  
  <p>Some gpu initialization? what is DMA?</p>
  
  <pre><code>I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 
  I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y 
  I tensorflow/core/common_runtime/gpu/gpu_device.cc:755] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: Tesla K40c, pci bus id: 0000:01:00.0)
  </code></pre>
  
  <p>Why does it throw an error <code>E</code>?</p>
  
  <pre><code>E tensorflow/stream_executor/cuda/cuda_driver.cc:932] failed to allocate 11.15G (11976531968 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
  </code></pre>
  
  <p>Great answer to what the <code>pool_allocator</code> does: <a href="https://stackoverflow.com/a/35166985/4233809">https://stackoverflow.com/a/35166985/4233809</a></p>
  
  <pre><code>I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 3160 get requests, put_count=2958 evicted_count=1000 eviction_rate=0.338066 and unsatisfied allocation rate=0.412025
  I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 100 to 110
  I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 1743 get requests, put_count=1970 evicted_count=1000 eviction_rate=0.507614 and unsatisfied allocation rate=0.456684
  I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 256 to 281
  I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 1986 get requests, put_count=2519 evicted_count=1000 eviction_rate=0.396983 and unsatisfied allocation rate=0.264854
  I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 655 to 720
  I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 28728 get requests, put_count=28680 evicted_count=1000 eviction_rate=0.0348675 and unsatisfied allocation rate=0.0418407
  I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 1694 to 1863
  </code></pre>