title: Tensorflow: How do you monitor GPU performance during model training in real-time?
body: <p>I am new to Ubuntu and GPUs and have recently been using a new PC with Ubuntu 16.04 and 4 NVIDIA 1080ti GPUs in our lab. The machine also has an i7 16 core processor. </p>
  
  <p>I have some basic questions:</p>
  
  <ol>
  <li><p>Tensorflow is installed for GPU. I presume then, that it automatically prioritises GPU usage? If so, does it use all 4 together or does it use 1 and then recruit another if needed?</p></li>
  <li><p>Can I monitor in real-time, the GPU use/activity during training of a model?</p></li>
  </ol>
  
  <p>I fully understand this is basic hardware stuff but clear definitive answers to these specific questions would be great. </p>
  
  <p>EDIT:</p>
  
  <p>Based on this output - it this really saying that nearly all the memory on each one of my GPUs is being used?  </p>
  
  <p><a href="https://i.stack.imgur.com/11zI5.png" rel="noreferrer"><img src="https://i.stack.imgur.com/11zI5.png" alt="enter image description here"></a></p>