title: TensorFlow strange memory usage
body: <p>I'm on an Ubuntu 19.10 machine (with KDE desktop environment) with 8GB of RAM, an i5 8250u and an MX130 gpu (2GB VRAM), running a Jupyter Notebook with tensorflow-gpu. </p>
  
  <p>I was just training some models to test their memory usage, and I can't see any sense in what I'm looking at.  I used KSysGUARD and NVIDIA System Monitor (<a href="https://github.com/congard/nvidia-system-monitor" rel="nofollow noreferrer">https://github.com/congard/nvidia-system-monitor</a>) to monitor my system during training.</p>
  
  <ol>
  <li><p>As I hit "train", on NVIDIA S.M. show me that memory usage is 100% (or near 100% like 95/97%) the GPU usage is fine.</p></li>
  <li><p>Always in NVIDIA S.M., I look at the processes list and "python" occupies only around 60MB of vram space.   </p></li>
  <li><p>In KSysGUARD, python's memory usage is always around 700mb.</p></li>
  </ol>
  
  <p>There might be some explanation for that, the problem is that the gpu's memory usage hits 90% with a model with literally 2 neurons (densely connected of course xD), just like a model with 200million parameters does. I'm using a batch size of 128. </p>
  
  <p>I thought around that mess, and if I'm not wrong, a model with 200million parameters should occupy 200000000*4bytes*128 bytes, which should be 1024gb. 
  That means I'm definitely wrong on something, but I'm too selfless to keep that riddle for me, so I decided to give you the chance to solve this ;D</p>
  
  <p>PS: English is not my main language.</p>