title: How to utilize resources fully
body: <p>I am training a CNN to classify CIFAR-10 dataset using Tensorflow. I am running <a href="https://github.com/Hvass-Labs/TensorFlow-Tutorials" rel="nofollow noreferrer">06_CIFAR-10.ipynb</a> Jupyter Notebook on an AWS p2.xlarge instance (1 GPU, 4 vCPUs, 61GB RAM). I set it up using <a href="https://aws.amazon.com/marketplace/pp/B01EYKBEQ0" rel="nofollow noreferrer">Ubuntu 14 Tensorflow AMI</a>.</p>
  
  <p>Training takes a long time. When I check system resources, I see that most of the resources are still available.</p>
  
  <pre><code>$ free -h
           total       used       free     shared    buffers     cached
  Mem:           59G       3.5G        56G        15M        55M       854M
  -/+ buffers/cache:       2.6G        57G
  Swap:           0B         0B         0B
  
  
  $ top
  top - 18:10:47 up  1:53,  1 user,  load average: 0.47, 0.63, 0.69
  Tasks: 134 total,   1 running, 133 sleeping,   0 stopped,   0 zombie
  %Cpu(s): 19.1 us,  4.6 sy,  0.0 ni, 73.2 id,  0.0 wa,  0.0 hi,  0.3 si,  2.8 st
  KiB Mem:  62881764 total,  3695184 used, 59186580 free,    56792 buffers
  KiB Swap:        0 total,        0 used,        0 free.   875028 cached Mem
  
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                       
   2282 ubuntu    20   0  0.099t 2.192g 202828 S 248.2  3.7 141:55.88 python3                                                                                       
  
  
  $ nvidia-smi 
  Sat May  6 18:12:28 2017       
  +------------------------------------------------------+                       
  | NVIDIA-SMI 352.99     Driver Version: 352.99         |                       
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |===============================+======================+======================|
  |   0  Tesla K80           On   | 0000:00:1E.0     Off |                    0 |
  | N/A   54C    P0    67W / 149W |  11012MiB / 11519MiB |     54%      Default |
  +-------------------------------+----------------------+----------------------+
  
  +-----------------------------------------------------------------------------+
  | Processes:                                                       GPU Memory |
  |  GPU       PID  Type  Process name                               Usage      |
  |=============================================================================|
  |    0      2282    C   /usr/bin/python3                             10954MiB |
  +-----------------------------------------------------------------------------+
  </code></pre>
  
  <p>How can I detect where the bottleneck is? And, any recommendations to utilize all system resources?</p>