title: Tensorflow: Training models with graphs that won't fit into memory of a single GPU
body: <p>The <code>MemoryError</code> is raised when building the graph, and before anything will be transferred to the GPU. The <code>MemoryError</code> is caused by trying to allocate more memory than the machine has available.</p>
  
  <ul>
  <li>What I have: Multiple GPUs</li>
  <li>What I don't have: Not large enough memory to fit the graph on any single GPU</li>
  <li>What I want to do: Train the model in a single GPU in several passes, or even better, use all the GPUs simultaneously to train different parts of my model (if that's even possible).</li>
  </ul>
  
  <p>But how do I train a model which requires a massive graph to be loaded into the memory (which it won't fit into)? Training in parts? Freezing layers? A quick google search does not give any hints at all :/ </p>
  
  <p><strong>What is the right and perhaps the best way to do this?</strong></p>