title: Does tensorflow convnet only duplicate model across multiple GPUs?
body: <p>I am currently running a Tensorflow convnet for image recognition and I am considering of buying new GPUs to enable more complex graphs, batch size, and input dimensions. I have read posts like <a href="http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/" rel="nofollow noreferrer">this</a> that do not recommend using AWS GPU instances to train convnets, but more opinions are always welcomed.</p>
  <p>I've read Tensorflow's guide <a href="https://www.tensorflow.org/tutorials/deep_cnn#training_a_model_using_multiple_gpu_cards" rel="nofollow noreferrer">'Training a Model Using Multiple GPU Cards'</a>, and it seems that the graph is duplicated across the GPUs. I would like to know is this the only way to use parallel GPUs in Tensorflow convnet?</p>
  <p>The reason I am asking this is because if Tensorflow can only duplicate graphs across multiple GPUs, it would mean each GPU must have at least the memory size that my model requires for one batch. (Example if the minimum memory size required is 5GB, two card of 4GB each would not do the job)</p>
  <p>Thank you in advance!</p>