title: Train multi-column CNN with memory contrains
body: <p>I deployed an Inception v3 model with tf.keras and it's working fine. At the moment, I'm trying to improve my model to let it handle multiple patches/images together, something along <a href="http://infolab.stanford.edu/~wangz/project/imsearch/Aesthetics/ICCV15/lu.pdf" rel="noreferrer">this article</a> by Lu et al.: there are several "columns" (aka the first layers of the Inception model) and they share weights. However, I have memory constrains (I'm using a GTX 980 with 4GB RAM) and I can't implement it in a naive way, because it goes OOM when trying to compute the gradient. It doesn't go OOM with really small batch sizes (2!) and 3 columns, so each batch is "actually" composed by 6 images. This does not resolve the problem: I want to test this architecture with significantly more columns. </p>
  
  <p>Is there a smarter way to let Keras (or directly Tensorflow) calculate a "column-wise" gradient? Let me introduce some notation to explain what I mean by column-wise: call d(x_2,w_2) the last dense layer and c(x_1,w_1) the column, where x_i is the input of the layer and w_i the weights. Then it is possible to calculate (within the memory contrain) the gradient of d(c(x_1,w_1),w_2). Suppose there is a function u that "unites" several k-dimensional tensors into a k-dimensional tensor (like max, average, ...). Then I want to calculate (within the memory contrain) the gradient of d(u(c(x_11,w_1),...,c(x_1n,w_1)),w_2). By column-wise I mean a trick that computes this gradient wrt each input c(x_1i,w_1) of the union function u. </p>