<p>In some cases, resizing the images appropriately (for example to keep the aspectratio) will be sufficient. But, this can introduce distortion, and in case this is harmful, another solution is to use Spatial Pyramidal Pooling (SPP). The problem with different image sizes is that it produces layers of different sizes, for example, taking the features of the <code>n-th</code> layer of some network, you can end up with a featuremap of size <code>128*fw*fh</code> where <code>fw</code>  and <code>fh</code> vary depending on the size of the input example. What SPP does in order to alleviate this problem, is to turn this variable size feature map into a fix-length vector of features. It operates on different scales, by dividing the image into equal patches and performing maxpooling on them. I think <a href="https://arxiv.org/pdf/1406.4729.pdf" rel="nofollow noreferrer">this paper</a> does a great job at explaining it. An example application can be seen <a href="https://arxiv.org/pdf/1702.01381.pdf" rel="nofollow noreferrer">here</a>.</p>

<p>As a quick explanation, imagine you have a feature map of size <code>k*fw*fh</code>. You can consider it as <code>k</code> maps of the form </p>

<pre><code>X Y
Z T
</code></pre>

<p>where each of the blocks are of size <code>fw/2*fh/2</code>. Now, performing maxpooling on each of those blocks separately gives you a vector of size <code>4</code>, and therefore, you can grossly describe the <code>k*fw*fh</code> map as a <code>k*4</code> fixed-size vector of features. </p>

<p>Now, call this fixed-size vector <code>w</code> and set it aside, and this time, consider the <code>k*fw*fh</code> featuremap as <code>k</code> featureplanes written as</p>

<pre><code> A B C D
 E F G H
 I J K L
 M N O P
</code></pre>

<p>and again, perform maxpooling separately on each block. So, using this, you obtain a more fine-grained representation, as a vector of length <code>v=k*16</code>.</p>

<p>Now, concatenating the two vectors <code>u=[v;w]</code> gives you a fixed-size representation. This is exaclty what a 2-scale SPP does  (well, of course you can change the number/sizes of divisions). </p>

<p>Hope this helps.</p>
How to enable a Convolutional NN to take variable size input?
<p>So, I've seen that many of the first CNN examples in Machine Learning use the MNIST dataset. Each image there is 28x28, and so we know the shape of the input before hand. How would this be done for variable size input, let's say you have some images that are 56x56 and some 28x28.</p>

<p>I'm looking for a language and framework agnostic answer if possible or in tensorflow terms preferable</p>