<p>You didn't say what architecture you're talking about. Since you said you want to classify images, I'm assuming it's a partly convolutional, partly fully connected network like AlexNet, GoogLeNet, etc. In general, the answer to your question depends on the network type you are working with. </p>

<p>If, for example, your network only contains convolutional units - that is to say, does not contain fully connected layers - it <em>can</em> be invariant to the input image's size. Such a network <em>could</em> process the input images and in turn return another image ("convolutional all the way"); you would have to make sure that the output matches what you expect, since you have to determine the loss in some way, of course.</p>

<p>If you are using fully connected units though, you're up for trouble: Here you have a fixed number of learned weights your network has to work with, so varying inputs would require a varying number of weights - and that's not possible.</p>

<p>If that is your problem, here's some things you can do:</p>

<ul>
<li>Don't care about squashing the images. A network might learn to make sense of the content anyway; does scale and perspective mean anything to the content anyway?</li>
<li>Center-crop the images to a specific size. If you fear you're losing data, do multiple crops and use these to augment your input data, so that the original image will be split into <code>N</code> different images of correct size.</li>
<li>Pad the images with a solid color to a squared size, then resize.</li>
<li>Do a combination of that.</li>
</ul>

<p>The padding option might introduce an additional error source to the network's prediction, as the network might (read: likely will) be biased to images that contain such a padded border.
If you need some ideas, have a look at the <a href="https://www.tensorflow.org/api_docs/python/tf/image" rel="noreferrer">Images</a> section of the TensorFlow documentation, there's pieces like <code>resize_image_with_crop_or_pad</code> that take away the bigger work.</p>

<p>As for just don't caring about squashing, <a href="https://github.com/tensorflow/models/blob/f98c5ded31d7da0c2d127c28b2c16f0307a368f0/slim/preprocessing/inception_preprocessing.py#L206-L216" rel="noreferrer">here's</a> a piece of the preprocessing pipeline of the famous Inception network:</p>

<pre class="lang-py prettyprint-override"><code># This resizing operation may distort the images because the aspect
# ratio is not respected. We select a resize method in a round robin
# fashion based on the thread number.
# Note that ResizeMethod contains 4 enumerated resizing methods.

# We select only 1 case for fast_mode bilinear.
num_resize_cases = 1 if fast_mode else 4
distorted_image = apply_with_random_selector(
    distorted_image,
    lambda x, method: tf.image.resize_images(x, [height, width], method=method),
    num_cases=num_resize_cases)
</code></pre>

<p>They're totally aware of it and do it anyway.</p>

<p>Depending on how far you want or need to go, there actually is a paper <a href="https://arxiv.org/abs/1406.4729" rel="noreferrer">here</a> called <em>Spatial Pyramid Pooling in Deep Convolution Networks for Visual Recognition</em> that handles inputs of arbitrary sizes by processing them in a very special way.</p>
how to format the image data for training/prediction when images are different in size?
<p>I am trying to train my model which classifies images. 
The problem I have is, they have different sizes. how should i format my images/or model architecture ?</p>