<p>You can try the following techniques:</p>

<ul>
<li>Make all the images of the same size using rescale. You wrote that <em>"As the pixel resolution are the same across all dataset, the element/features are in the same scale despite the different size. I expect that resizing would severely deteriorate the information."</em> This is right as CNNs are not scale invariant but there exists many techniques to overcome this issue, for example see <a href="https://arxiv.org/abs/1612.03144" rel="nofollow noreferrer">Feature Pyramid Networks for Object Detection</a></li>
<li>Make all the images of the same size using zero-padding. This would be the easiest way as you will not have to change architecture, zero-padding is used to control the shrinkage of the representations inside the CNN, zero-padding the input could lead to negative side effects but it is worth to give it a try. I would recommend to use this technique if all the images have the same background color (for example if you are using X-ray images) and zero-pad with the same color of the background. </li>
</ul>
CNN-VAE, best practices to deal with a dataset with varying size BUT same pixel resolution
<p>I am aware that a lot of solutions have been proposed to train a CNN with variable input size, but the situation I'm facing is different :</p>

<p>My dataset is composed of single-cell images that are all of same pixel resolutions (0.31x0.31 µm) but different sizes (crop from a cell population image after a cell-segmentation process).
Moreover, I'm implementing a VAE for manifold learning purpose.</p>

<p>Therefore :</p>

<ul>
<li>Going for a fully convolutional network or using an AdaptiveAvgPooling (or equivalent for different deep learning framework than pytorch) is not a solution. It's indeed trivial for the encoding part, but then the decoding part of the VAE will have to retrieve the original size of the input which is - as far as I know - not possible.</li>
<li>Resizing all image to a given shape is often proposed, but : As the pixel resolution are the same across all dataset, the element/features are in the same scale despite the different size. I expect that resizing would severely deteriorate the information. The difference in size between cells are meaningful (due to morphological changes) and all other features are still in similar scale.</li>
<li>The only solution that I could find is to zero-pad / crop all the image to a given size.</li>
<li>Multi-scale training won't probably help, as CNN can focus on features at a given scale as all data has same pixel resolution. </li>
</ul>

<p>My questions are :</p>

<ul>
<li>Has anyone ever faced this situation ? Did I miss another approach that would be better ?</li>
<li>If no, does zero-padding could do the job, and could it deteriorate the VAE learning ? The network will need to learn that for some images huge part need to be ignored while not for others images. Some single-cell images would in the end only be represented in a very small portion of the padded-image. The position of the cell in the padded-image might be kept as latent features, but is not relevant.</li>
</ul>

<p>Thanks a lot for your help</p>