<p>I have bothered on this question for a while too, and I have also seen some papers mention this same issue. Here is a recent paper I found; <a href="https://arxiv.org/pdf/1511.07356.pdf" rel="nofollow noreferrer">Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation</a>. I have not fully read the paper but it seems to bother on your question. I can update this answer as soon as I fully grasp the paper.</p>
Max-pooling vs. zero padding: Loosing spatial information
<p>When it comes to convolutional neural networks there are normally many papers recommending different strategies. I have heard people say that it is an absolute must to add padding to the images before a convolution, otherwise to much spatial information is lost. On the other hand they are happy to use pooling, normally max-pooling, to reduce the size of the images. I guess the thought here is that max pooling reduces the spatial information but also reduces the sensitivity to relative positions, so it is a trade-off?</p>

<p>I have heard other people saying that zero-padding does not keep more information, just more empty data. This is because by adding zeros you will not get a reaction from your kernel anyway when part of the information is missing.</p>

<p>I can imagine that zero-padding works if you have big kernels with "scrap values" in the edges and the source of activation centered in a smaller region of the kernel?</p>

<p>I would be happy to read some papers about the effect of down-sampling using pooling contra not using padding, but I cant find much about it. Any good recommendations or thoughts?
<a href="https://i.stack.imgur.com/mGHdI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mGHdI.png" alt="Spatial down-sampling using convolution contra pooling"></a></p>

<p>Figure: Spatial down-sampling using convolution contra pooling (Researchgate)</p>