<p>Let me start simple; since you have square matrices for both input and filter let me get one dimension. Then you can apply the same for other dimension(s). Imagine your are building fences between trees, if there are N trees, you have to build N-1 fences. Now apply that analogy to convolution layers.</p>
<p>Your output size will be: input size - filter size + 1</p>
<p><em>Because your filter can only have n-1 steps as fences I mentioned.</em></p>
<p>Let's calculate your output with that idea.
128 - 5 + 1 = 124
Same for other dimension too. So now you have a 124 x 124 image.</p>
<p>That is for one filter.</p>
<p>If you apply this 40 times you will have another dimension: 124 x 124 x 40</p>
<p>Here is a great guide if you want to know more about advanced convolution arithmetic: <a href="https://arxiv.org/pdf/1603.07285.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1603.07285.pdf</a></p>
Calculate the output size in convolution layer
<p>How do I calculate the output size in a convolution layer?</p>
<p>For example, I have a 2D convolution layer that takes a 3x128x128 input and has 40 filters of size 5x5.</p>