<p>It would be great if you add details what are you trying to extract from the image and details of the dataset that you are trying to use.</p>

<p>A general assumption can be drawn from <a href="https://adeshpande3.github.io/adeshpande3.github.io/The-9-Deep-Learning-Papers-You-Need-To-Know-About.html" rel="nofollow noreferrer">Alexnet and ZFnet</a> about the filter mask sizes that are needed to be considered. There is no specific formulation which size should be considered for particular format but the size is kept low if a deeper analysis is required as many smaller details might miss with larger filter sizes. In the above link with <a href="https://arxiv.org/pdf/1409.4842v1.pdf" rel="nofollow noreferrer">Inception networks</a> describes how effectively you can utilize the computing resources. If you dont have the issue of the resources, then from ZFNet you can observe the visualizations in multiple layers, there are many finer details visible. We can call it CNN even if it has one layer of convolution and pooling layer. The number of layers depends on the deep finer requirements. </p>

<p>I am not expert, but can recommend if your dataset is small as few thousands and not many features extraction is required, and if you are not sure about the size you can just simply go with the small sizes (small best and popular is 5x5 - Lenet5).</p>
How do you decide on the dimensions of the convolutional neural filter?
<p>If I have an image which is WxHx3 (RGB), how do I decide how big to make the filter masks?  Is it a function of the dimensions (W and H) or something else?  How does the dimensions of the second, third, ... filters compare to the dimensions of the first filter?  (Any concrete pointers would be appreciated.)</p>

<p>I have seen the following, but they don't answer the question.</p>

<p><a href="https://stackoverflow.com/questions/42712219/dimensions-in-convolutional-neural-network">Dimensions in convolutional neural network</a>  </p>

<p><a href="https://stackoverflow.com/questions/36646618/convolutional-neural-networks-how-many-pixels-will-be-covered-by-each-of-the-fi">Convolutional Neural Networks: How many pixels will be covered by each of the filters?</a>  </p>

<p><a href="https://stackoverflow.com/questions/24509921/how-do-you-decide-the-parameters-of-a-convolutional-neural-network-for-image-cla">How do you decide the parameters of a Convolutional Neural Network for image classification?</a></p>