<p>The square filter is often selected just because there's no preference in which direction a pattern can be found. For example, it can be a horizontal or a vertical line, both can be important features in an image and the network should capture any of those, if they are important. In other words, you might want your network to be <em>symmetric</em>.</p>

<p>Asymmetric filters became much more popular in the last years, after they were successfully used in <a href="https://arxiv.org/abs/1409.4842" rel="nofollow noreferrer">Inception network</a>. The idea is that <code>n x n</code> filter has the same receptive field as a sequence of <code>1 x n</code> and <code>n x 1</code> convolutions (called <em>effective receptive field</em>, see <a href="http://cs231n.github.io/convolutional-networks/#layerpat" rel="nofollow noreferrer">CS231n tutorial</a> for details), but the latter requires less floating point operations and stores fewer parameters. The architecture is still symmetric in both directions (vertical patterns can be discovered as easily as horizontal ones), but the trick makes it more efficient.</p>

<p>Here's a picture of inception module from <a href="https://arxiv.org/abs/1512.00567" rel="nofollow noreferrer">Inception v2</a>:</p>

<p><a href="https://i.stack.imgur.com/1fnBw.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/1fnBw.jpg" alt="inception-module"></a></p>

<p>In smaller applications, where this kind of optimization isn't critical, there's no big reason to go with such complex architectures and simply use <code>n x n</code> filters.</p>
Why Convolutional NN kernel size is often selected as a square matrix
<p>Given a filter of shape <code>(f1, f2, depth)</code>, what's the implications if <code>f1 != f2</code>?</p>