<p>This is simply a choice of a hyper-parameter. Such choices can be made by cross validation of hyper-parameter search, meaning training a few models with different choices of a hyper-parameter, and seeing who got the best performance on the validation set.
Specifically for 3x3 convolution, this has been made popular since the <a href="https://arxiv.org/pdf/1409.1556.pdf" rel="nofollow noreferrer">VGG paper</a> which suggested that stacking many 3x3 convolutions (which is considered a small kernel) can give good performance.</p>
Why faster-rcnn ssd use 3x3 filter to predict box position and class label?
<p>I am reading the faster-rcnn and ssd code for object detection. The prediction layer use the 3x3 filter to predict box position and class label. </p>

<p>Why not use 2x2 filter or 4x4 filter or 5x5 filter to predict them?</p>

<p><a href="https://i.stack.imgur.com/OrXb0.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/OrXb0.png" alt="enter image description here"></a></p>