<p>I found the solution.</p>
<p>Element-wise multiplication between input and the mask before feeding it to a Conv2d method would be enough.(masking input is much easier than masking kernel itself !!):</p>
<pre><code>mask = torch.tensor([[[1, 1, 1, 0]], [[1, 0, 1, 1]], [[1, 1, 0, 1]], [[0, 1, 1, 1]]], dtype=torch.float, requires_grad=False).reshape(1, 1, 4, 4)

&gt;&gt;layer(torch.mul(x, mask))
tensor([[[[5., 15.],
          [30., 40.]]]], grad_fn=&lt;MkldnnConvolutionBackward&gt;)
</code></pre>
<p><strong>P.S</strong> Thanks to @Shai I got the idea from partial convolution represented in this <a href="https://arxiv.org/abs/1804.07723" rel="nofollow noreferrer">paper</a>. However it does some extra manipulation on output. it defines a mask ratio and I guess does some weighting the final output based on it.</p>
masked kernel in convolution in pytorch
<p>Let's say:</p>
<pre><code>x = torch.arange(16, dtype=torch.float).reshape(1, 1, 4, 4)
</code></pre>
<p>and a 2d convolution layer is:</p>
<pre><code>layer = torch.nn.Conv2d(in_channels=1, out_channels=1, kernel_size=2, stride=2)
layer.weight.data[:] = 1.
layer.bias.data[:] = 0.
</code></pre>
<p>Normally, passing <em>x</em> to <em>layer</em> will give:</p>
<pre><code>&gt;&gt;layer(x)    
tensor([[[[10., 18.],
              [42., 50.]]]], grad_fn=&lt;MkldnnConvolutionBackward&gt;)
</code></pre>
<p>Considering having 4 mask filters, how it is done to mask kernel in each step?
for example the following picture indicates 4 filters(white: True, black: False)
<a href="https://i.stack.imgur.com/KsWw2.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KsWw2.png" alt="stride=2 and kernel_size=2 for given x" /></a></p>
<p>The output should be:</p>
<pre><code>tensor([[[[5., 15.],
          [30., 40.]]]], grad_fn=&lt;MkldnnConvolutionBackward&gt;)
</code></pre>
<p><strong>P.S:</strong> all masks are obtained by missing pixels in 2d input array. So 4 masks above are actually a matrix with the same shape as input.</p>