title: pytorch masked_fill: why can't I mask all zeros?
body: <p>I want to mask the all the zeros in the score matrix with <code>-np.inf</code>, but I can only get part of zeros masked, looked like </p>
  
  <p><a href="https://i.stack.imgur.com/sR3Rw.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/sR3Rw.png" alt="enter image description here"></a></p>
  
  <p>you see in the upper right corner there are still zeros that didn't get masked with <code>-np.inf</code></p>
  
  <p>Here's my codes: </p>
  
  <pre><code>q = torch.Tensor([np.random.random(10),np.random.random(10),np.random.random(10), np.random.random(10), np.zeros((10,1)), np.zeros((10,1))])
  k = torch.Tensor([np.random.random(10),np.random.random(10),np.random.random(10), np.random.random(10), np.zeros((10,1)), np.zeros((10,1))])
  scores = torch.matmul(q, k.transpose(0,1)) / math.sqrt(10)
  mask = torch.Tensor([1,1,1,1,0,0])
  mask = mask.unsqueeze(1)
  scores = scores.masked_fill(mask==0, -np.inf)
  </code></pre>
  
  <p>Maybe the mask is wrong?</p>