title: pytorch Dropout: "channel will be zeroed out independently"
body: <p>In the pytorch <a href="https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html" rel="nofollow noreferrer">doc</a> it states: <em>During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution. Each channel will be zeroed out independently on every forward call.</em></p>
  <p>What does it mean by &quot;zero out independently on every forward call&quot;? below is the pseudocode of my implementation, are they equivalent to pytorch version?</p>
  <pre><code>import numpy as np
  p = 0.3
  inpt = np.random.randn((2, 3, 3))  # input tensor
  
  # forward, when training is true
  mask = np.random.choice(a=[False, True], size=inpt.shape, p=[p, 1 - p])
  output = inpt * mask / (1 - p)  # output tensor
  return output
  
  # forward, when training is false
  return inpt # does nothing
  
  # backward propagation
  inpt.grad += incoming_gradient * mask # apply the same mask on incoming gradient
  </code></pre>