title: Chainerrl : Divide by zero encountered in xp.log(batch_probs) + xp.random.gumbel(size=batch_probs.shape) when I have 4 actions
body: <p>I am using Chainerrl to run an A3C agent on a discrete action space. I have 4 actions that act on an observation space of shape (1,2500). However, I get this runtime warning and I think because of this the agent only takes action 1 and never tries any other action.
  The policy is defined as, where ndim_obs=2500 and n_actions=4:</p>
  <pre><code>class A3CFFSoftmax(chainer.ChainList, a3c.A3CModel):
          &quot;&quot;&quot;An example of A3C feedforward softmax policy.&quot;&quot;&quot;
  
          def __init__(self, ndim_obs, n_actions, hidden_sizes=(200, 200)):
              self.pi = policies.SoftmaxPolicy(
                  model=links.MLP(ndim_obs, n_actions, hidden_sizes))
              self.v = links.MLP(ndim_obs, 1, hidden_sizes=hidden_sizes)
              super().__init__(self.pi, self.v)
  
          def pi_and_v(self, state):
              return self.pi(state), self.v(state)
  </code></pre>