original_docstring: Clip gradients (if needed) and step the base optimizer.
          Always return successful since there is no overflow.
docstring: Clip gradients (if needed) and step the base optimizer.
  Always return successful since there is no overflow.
docstring_tokens: Clip gradients ( if needed ) and step the base optimizer . Always return successful since there is no overflow .
code: def step(self):
          """Clip gradients (if needed) and step the base optimizer.
          Always return successful since there is no overflow."""
  
          # Copy main_grads to grads.
          if self.params_have_main_grad:
              for param_group in self.optimizer.param_groups:
                  for param in param_group['params']:
                      param.grad = param.main_grad
  
                      # Safe to de-reference model's main_grad after copying.
                      # (If using contiguous buffers, main_grad's memory should
                      # persist and therefore should not be deallocated.)
                      if not self.use_contiguous_buffers_in_local_ddp:
                          param.main_grad = None
  
          # Clip gradients.
          grad_norm = None
          if self.clip_grad > 0.0:
              grad_norm = self.clip_grad_norm(self.clip_grad)
  
          # count the zeros in the grads
          num_zeros_in_grad = self.count_zeros() if \
                              self.log_num_zeros_in_grad else None
  
          # Update parameters.
          self.optimizer.step()
  
          # No overflow for FP32 optimizer.
          return True, grad_norm, num_zeros_in_grad
code_tokens: def step ( self ) : if self . params_have_main_grad : for param_group in self . optimizer . param_groups : for param in param_group [ 'params' ] : param . grad = param . main_grad if not self . use_contiguous_buffers_in_local_ddp : param . main_grad = None grad_norm = None if self . clip_grad > 0.0 : grad_norm = self . clip_grad_norm ( self . clip_grad ) num_zeros_in_grad = self . count_zeros ( ) if self . log_num_zeros_in_grad else None self . optimizer . step ( ) return True , grad_norm , num_zeros_in_grad
short_docstring: Clip gradients (if needed) and step the base optimizer.
short_docstring_tokens: Clip gradients ( if needed ) and step the base optimizer .
comment: """Clip gradients (if needed) and step the base optimizer.
          Always return successful since there is no overflow."""$SEP$# Copy main_grads to grads.$SEP$# Safe to de-reference model's main_grad after copying.$SEP$# (If using contiguous buffers, main_grad's memory should$SEP$# persist and therefore should not be deallocated.)$SEP$# Clip gradients.$SEP$# count the zeros in the grads$SEP$# Update parameters.$SEP$# No overflow for FP32 optimizer.