    def all_reduce_grads(self, module):
        """Manually all-reduce gradients (if required)."""
        if hasattr(module, "all_reduce_grads"):
            module.all_reduce_grads()
Manually all-reduce gradients (if required).
fairseq/optim/fairseq_optimizer.py