    def tie_weights(self):
        """
        Should be called after loading state dict to make sure embedding weigths are tied.
        """
        self.decoder.linear.weight = self.embedding.weight
Should be called after loading state dict to make sure embedding weigths are tied.
tests/nn/parallel/fairscale_fsdp_accelerator_test.py