Yes, can definitely elaborate:
This is the `datasets` repository, and the issue should be opened in the `transformers` repo instead.
              Yes, can definitely elaborate:

Say I want to use HF Trainer with an arbitrary PyTorch optimizer (`AdamW` here just as an example). Then I should intuitively extend `Trainer` like:

```python
class CustomOptimizerTrainer(Trainer):
    @staticmethod
    def get_optimizer_cls_and_kwargs(args: HfTrainingArguments, model=None) -> tuple[type[torch.optim.Optimizer], dict[str, Any]]:
        optimizer = torch.optim.AdamW
        optimizer_kwargs = {
            "lr": 4e-3,
            "betas": (0.9, 0.999),
            "weight_decay": 0.05,
        }
        return optimizer, optimizer_kwargs
```

However, this won't take effect, because `Trainer.create_optimizer` hardcodes the `Trainer` class name when calling `get_optimizer_cls_and_kwargs`:

https://github.com/huggingface/transformers/blob/6c1d0b069de22d7ed8aa83f733c25045eea0585d/src/transformers/trainer.py#L1076

`CustomOptimizerTrainer.get_optimizer_cls_and_kwargs` will never be called.

So I could either:
- also override the entire `create_optimizer` and rewrite `Trainer.get_optimizer_cls_and_kwargs` to `self.get_optimizer_cls_and_kwargs` (overkill)
- or monkey-patch (not ideal):
```python
class CustomOptimizerTrainer(Trainer):
    # def get_optimizer_cls_and_kwargs ...

    def create_optimizer(self):
        trainer_get_optimizer_fn = Trainer.get_optimizer_cls_and_kwargs
        Trainer.get_optimizer_cls_and_kwargs = self.get_optimizer_cls_and_kwargs
        optimizer = super().create_optimizer()
        Trainer.get_optimizer_cls_and_kwargs = trainer_get_optimizer_fn
        return optimizer
```

But I think the best fix is to change `Trainer.get_optimizer_cls_and_kwargs` to `self.get_optimizer_cls_and_kwargs` in the original source of `Trainer.create_optimizer`.

I also made `get_optimizer_cls_and_kwargs` an instance method instead of a static method, but that probably doesn't matter as much and can be reverted. It breaks the syntax of the tests.

Please let me know if that's clearer and if you agree! Thanks!

_Originally posted by @apoorvkh in https://github.com/huggingface/transformers/issues/31875#issuecomment-2221491647_
            