Patching fails for modules that are not installed or don't exist
Reported in https://github.com/huggingface/huggingface_hub/runs/6894703718?check_suite_focus=true

When trying to patch `scipy.io.loadmat`:

```python
ModuleNotFoundError: No module named 'scipy'
```

Instead it shouldn't raise an error and do nothing

We use patching to extend such functions to support remote URLs and work in streaming mode