Autoeval config
_The documentation is not available anymore as the PR was closed or merged._ Related to: https://github.com/huggingface/autonlp-backend/issues/414 and https://github.com/huggingface/autonlp-backend/issues/424 The tests are failing due to the changed metadata:

```
got an unexpected keyword argument 'train-eval-index'
```

I think you can fix this by updating the `DatasetMetadata` class and implementing an appropriate `validate_train_eval_index()` function

@lhoestq we are working with an arbitrary set of tags for `autoeval config`. See https://github.com/huggingface/autonlp-backend/issues/414
I need to add a validator function though for the tests to pass. Our set is not well-defined as in the rest https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources. What's a workaround for this? On the question of validating the `train-eval-index` metadata, I think the simplest approach would be to validate that the required fields exist and not worry about their values (which are open-ended).

For me, the required fields include:

* `config`
* `task`
* `task_id`
* `splits` (train / validation / eval)
* `col_mapping`
* `metrics` (checking that each one has `type`, `name`) 

Here I'm using the spec defined in https://github.com/huggingface/autonlp-backend/issues/414 as a guide.

WDYT @lhoestq ? Makes sense ! Currently the metadata type validator doesn't support subfields - let me open a PR to add it I ended up improving the metadata validation in this PR x)

In particular:
- I added support YAML keys with dashes instead of underscores for `train-eval-index`
- I added `train-eval-index` validation  with `validate_train_eval_index`. It does nothing fancy, it just checks that it is a list if it exists in the YAML, but feel free to improve it if you want

Let me know if it sounds good to you ! I think we can improve `validate_train_eval_index` in another PR Come on windows... I didn't do anything advanced...

Anyway, will try to fix this when I get back home x) > Come on windows... I didn't do anything advanced...
> 
> Anyway, will try to fix this when I get back home x)

Hehe, thanks! Thanks, @lhoestq this is great!  Did I just fix it for windows and now it fails on linux ? xD > Did I just fix it for windows and now it fails on linux ? xD

Looks like the Heisenberg uncertainty principle is at play here - you cannot simultaneously have unit tests passing in both Linux and Windows 😅  The worst is that the tests pass locally both on my windows and my linux x) Ok fixed it, the issue came from python 3.6 that doesn't return the right `__origin__` for Dict and List types > Alright thanks for adding the first Autoeval config ! :D

Woohoo! Thank you so much 🤗  This is cool!
Added autoeval config to imdb as pilot