Loading partial dataset when debugging
Hi ! `load_dataset` downloads the full dataset once and caches it, so that subsequent calls to `load_dataset` just reloads the dataset from your disk.
Then when you specify a `split` in `load_dataset`, it will just load the requested split from the disk. If your specified split is a sliced split (e.g. `"train[:10]"`), then it will load the 10 first rows of the train split that you have on disk.

Therefore, as long as you don't delete your cache, all your calls to `load_dataset` will be very fast. Except the first call that downloads the dataset of course ^^ That’s a use case for the new streaming feature, no? Hi @reachtarunhere.

Besides the above insights provided by @lhoestq and @thomwolf, there is also a Dataset feature in progress (I plan to finish it this week): #2249, which will allow you, when calling `load_dataset`, to pass the option to download/preprocess/cache only some specific split(s), which will definitely speed up your workflow.

If this feature is interesting for you, I can ping you once it will be merged into the master branch. Thanks all for responding.

Hey @albertvillanova 

Thanks. Yes, I would be interested.

@lhoestq I think even if a small split is specified it loads up the full dataset from the disk (please correct me if this is not the case). Because it does seem to be slow to me even on subsequent calls. There is no repeated downloading so it seems that the cache is working.

I am not aware of the streaming feature @thomwolf mentioned. So I might need to read up on it. @reshinthadithyan  I use the .select function to have a fraction of indices.
I am using PyTorch Lightning along with datasets (thanks for so many datasets already prepared and the great splits). 

Every time I execute load_dataset  for the imdb dataset it takes some time even if I specify a split involving very few samples. I guess this due to hashing as per the other issues.

Is there a way to only load part of the dataset on load_dataset? This would really speed up my workflow.
Something like a debug mode would really help. Thanks!