Loading a faiss index KeyError

Loading a faiss index KeyError
 I've recently been testing out RAG and DPR embeddings, and I've run into an issue that is not apparent in the documentation.

The basic steps are:

1. Create a dataset (dataset1)
2. Create an embeddings column using DPR
3. Add a faiss index to the dataset
4. Save faiss index to a file
5. Create a new dataset (dataset2) with the same text and label information as dataset1
6. Try to load the faiss index from file to dataset2
7. Get `KeyError: "Column embeddings not in the dataset"`

I've made a colab notebook that should show exactly what I did. Please switch to GPU runtime; I didn't check on CPU.

https://colab.research.google.com/drive/1X0S9ZuZ8k0ybcoei4w7so6dS_WrABmIx?usp=sharing

Ubuntu Version
VERSION="18.04.5 LTS (Bionic Beaver)"

datasets==1.4.1
faiss==1.5.3
faiss-gpu==1.7.0
torch==1.8.0+cu101
transformers==4.3.3

NVIDIA-SMI 460.56
Driver Version: 460.32.03
CUDA Version: 11.2
Tesla K80

I was basically following the steps here: https://huggingface.co/docs/datasets/faiss_and_ea.html#adding-a-faiss-index

I included the exact code from the documentation at the end of the notebook to show that they don't work either.

 > If I understand correctly by reading this example you thought that it was re-adding the "embeddings" column.
Yes. I was trying to use the dataset in RAG and it complained that the dataset didn't have the right columns. No problems when loading the dataset with `load_from_disk` and then doing `load_faiss_index`

What I learned was
1. column and index are different
2. loading the index does not create a column
3. the column is not needed to be able to use the index
4. RAG needs both the embeddings column and the index

If I can come up with a way to articulate this in the right spot in the docs, I'll open a PR


KeyError on using map after renaming a column
 Hi,

I'm trying to use `cifar10` dataset. I want to rename the `img` feature to `image` in order to make it consistent with `mnist`, which I'm also planning to use. By doing this, I was trying to avoid modifying `prepare_train_features` function.

Here is what I try:

```python
transform = Compose([ToPILImage(),ToTensor(),Normalize([0.0,0.0,0.0],[1.0,1.0,1.0])])
def prepare_features(examples):
    images = []
    labels = []
    print(examples)
    for example_idx, example in enumerate(examples["image"]):
        if transform is not None:
            images.append(transform(examples["image"][example_idx].permute(2,0,1)))
        else:
            images.append(examples["image"][example_idx].permute(2,0,1))
        labels.append(examples["label"][example_idx])
    output = {"label":labels, "image":images}
    return output

raw_dataset = load_dataset('cifar10')
raw_dataset.set_format('torch',columns=['img','label'])
raw_dataset = raw_dataset.rename_column('img','image')

features = datasets.Features({
            "image": datasets.Array3D(shape=(3,32,32),dtype="float32"),
            "label": datasets.features.ClassLabel(names=[
                            "airplane",
                            "automobile",
                            "bird",
                            "cat",
                            "deer",
                            "dog",
                            "frog",
                            "horse",
                            "ship",
                            "truck",
                        ]),
        })
train_dataset = raw_dataset.map(prepare_features, features = features,batched=True, batch_size=10000)
```
The error:
```python
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-54-bf29672c53ee> in <module>()
     14                         ]),
     15         })
---> 16 train_dataset = raw_dataset.map(prepare_features, features = features,batched=True, batch_size=10000)

2 frames
/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1287         test_inputs = self[:2] if batched else self[0]
   1288         test_indices = [0, 1] if batched else 0
-> 1289         update_data = does_function_return_dict(test_inputs, test_indices)
   1290         logger.info("Testing finished, running the mapping function on the dataset")
   1291

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in does_function_return_dict(inputs, indices)
   1258             fn_args = [inputs] if input_columns is None else [inputs[col] for col in input_columns]
   1259             processed_inputs = (
-> 1260                 function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1261             )
   1262             does_return_dict = isinstance(processed_inputs, Mapping)

<ipython-input-52-b4dccbafb70d> in prepare_features(examples)
      3     labels = []
      4     print(examples)
----> 5     for example_idx, example in enumerate(examples["image"]):
      6         if transform is not None:
      7             images.append(transform(examples["image"][example_idx].permute(2,0,1)))

KeyError: 'image'
```

The print statement inside returns this:
```python
{'label': tensor([6, 9])}
```
Apparently, both `img` and `image` do not exist after renaming.

Note that this code works fine with `img` everywhere.

Notebook: https://colab.research.google.com/drive/1SzESAlz3BnVYrgQeJ838vbMp1OsukiA2?usp=sharing

 Hi,

Actually, the error occurs due to these two lines:
```python
raw_dataset.set_format('torch',columns=['img','label'])
raw_dataset = raw_dataset.rename_column('img','image')
```
`Dataset.rename_column` doesn't update the `_format_columns` attribute, previously defined by `Dataset.set_format`, with a new column name which is why this new column is missing in the output.


KeyError on using map after renaming a column
 Hi,

I'm trying to use `cifar10` dataset. I want to rename the `img` feature to `image` in order to make it consistent with `mnist`, which I'm also planning to use. By doing this, I was trying to avoid modifying `prepare_train_features` function.

Here is what I try:

```python
transform = Compose([ToPILImage(),ToTensor(),Normalize([0.0,0.0,0.0],[1.0,1.0,1.0])])
def prepare_features(examples):
    images = []
    labels = []
    print(examples)
    for example_idx, example in enumerate(examples["image"]):
        if transform is not None:
            images.append(transform(examples["image"][example_idx].permute(2,0,1)))
        else:
            images.append(examples["image"][example_idx].permute(2,0,1))
        labels.append(examples["label"][example_idx])
    output = {"label":labels, "image":images}
    return output

raw_dataset = load_dataset('cifar10')
raw_dataset.set_format('torch',columns=['img','label'])
raw_dataset = raw_dataset.rename_column('img','image')

features = datasets.Features({
            "image": datasets.Array3D(shape=(3,32,32),dtype="float32"),
            "label": datasets.features.ClassLabel(names=[
                            "airplane",
                            "automobile",
                            "bird",
                            "cat",
                            "deer",
                            "dog",
                            "frog",
                            "horse",
                            "ship",
                            "truck",
                        ]),
        })
train_dataset = raw_dataset.map(prepare_features, features = features,batched=True, batch_size=10000)
```
The error:
```python
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-54-bf29672c53ee> in <module>()
     14                         ]),
     15         })
---> 16 train_dataset = raw_dataset.map(prepare_features, features = features,batched=True, batch_size=10000)

2 frames
/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1287         test_inputs = self[:2] if batched else self[0]
   1288         test_indices = [0, 1] if batched else 0
-> 1289         update_data = does_function_return_dict(test_inputs, test_indices)
   1290         logger.info("Testing finished, running the mapping function on the dataset")
   1291

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in does_function_return_dict(inputs, indices)
   1258             fn_args = [inputs] if input_columns is None else [inputs[col] for col in input_columns]
   1259             processed_inputs = (
-> 1260                 function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1261             )
   1262             does_return_dict = isinstance(processed_inputs, Mapping)

<ipython-input-52-b4dccbafb70d> in prepare_features(examples)
      3     labels = []
      4     print(examples)
----> 5     for example_idx, example in enumerate(examples["image"]):
      6         if transform is not None:
      7             images.append(transform(examples["image"][example_idx].permute(2,0,1)))

KeyError: 'image'
```

The print statement inside returns this:
```python
{'label': tensor([6, 9])}
```
Apparently, both `img` and `image` do not exist after renaming.

Note that this code works fine with `img` everywhere.

Notebook: https://colab.research.google.com/drive/1SzESAlz3BnVYrgQeJ838vbMp1OsukiA2?usp=sharing

 Hi @mariosasko,

Thanks for opening a PR on this :)
Why does the old name also disappear?


KeyError on using map after renaming a column
 Hi,

I'm trying to use `cifar10` dataset. I want to rename the `img` feature to `image` in order to make it consistent with `mnist`, which I'm also planning to use. By doing this, I was trying to avoid modifying `prepare_train_features` function.

Here is what I try:

```python
transform = Compose([ToPILImage(),ToTensor(),Normalize([0.0,0.0,0.0],[1.0,1.0,1.0])])
def prepare_features(examples):
    images = []
    labels = []
    print(examples)
    for example_idx, example in enumerate(examples["image"]):
        if transform is not None:
            images.append(transform(examples["image"][example_idx].permute(2,0,1)))
        else:
            images.append(examples["image"][example_idx].permute(2,0,1))
        labels.append(examples["label"][example_idx])
    output = {"label":labels, "image":images}
    return output

raw_dataset = load_dataset('cifar10')
raw_dataset.set_format('torch',columns=['img','label'])
raw_dataset = raw_dataset.rename_column('img','image')

features = datasets.Features({
            "image": datasets.Array3D(shape=(3,32,32),dtype="float32"),
            "label": datasets.features.ClassLabel(names=[
                            "airplane",
                            "automobile",
                            "bird",
                            "cat",
                            "deer",
                            "dog",
                            "frog",
                            "horse",
                            "ship",
                            "truck",
                        ]),
        })
train_dataset = raw_dataset.map(prepare_features, features = features,batched=True, batch_size=10000)
```
The error:
```python
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-54-bf29672c53ee> in <module>()
     14                         ]),
     15         })
---> 16 train_dataset = raw_dataset.map(prepare_features, features = features,batched=True, batch_size=10000)

2 frames
/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1287         test_inputs = self[:2] if batched else self[0]
   1288         test_indices = [0, 1] if batched else 0
-> 1289         update_data = does_function_return_dict(test_inputs, test_indices)
   1290         logger.info("Testing finished, running the mapping function on the dataset")
   1291

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in does_function_return_dict(inputs, indices)
   1258             fn_args = [inputs] if input_columns is None else [inputs[col] for col in input_columns]
   1259             processed_inputs = (
-> 1260                 function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1261             )
   1262             does_return_dict = isinstance(processed_inputs, Mapping)

<ipython-input-52-b4dccbafb70d> in prepare_features(examples)
      3     labels = []
      4     print(examples)
----> 5     for example_idx, example in enumerate(examples["image"]):
      6         if transform is not None:
      7             images.append(transform(examples["image"][example_idx].permute(2,0,1)))

KeyError: 'image'
```

The print statement inside returns this:
```python
{'label': tensor([6, 9])}
```
Apparently, both `img` and `image` do not exist after renaming.

Note that this code works fine with `img` everywhere.

Notebook: https://colab.research.google.com/drive/1SzESAlz3BnVYrgQeJ838vbMp1OsukiA2?usp=sharing

 I just merged a @mariosasko 's PR that fixes this issue.
If it happens again, feel free to re-open :)


ValueError when rename_column on splitted dataset
 Hi there,
I am loading `.tsv` file via `load_dataset` and subsequently split the rows into training and test set via the `ReadInstruction` API like so:

```python
split = {
    'train': ReadInstruction('train', to=90, unit='%'),
    'test': ReadInstruction('train', from_=-10, unit='%')
}

dataset = load_dataset(
    path='csv',               # use 'text' loading script to load from local txt-files
    delimiter='\t',           # xxx
    data_files=text_files,    # list of paths to local text files
    split=split,              # xxx
)

dataset
```

Part of output:
```python
DatasetDict({
    train: Dataset({
        features: ['sentence', 'sentiment'],
        num_rows: 900
    })
    test: Dataset({
        features: ['sentence', 'sentiment'],
        num_rows: 100
    })
})
```
Afterwards I'd like to rename the 'sentence' column to 'text' in order to be compatible with my modelin pipeline. If I run the following code I experience a `ValueError` however:
```python
dataset['train'].rename_column('sentence', 'text')
```
```python
/usr/local/lib/python3.7/dist-packages/datasets/splits.py in __init__(self, name)
    353         for split_name in split_names_from_instruction:
    354             if not re.match(_split_re, split_name):
--> 355                 raise ValueError(f"Split name should match '{_split_re}'' but got '{split_name}'.")
    356
    357     def __str__(self):

ValueError: Split name should match '^\w+(\.\w+)*$'' but got 'ReadInstruction('.
```
In particular, these behavior does not arise if I use the deprecated `rename_column_` method. Any idea what causes the error? Would assume something in the way I defined the split.

Thanks in advance! :)
 Hi,

This is a bug so thanks for reporting it. `Dataset.__setstate__`  is the problem, which is called when `Dataset.rename_column` tries to copy the dataset with `copy.deepcopy(self)`. This only happens if the `split` arg in `load_dataset` was defined as `ReadInstruction`.

To overcome this issue, use the named splits API (for now):
```python
train_ds, test_ds = load_dataset(
    path='csv',
    delimiter='\t',
    data_files=text_files,
    split=['train[:90%]', 'train[-10%:]'],
)

train_ds = train_ds.rename_column('sentence', 'text')
```


ValueError when rename_column on splitted dataset
 Hi there,
I am loading `.tsv` file via `load_dataset` and subsequently split the rows into training and test set via the `ReadInstruction` API like so:

```python
split = {
    'train': ReadInstruction('train', to=90, unit='%'),
    'test': ReadInstruction('train', from_=-10, unit='%')
}

dataset = load_dataset(
    path='csv',               # use 'text' loading script to load from local txt-files
    delimiter='\t',           # xxx
    data_files=text_files,    # list of paths to local text files
    split=split,              # xxx
)

dataset
```

Part of output:
```python
DatasetDict({
    train: Dataset({
        features: ['sentence', 'sentiment'],
        num_rows: 900
    })
    test: Dataset({
        features: ['sentence', 'sentiment'],
        num_rows: 100
    })
})
```
Afterwards I'd like to rename the 'sentence' column to 'text' in order to be compatible with my modelin pipeline. If I run the following code I experience a `ValueError` however:
```python
dataset['train'].rename_column('sentence', 'text')
```
```python
/usr/local/lib/python3.7/dist-packages/datasets/splits.py in __init__(self, name)
    353         for split_name in split_names_from_instruction:
    354             if not re.match(_split_re, split_name):
--> 355                 raise ValueError(f"Split name should match '{_split_re}'' but got '{split_name}'.")
    356
    357     def __str__(self):

ValueError: Split name should match '^\w+(\.\w+)*$'' but got 'ReadInstruction('.
```
In particular, these behavior does not arise if I use the deprecated `rename_column_` method. Any idea what causes the error? Would assume something in the way I defined the split.

Thanks in advance! :)
 This has been fixed in #2043 , thanks @mariosasko
The fix is available on master and we'll do a new release soon :)

feel free to re-open if you still have issues


Interactively doing  save_to_disk and load_from_disk corrupts the datasets object?
  dataset_info.json file saved after using  save_to_disk gets corrupted as follows.

![image](https://user-images.githubusercontent.com/16892570/110568474-ed969880-81b7-11eb-832f-2e5129656016.png)

Is there a way to disable the cache that will save to /tmp/huggiface/datastes ?
I have a feeling there is a serious issue with cashing.
 Hi,

Can you give us a minimal reproducible example? This [part](https://huggingface.co/docs/datasets/master/processing.html#controling-the-cache-behavior) of the docs explains how to control caching.


No upstream branch

No upstream branch
 Feels like the documentation on adding a new dataset is outdated?

https://github.com/huggingface/datasets/blob/987df6b4e9e20fc0c92bc9df48137d170756fd7b/ADD_NEW_DATASET.md#L49-L54

There is no upstream branch on remote.
 What's the issue exactly ?

Given an `upstream` remote repository with url `https://github.com/huggingface/datasets.git`, you can totally rebase from `upstream/master`.

It's mentioned at the beginning how to add the `upstream` remote repository

https://github.com/huggingface/datasets/blob/987df6b4e9e20fc0c92bc9df48137d170756fd7b/ADD_NEW_DATASET.md#L10-L14


No upstream branch

No upstream branch
 Feels like the documentation on adding a new dataset is outdated?

https://github.com/huggingface/datasets/blob/987df6b4e9e20fc0c92bc9df48137d170756fd7b/ADD_NEW_DATASET.md#L49-L54

There is no upstream branch on remote.
 ~~What difference is there with the default `origin` remote that is set when you clone the repo?~~ I've just understood that this applies to **forks** of the repo 🤡


Local testing fails

Local testing fails
 I'm following the CI setup as described in

https://github.com/huggingface/datasets/blob/8eee4fa9e133fe873a7993ba746d32ca2b687551/.circleci/config.yml#L16-L19

in a new conda environment, at commit https://github.com/huggingface/datasets/commit/4de6dbf84e93dad97e1000120d6628c88954e5d4

and getting

```
FAILED tests/test_caching.py::RecurseDumpTest::test_dump_ipython_function - TypeError: an integer is required (got type bytes)
1 failed, 2321 passed, 5109 skipped, 10 warnings in 124.32s (0:02:04)
```

Seems like a discrepancy with CI, perhaps a lib version that's not controlled?
Tried with `pyarrow=={1.0.0,0.17.1,2.0.0}`
 I'm not able to reproduce on my side.
Can you provide the full stacktrace please ?
What version of `python` and `dill` do you have ? Which OS are you using ?


Local testing fails

Local testing fails
 I'm following the CI setup as described in

https://github.com/huggingface/datasets/blob/8eee4fa9e133fe873a7993ba746d32ca2b687551/.circleci/config.yml#L16-L19

in a new conda environment, at commit https://github.com/huggingface/datasets/commit/4de6dbf84e93dad97e1000120d6628c88954e5d4

and getting

```
FAILED tests/test_caching.py::RecurseDumpTest::test_dump_ipython_function - TypeError: an integer is required (got type bytes)
1 failed, 2321 passed, 5109 skipped, 10 warnings in 124.32s (0:02:04)
```

Seems like a discrepancy with CI, perhaps a lib version that's not controlled?
Tried with `pyarrow=={1.0.0,0.17.1,2.0.0}`
 ```
co_filename = '<ipython-input-2-e0383a102aae>', returned_obj = [0]

    def create_ipython_func(co_filename, returned_obj):
        def func():
            return returned_obj

        code = func.__code__
>       code = CodeType(*[getattr(code, k) if k != "co_filename" else co_filename for k in code_args])
E       TypeError: an integer is required (got type bytes)

tests/test_caching.py:152: TypeError
```

Python 3.8.8
dill==0.3.1.1


Local testing fails

Local testing fails
 I'm following the CI setup as described in

https://github.com/huggingface/datasets/blob/8eee4fa9e133fe873a7993ba746d32ca2b687551/.circleci/config.yml#L16-L19

in a new conda environment, at commit https://github.com/huggingface/datasets/commit/4de6dbf84e93dad97e1000120d6628c88954e5d4

and getting

```
FAILED tests/test_caching.py::RecurseDumpTest::test_dump_ipython_function - TypeError: an integer is required (got type bytes)
1 failed, 2321 passed, 5109 skipped, 10 warnings in 124.32s (0:02:04)
```

Seems like a discrepancy with CI, perhaps a lib version that's not controlled?
Tried with `pyarrow=={1.0.0,0.17.1,2.0.0}`
 I managed to reproduce. This comes from the CodeType init signature that is different in python 3.8.8
I opened a PR to fix this test
Thanks !


Ambiguous documentation

Ambiguous documentation
 https://github.com/huggingface/datasets/blob/2ac9a0d24a091989f869af55f9f6411b37ff5188/templates/new_dataset_script.py#L156-L158

Looking at the template, I find this documentation line to be confusing, the method parameters don't include the `gen_kwargs` so I'm unclear where they're coming from.

Happy to push a PR with a clearer statement when I understand the meaning.
 Hi @theo-m !

A few lines above this line, you'll find that the `_split_generators` method returns a list of `SplitGenerator`s objects:

```python
datasets.SplitGenerator(
    name=datasets.Split.VALIDATION,
    # These kwargs will be passed to _generate_examples
    gen_kwargs={
        "filepath": os.path.join(data_dir, "dev.jsonl"),
        "split": "dev",
    },
),
```

Notice the `gen_kwargs` argument passed to the constructor of `SplitGenerator`: this dict will be unpacked as keyword arguments to pass to the `_generat_examples` method (in this case the `filepath` and `split` arguments).

Let me know if that helps!


Ambiguous documentation

Ambiguous documentation
 https://github.com/huggingface/datasets/blob/2ac9a0d24a091989f869af55f9f6411b37ff5188/templates/new_dataset_script.py#L156-L158

Looking at the template, I find this documentation line to be confusing, the method parameters don't include the `gen_kwargs` so I'm unclear where they're coming from.

Happy to push a PR with a clearer statement when I understand the meaning.
 Oh ok I hadn't made the connection between those two, will offer a tweak to the comment and the template then - thanks!


How to not load huggingface datasets into memory
 Hi
I am running this example from transformers library version 4.3.3:
(Here is the full documentation https://github.com/huggingface/transformers/issues/8771 but the running command should work out of the box)

 USE_TF=0  deepspeed  run_seq2seq.py --model_name_or_path google/mt5-base --dataset_name wmt16 --dataset_config_name ro-en --source_prefix "translate English to Romanian: " --task translation_en_to_ro   --output_dir /test/test_large  --do_train --do_eval --predict_with_generate  --max_train_samples 500   --max_val_samples 500  --max_source_length 128 --max_target_length 128 --sortish_sampler --per_device_train_batch_size 8   --val_max_target_length 128 --deepspeed ds_config.json --num_train_epochs 1 --eval_steps 25000 --warmup_steps 500 --overwrite_output_dir

(Here please find the script: https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_seq2seq.py)

If you do not pass max_train_samples in above command to load the full dataset, then I get memory issue on a gpu with 24 GigBytes of memory.

I need to train large-scale mt5 model on large-scale datasets of wikipedia (multiple of them concatenated or other datasets in multiple languages like OPUS), could you help me how I can avoid loading the full data into memory? to make the scripts not related to data size?

In above example, I was hoping the script could work without relying on dataset size, so I can still train the model without subsampling training set.

thank you so much @lhoestq for your great help in advance

 So maybe a summary here:
If I could fit a large model with batch_size = X into memory, is there a way I could train this model for huge datasets with keeping setting the same? thanks


How to not load huggingface datasets into memory
 Hi
I am running this example from transformers library version 4.3.3:
(Here is the full documentation https://github.com/huggingface/transformers/issues/8771 but the running command should work out of the box)

 USE_TF=0  deepspeed  run_seq2seq.py --model_name_or_path google/mt5-base --dataset_name wmt16 --dataset_config_name ro-en --source_prefix "translate English to Romanian: " --task translation_en_to_ro   --output_dir /test/test_large  --do_train --do_eval --predict_with_generate  --max_train_samples 500   --max_val_samples 500  --max_source_length 128 --max_target_length 128 --sortish_sampler --per_device_train_batch_size 8   --val_max_target_length 128 --deepspeed ds_config.json --num_train_epochs 1 --eval_steps 25000 --warmup_steps 500 --overwrite_output_dir

(Here please find the script: https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_seq2seq.py)

If you do not pass max_train_samples in above command to load the full dataset, then I get memory issue on a gpu with 24 GigBytes of memory.

I need to train large-scale mt5 model on large-scale datasets of wikipedia (multiple of them concatenated or other datasets in multiple languages like OPUS), could you help me how I can avoid loading the full data into memory? to make the scripts not related to data size?

In above example, I was hoping the script could work without relying on dataset size, so I can still train the model without subsampling training set.

thank you so much @lhoestq for your great help in advance

 The `datastets` library doesn't load datasets into memory. Therefore you can load a dataset that is terabytes big without filling up your RAM.

The only thing that's loaded into memory during training is the batch used in the training step.
So as long as your model works with batch_size = X, then you can load an even bigger dataset and it will work as well with the same batch_size.

Note that you still have to take into account that some batches take more memory than others, depending on the texts lengths. If it works for a batch with batch_size = X and with texts of maximum length, then it will work for all batches.

In your case I guess that there are a few long sentences in the dataset. For those long sentences you get a memory error on your GPU because they're too long. By passing `max_train_samples` you may have taken a subset of the dataset that only contain short sentences. That's probably why in your case it worked only when you set `max_train_samples`.
I'd suggest you to reduce the batch size so that the batches with long sentences can be loaded on the GPU.

Let me know if that helps or if you have other questions


Setting to torch format not working with torchvision and MNIST
 Hi

I am trying to use `torchvision.transforms` to handle the transformation of the image data in the `mnist` dataset. Assume I have a `transform` variable which contains the `torchvision.transforms` object.

A snippet of what I am trying to do:
```python
def prepare_features(examples):
    images = []
    labels = []
    for example_idx, example in enumerate(examples["image"]):
        if transform is not None:
            images.append(transform(
                np.array(examples["image"][example_idx], dtype=np.uint8)
            ))
        else:
            images.append(torch.tensor(np.array(examples["image"][example_idx], dtype=np.uint8)))
        labels.append(torch.tensor(examples["label"][example_idx]))
    output = {"label":labels, "image":images}
    return output

raw_dataset = load_dataset('mnist')
train_dataset = raw_dataset.map(prepare_features, batched=True, batch_size=10000)
train_dataset.set_format("torch",columns=["image","label"])
```

After this, I check the type of the following:
```python
print(type(train_dataset["train"]["label"]))
print(type(train_dataset["train"]["image"][0]))
```
This leads to the following output:

```python
<class 'torch.Tensor'>
<class 'list'>
```
I use `torch.utils.DataLoader` for batches, the type of `batch["train"]["image"]` is also `<class 'list'>`.

I don't understand why only the `label` is converted to a torch tensor, why does the image not get converted? How can I fix this issue?

Thanks,
Gunjan

EDIT:
I just checked the shapes, and the types, `batch[image]` is a actually a list of list of tensors. Shape is (1,28,2,28), where `batch_size` is 2. I don't understand why this is happening. Ideally it should be a tensor of shape (2,1,28,28).

EDIT 2:
Inside `prepare_train_features`, the shape of `images[0]` is `torch.Size([1,28,28])`, the conversion is working. However, the output of the `map` is a list of list of list of list.
 Adding to the previous information, I think `torch.utils.data.DataLoader` is doing some conversion.
What I tried:
```python
train_dataset = load_dataset('mnist')
```
I don't use any `map` or `set_format` or any `transform`. I use this directly, and try to load batches using the `DataLoader` with batch size 2, I get an output like this for the `image`:

```
[[tensor([0, 0]), tensor([0, 0]), tensor([0, 0]), tensor([0, 0]), tensor([0, 0]), tensor([0, 0]), tensor([0, 0]), tensor([0, 0]), tensor([0, 0]), tensor...
```
For `label`, it works fine:
```
tensor([7, 6])
```
Note that I didn't specify conversion to torch tensors anywhere.

Basically, there are two problems here:
1. `dataset.map` doesn't return tensor type objects, even though it uses the transforms, the grayscale conversion in transform was done, but the output was lists only.
2.  The `DataLoader` performs its own conversion, which may be not desired.

I understand that we can't change `DataLoader` because it is a torch functionality, however, is there a way we can handle image data to allow using it with torch `DataLoader` and `torchvision` properly?

I think if the `image` was a torch tensor (N,H,W,C), or  a list of torch tensors (H,W,C), before it is passed to `DataLoader`, then we might not face this issue.


Setting to torch format not working with torchvision and MNIST
 Hi

I am trying to use `torchvision.transforms` to handle the transformation of the image data in the `mnist` dataset. Assume I have a `transform` variable which contains the `torchvision.transforms` object.

A snippet of what I am trying to do:
```python
def prepare_features(examples):
    images = []
    labels = []
    for example_idx, example in enumerate(examples["image"]):
        if transform is not None:
            images.append(transform(
                np.array(examples["image"][example_idx], dtype=np.uint8)
            ))
        else:
            images.append(torch.tensor(np.array(examples["image"][example_idx], dtype=np.uint8)))
        labels.append(torch.tensor(examples["label"][example_idx]))
    output = {"label":labels, "image":images}
    return output

raw_dataset = load_dataset('mnist')
train_dataset = raw_dataset.map(prepare_features, batched=True, batch_size=10000)
train_dataset.set_format("torch",columns=["image","label"])
```

After this, I check the type of the following:
```python
print(type(train_dataset["train"]["label"]))
print(type(train_dataset["train"]["image"][0]))
```
This leads to the following output:

```python
<class 'torch.Tensor'>
<class 'list'>
```
I use `torch.utils.DataLoader` for batches, the type of `batch["train"]["image"]` is also `<class 'list'>`.

I don't understand why only the `label` is converted to a torch tensor, why does the image not get converted? How can I fix this issue?

Thanks,
Gunjan

EDIT:
I just checked the shapes, and the types, `batch[image]` is a actually a list of list of tensors. Shape is (1,28,2,28), where `batch_size` is 2. I don't understand why this is happening. Ideally it should be a tensor of shape (2,1,28,28).

EDIT 2:
Inside `prepare_train_features`, the shape of `images[0]` is `torch.Size([1,28,28])`, the conversion is working. However, the output of the `map` is a list of list of list of list.
 What's the feature types of your new dataset after `.map` ?

Can you try with adding `features=` in the `.map` call in order to set the "image" feature type to `Array2D` ?
The default feature type is lists of lists, we've not implemented shape verification to use ArrayXD instead of nested lists yet


Setting to torch format not working with torchvision and MNIST
 Hi

I am trying to use `torchvision.transforms` to handle the transformation of the image data in the `mnist` dataset. Assume I have a `transform` variable which contains the `torchvision.transforms` object.

A snippet of what I am trying to do:
```python
def prepare_features(examples):
    images = []
    labels = []
    for example_idx, example in enumerate(examples["image"]):
        if transform is not None:
            images.append(transform(
                np.array(examples["image"][example_idx], dtype=np.uint8)
            ))
        else:
            images.append(torch.tensor(np.array(examples["image"][example_idx], dtype=np.uint8)))
        labels.append(torch.tensor(examples["label"][example_idx]))
    output = {"label":labels, "image":images}
    return output

raw_dataset = load_dataset('mnist')
train_dataset = raw_dataset.map(prepare_features, batched=True, batch_size=10000)
train_dataset.set_format("torch",columns=["image","label"])
```

After this, I check the type of the following:
```python
print(type(train_dataset["train"]["label"]))
print(type(train_dataset["train"]["image"][0]))
```
This leads to the following output:

```python
<class 'torch.Tensor'>
<class 'list'>
```
I use `torch.utils.DataLoader` for batches, the type of `batch["train"]["image"]` is also `<class 'list'>`.

I don't understand why only the `label` is converted to a torch tensor, why does the image not get converted? How can I fix this issue?

Thanks,
Gunjan

EDIT:
I just checked the shapes, and the types, `batch[image]` is a actually a list of list of tensors. Shape is (1,28,2,28), where `batch_size` is 2. I don't understand why this is happening. Ideally it should be a tensor of shape (2,1,28,28).

EDIT 2:
Inside `prepare_train_features`, the shape of `images[0]` is `torch.Size([1,28,28])`, the conversion is working. However, the output of the `map` is a list of list of list of list.
 Hi @lhoestq

Raw feature types are like this:
```
Image:
<class 'list'> 60000  #(type, len)
<class 'list'> 28
<class 'list'> 28
<class 'int'>
Label:
<class 'list'> 60000
<class 'int'>
```
Inside the `prepare_feature` method with batch size 100000 , after processing, they are like this:

Inside Prepare Train Features
```
Image:
<class 'list'> 10000
<class 'torch.Tensor'> 1
<class 'torch.Tensor'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'list'> 10000
<class 'torch.Tensor'>
```

After map, the feature type are like this:
```
Image:
<class 'list'> 60000
<class 'list'> 1
<class 'list'> 28
<class 'list'> 28
<class 'float'>
Label:
<class 'list'> 60000
<class 'int'>
```

After dataloader with batch size 2, the batch features are like this:
```
Image:
<class 'list'> 1
<class 'list'> 28
<class 'list'> 28
<class 'torch.Tensor'> 2
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 2
<class 'torch.Tensor'>
```
<hr>

When I was setting the format of `train_dataset` to 'torch' after mapping  -
```
Image:
<class 'list'> 60000
<class 'list'> 1
<class 'list'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 60000
<class 'torch.Tensor'>
```

Corresponding DataLoader batch:
```
From DataLoader batch features
Image:
<class 'list'> 1
<class 'list'> 28
<class 'torch.Tensor'> 2
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 2
<class 'torch.Tensor'>
```

I will check with features and get back.


Setting to torch format not working with torchvision and MNIST
 Hi

I am trying to use `torchvision.transforms` to handle the transformation of the image data in the `mnist` dataset. Assume I have a `transform` variable which contains the `torchvision.transforms` object.

A snippet of what I am trying to do:
```python
def prepare_features(examples):
    images = []
    labels = []
    for example_idx, example in enumerate(examples["image"]):
        if transform is not None:
            images.append(transform(
                np.array(examples["image"][example_idx], dtype=np.uint8)
            ))
        else:
            images.append(torch.tensor(np.array(examples["image"][example_idx], dtype=np.uint8)))
        labels.append(torch.tensor(examples["label"][example_idx]))
    output = {"label":labels, "image":images}
    return output

raw_dataset = load_dataset('mnist')
train_dataset = raw_dataset.map(prepare_features, batched=True, batch_size=10000)
train_dataset.set_format("torch",columns=["image","label"])
```

After this, I check the type of the following:
```python
print(type(train_dataset["train"]["label"]))
print(type(train_dataset["train"]["image"][0]))
```
This leads to the following output:

```python
<class 'torch.Tensor'>
<class 'list'>
```
I use `torch.utils.DataLoader` for batches, the type of `batch["train"]["image"]` is also `<class 'list'>`.

I don't understand why only the `label` is converted to a torch tensor, why does the image not get converted? How can I fix this issue?

Thanks,
Gunjan

EDIT:
I just checked the shapes, and the types, `batch[image]` is a actually a list of list of tensors. Shape is (1,28,2,28), where `batch_size` is 2. I don't understand why this is happening. Ideally it should be a tensor of shape (2,1,28,28).

EDIT 2:
Inside `prepare_train_features`, the shape of `images[0]` is `torch.Size([1,28,28])`, the conversion is working. However, the output of the `map` is a list of list of list of list.
 Hi @lhoestq

# Using Array3D
I tried this:
```python
features = datasets.Features({
            "image": datasets.Array3D(shape=(1,28,28),dtype="float32"),
            "label": datasets.features.ClassLabel(names=["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"]),
        })
train_dataset = raw_dataset.map(prepare_features, features = features,batched=True, batch_size=10000)
```
and it didn't fix the issue.

During the `prepare_train_features:
```
Image:
<class 'list'> 10000
<class 'torch.Tensor'> 1
<class 'torch.Tensor'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'list'> 10000
<class 'torch.Tensor'>
```

After the `map`:

```
Image:
<class 'list'> 60000
<class 'list'> 1
<class 'list'> 28
<class 'list'> 28
<class 'float'>
Label:
<class 'list'> 60000
<class 'int'>
```
From the DataLoader batch:
```
Image:
<class 'list'> 1
<class 'list'> 28
<class 'list'> 28
<class 'torch.Tensor'> 2
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 2
<class 'torch.Tensor'>
```
It is the same as before.

---

Using `datasets.Sequence(datasets.Array2D(shape=(28,28),dtype="float32"))` gave an error during `map`:

```python
ArrowNotImplementedError                  Traceback (most recent call last)
<ipython-input-95-d28e69289084> in <module>()
      3             "label": datasets.features.ClassLabel(names=["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"]),
      4         })
----> 5 train_dataset = raw_dataset.map(prepare_features, features = features,batched=True, batch_size=10000)

15 frames
/usr/local/lib/python3.7/dist-packages/datasets/dataset_dict.py in map(self, function, with_indices, input_columns, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_names, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc)
    446                     num_proc=num_proc,
    447                 )
--> 448                 for k, dataset in self.items()
    449             }
    450         )

/usr/local/lib/python3.7/dist-packages/datasets/dataset_dict.py in <dictcomp>(.0)
    446                     num_proc=num_proc,
    447                 )
--> 448                 for k, dataset in self.items()
    449             }
    450         )

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1307                 fn_kwargs=fn_kwargs,
   1308                 new_fingerprint=new_fingerprint,
-> 1309                 update_data=update_data,
   1310             )
   1311         else:

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    202         }
    203         # apply actual function
--> 204         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    205         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    206         # re-apply format to the output

/usr/local/lib/python3.7/dist-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    335             # Call actual function
    336
--> 337             out = func(self, *args, **kwargs)
    338
    339             # Update fingerprint of in-place transforms + update in-place history of transforms

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, update_data)
   1580                     if update_data:
   1581                         batch = cast_to_python_objects(batch)
-> 1582                         writer.write_batch(batch)
   1583             if update_data:
   1584                 writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file

/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in write_batch(self, batch_examples, writer_batch_size)
    274             typed_sequence = TypedSequence(batch_examples[col], type=col_type, try_type=col_try_type)
    275             typed_sequence_examples[col] = typed_sequence
--> 276         pa_table = pa.Table.from_pydict(typed_sequence_examples)
    277         self.write_table(pa_table, writer_batch_size)
    278

/usr/local/lib/python3.7/dist-packages/pyarrow/table.pxi in pyarrow.lib.Table.from_pydict()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib.asarray()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib.array()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib._handle_arrow_array_protocol()

/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in __arrow_array__(self, type)
     95                 out = pa.ExtensionArray.from_storage(type, pa.array(self.data, type.storage_dtype))
     96             else:
---> 97                 out = pa.array(self.data, type=type)
     98             if trying_type and out[0].as_py() != self.data[0]:
     99                 raise TypeError(

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib.array()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: extension
```


Setting to torch format not working with torchvision and MNIST
 Hi

I am trying to use `torchvision.transforms` to handle the transformation of the image data in the `mnist` dataset. Assume I have a `transform` variable which contains the `torchvision.transforms` object.

A snippet of what I am trying to do:
```python
def prepare_features(examples):
    images = []
    labels = []
    for example_idx, example in enumerate(examples["image"]):
        if transform is not None:
            images.append(transform(
                np.array(examples["image"][example_idx], dtype=np.uint8)
            ))
        else:
            images.append(torch.tensor(np.array(examples["image"][example_idx], dtype=np.uint8)))
        labels.append(torch.tensor(examples["label"][example_idx]))
    output = {"label":labels, "image":images}
    return output

raw_dataset = load_dataset('mnist')
train_dataset = raw_dataset.map(prepare_features, batched=True, batch_size=10000)
train_dataset.set_format("torch",columns=["image","label"])
```

After this, I check the type of the following:
```python
print(type(train_dataset["train"]["label"]))
print(type(train_dataset["train"]["image"][0]))
```
This leads to the following output:

```python
<class 'torch.Tensor'>
<class 'list'>
```
I use `torch.utils.DataLoader` for batches, the type of `batch["train"]["image"]` is also `<class 'list'>`.

I don't understand why only the `label` is converted to a torch tensor, why does the image not get converted? How can I fix this issue?

Thanks,
Gunjan

EDIT:
I just checked the shapes, and the types, `batch[image]` is a actually a list of list of tensors. Shape is (1,28,2,28), where `batch_size` is 2. I don't understand why this is happening. Ideally it should be a tensor of shape (2,1,28,28).

EDIT 2:
Inside `prepare_train_features`, the shape of `images[0]` is `torch.Size([1,28,28])`, the conversion is working. However, the output of the `map` is a list of list of list of list.
 # Convert raw tensors to torch format
Strangely, converting to torch tensors works perfectly on `raw_dataset`:
```python
raw_dataset.set_format('torch',columns=['image','label'])
```
Types:
```
Image:
<class 'torch.Tensor'> 60000
<class 'torch.Tensor'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 60000
<class 'torch.Tensor'>
```

Using this for transforms:
```python
def prepare_features(examples):
    images = []
    labels = []
    for example_idx, example in enumerate(examples["image"]):
        if transform is not None:
            images.append(transform(
                examples["image"][example_idx].numpy()
            ))
        else:
            images.append(examples["image"][example_idx].numpy())
        labels.append(examples["label"][example_idx])
    output = {"label":labels, "image":images}
    return output
```

Inside `prepare_train_features`:
```
Image:
<class 'list'> 10000
<class 'torch.Tensor'> 1
<class 'torch.Tensor'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'list'> 10000
<class 'torch.Tensor'>
```

After `map`:
```
Image:
<class 'list'> 60000
<class 'list'> 1
<class 'list'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 60000
<class 'torch.Tensor'>
```
DataLoader batch:

```
Image:
<class 'list'> 1
<class 'list'> 28
<class 'torch.Tensor'> 2
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 2
<class 'torch.Tensor'>
```

---

## Using `torch` format:
```
Image:
<class 'list'> 60000
<class 'list'> 1
<class 'list'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 60000
<class 'torch.Tensor'>
```
DataLoader batches:

```
Image:
<class 'list'> 1
<class 'list'> 28
<class 'torch.Tensor'> 2
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 2
<class 'torch.Tensor'>
```

---
## Using the features - `Array3D`:

```
Image:
<class 'list'> 10000
<class 'torch.Tensor'> 1
<class 'torch.Tensor'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'list'> 10000
<class 'torch.Tensor'>
```

After `map`:
```
Image:
<class 'torch.Tensor'> 60000
<class 'torch.Tensor'> 1
<class 'torch.Tensor'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 60000
<class 'torch.Tensor'>
```

After DataLoader `batch`:
```
Image:
<class 'torch.Tensor'> 2
<class 'torch.Tensor'> 1
<class 'torch.Tensor'> 28
<class 'torch.Tensor'> 28
<class 'torch.Tensor'>
Label:
<class 'torch.Tensor'> 2
<class 'torch.Tensor'>
```

The last one works perfectly.

![image](https://user-images.githubusercontent.com/29076344/110491452-4cf09c00-8117-11eb-8a47-73bf3fc0c3dc.png)

I wonder why this worked, and others didn't.


Setting to torch format not working with torchvision and MNIST
 Hi

I am trying to use `torchvision.transforms` to handle the transformation of the image data in the `mnist` dataset. Assume I have a `transform` variable which contains the `torchvision.transforms` object.

A snippet of what I am trying to do:
```python
def prepare_features(examples):
    images = []
    labels = []
    for example_idx, example in enumerate(examples["image"]):
        if transform is not None:
            images.append(transform(
                np.array(examples["image"][example_idx], dtype=np.uint8)
            ))
        else:
            images.append(torch.tensor(np.array(examples["image"][example_idx], dtype=np.uint8)))
        labels.append(torch.tensor(examples["label"][example_idx]))
    output = {"label":labels, "image":images}
    return output

raw_dataset = load_dataset('mnist')
train_dataset = raw_dataset.map(prepare_features, batched=True, batch_size=10000)
train_dataset.set_format("torch",columns=["image","label"])
```

After this, I check the type of the following:
```python
print(type(train_dataset["train"]["label"]))
print(type(train_dataset["train"]["image"][0]))
```
This leads to the following output:

```python
<class 'torch.Tensor'>
<class 'list'>
```
I use `torch.utils.DataLoader` for batches, the type of `batch["train"]["image"]` is also `<class 'list'>`.

I don't understand why only the `label` is converted to a torch tensor, why does the image not get converted? How can I fix this issue?

Thanks,
Gunjan

EDIT:
I just checked the shapes, and the types, `batch[image]` is a actually a list of list of tensors. Shape is (1,28,2,28), where `batch_size` is 2. I don't understand why this is happening. Ideally it should be a tensor of shape (2,1,28,28).

EDIT 2:
Inside `prepare_train_features`, the shape of `images[0]` is `torch.Size([1,28,28])`, the conversion is working. However, the output of the `map` is a list of list of list of list.
 Concluding, the way it works right now is:

1. Converting raw dataset to `torch` format.
2. Use the transform and apply using `map`, ensure the returned values are tensors.
3. When mapping, use `features` with `image` being `Array3D` type.


Setting to torch format not working with torchvision and MNIST
 Hi

I am trying to use `torchvision.transforms` to handle the transformation of the image data in the `mnist` dataset. Assume I have a `transform` variable which contains the `torchvision.transforms` object.

A snippet of what I am trying to do:
```python
def prepare_features(examples):
    images = []
    labels = []
    for example_idx, example in enumerate(examples["image"]):
        if transform is not None:
            images.append(transform(
                np.array(examples["image"][example_idx], dtype=np.uint8)
            ))
        else:
            images.append(torch.tensor(np.array(examples["image"][example_idx], dtype=np.uint8)))
        labels.append(torch.tensor(examples["label"][example_idx]))
    output = {"label":labels, "image":images}
    return output

raw_dataset = load_dataset('mnist')
train_dataset = raw_dataset.map(prepare_features, batched=True, batch_size=10000)
train_dataset.set_format("torch",columns=["image","label"])
```

After this, I check the type of the following:
```python
print(type(train_dataset["train"]["label"]))
print(type(train_dataset["train"]["image"][0]))
```
This leads to the following output:

```python
<class 'torch.Tensor'>
<class 'list'>
```
I use `torch.utils.DataLoader` for batches, the type of `batch["train"]["image"]` is also `<class 'list'>`.

I don't understand why only the `label` is converted to a torch tensor, why does the image not get converted? How can I fix this issue?

Thanks,
Gunjan

EDIT:
I just checked the shapes, and the types, `batch[image]` is a actually a list of list of tensors. Shape is (1,28,2,28), where `batch_size` is 2. I don't understand why this is happening. Ideally it should be a tensor of shape (2,1,28,28).

EDIT 2:
Inside `prepare_train_features`, the shape of `images[0]` is `torch.Size([1,28,28])`, the conversion is working. However, the output of the `map` is a list of list of list of list.
 What the dataset returns depends on the feature type.
For a feature type that is Sequence(Sequence(Sequence(Value("uint8")))), a dataset formatted as "torch" return lists of lists of tensors. This is because the lists lengths may vary.
For a feature type that is Array3D on the other hand it returns one tensor. This is because the size of the tensor is fixed and defined bu the Array3D type.


Setting to torch format not working with torchvision and MNIST
 Hi

I am trying to use `torchvision.transforms` to handle the transformation of the image data in the `mnist` dataset. Assume I have a `transform` variable which contains the `torchvision.transforms` object.

A snippet of what I am trying to do:
```python
def prepare_features(examples):
    images = []
    labels = []
    for example_idx, example in enumerate(examples["image"]):
        if transform is not None:
            images.append(transform(
                np.array(examples["image"][example_idx], dtype=np.uint8)
            ))
        else:
            images.append(torch.tensor(np.array(examples["image"][example_idx], dtype=np.uint8)))
        labels.append(torch.tensor(examples["label"][example_idx]))
    output = {"label":labels, "image":images}
    return output

raw_dataset = load_dataset('mnist')
train_dataset = raw_dataset.map(prepare_features, batched=True, batch_size=10000)
train_dataset.set_format("torch",columns=["image","label"])
```

After this, I check the type of the following:
```python
print(type(train_dataset["train"]["label"]))
print(type(train_dataset["train"]["image"][0]))
```
This leads to the following output:

```python
<class 'torch.Tensor'>
<class 'list'>
```
I use `torch.utils.DataLoader` for batches, the type of `batch["train"]["image"]` is also `<class 'list'>`.

I don't understand why only the `label` is converted to a torch tensor, why does the image not get converted? How can I fix this issue?

Thanks,
Gunjan

EDIT:
I just checked the shapes, and the types, `batch[image]` is a actually a list of list of tensors. Shape is (1,28,2,28), where `batch_size` is 2. I don't understand why this is happening. Ideally it should be a tensor of shape (2,1,28,28).

EDIT 2:
Inside `prepare_train_features`, the shape of `images[0]` is `torch.Size([1,28,28])`, the conversion is working. However, the output of the `map` is a list of list of list of list.
 Okay, that makes sense.
Raw images are list of Array2D, hence we get a single tensor when `set_format` is used. But, why should I need to convert the raw images to `torch` format when `map` does this internally?

Using `Array3D` did not work with `map` when raw images weren't `set_format`ted to torch type.


Setting to torch format not working with torchvision and MNIST
 Hi

I am trying to use `torchvision.transforms` to handle the transformation of the image data in the `mnist` dataset. Assume I have a `transform` variable which contains the `torchvision.transforms` object.

A snippet of what I am trying to do:
```python
def prepare_features(examples):
    images = []
    labels = []
    for example_idx, example in enumerate(examples["image"]):
        if transform is not None:
            images.append(transform(
                np.array(examples["image"][example_idx], dtype=np.uint8)
            ))
        else:
            images.append(torch.tensor(np.array(examples["image"][example_idx], dtype=np.uint8)))
        labels.append(torch.tensor(examples["label"][example_idx]))
    output = {"label":labels, "image":images}
    return output

raw_dataset = load_dataset('mnist')
train_dataset = raw_dataset.map(prepare_features, batched=True, batch_size=10000)
train_dataset.set_format("torch",columns=["image","label"])
```

After this, I check the type of the following:
```python
print(type(train_dataset["train"]["label"]))
print(type(train_dataset["train"]["image"][0]))
```
This leads to the following output:

```python
<class 'torch.Tensor'>
<class 'list'>
```
I use `torch.utils.DataLoader` for batches, the type of `batch["train"]["image"]` is also `<class 'list'>`.

I don't understand why only the `label` is converted to a torch tensor, why does the image not get converted? How can I fix this issue?

Thanks,
Gunjan

EDIT:
I just checked the shapes, and the types, `batch[image]` is a actually a list of list of tensors. Shape is (1,28,2,28), where `batch_size` is 2. I don't understand why this is happening. Ideally it should be a tensor of shape (2,1,28,28).

EDIT 2:
Inside `prepare_train_features`, the shape of `images[0]` is `torch.Size([1,28,28])`, the conversion is working. However, the output of the `map` is a list of list of list of list.
 I understand that `map` needs to know what kind of output tensors are expected, and thus converting the raw dataset to `torch` format is necessary. Closing the issue since it is resolved.


Messages are being printed to the `stdout`
 In this code segment, we can see some messages are being printed to the `stdout`.
https://github.com/huggingface/datasets/blob/7e60bb509b595e8edc60a87f32b2bacfc065d607/src/datasets/builder.py#L545-L554
According to the comment, it is done intentionally, but I don't really understand why don't we log it with a higher level or print it directly to the `stderr`.
In my opinion, this kind of messages should never printed to the stdout. At least some configuration/flag should make it possible to provide in order to explicitly prevent the package to contaminate the stdout.

 This is expected to show this message to the user via stdout.
This way the users see it directly and can cancel the downloading if they want to.
Could you elaborate why it would be better to have it in stderr instead of stdout ?


Messages are being printed to the `stdout`
 In this code segment, we can see some messages are being printed to the `stdout`.
https://github.com/huggingface/datasets/blob/7e60bb509b595e8edc60a87f32b2bacfc065d607/src/datasets/builder.py#L545-L554
According to the comment, it is done intentionally, but I don't really understand why don't we log it with a higher level or print it directly to the `stderr`.
In my opinion, this kind of messages should never printed to the stdout. At least some configuration/flag should make it possible to provide in order to explicitly prevent the package to contaminate the stdout.

 @lhoestq, sorry for the late reply

I completely understand why you decided to output a message that is always shown. The only problem is that the message is printed to the `stdout`. For example, if the user runs `python run_glue.py > log_file`, it will redirect `stdout` to the file named  `log_file`, and the message will not be shown to the user.

Instead, we should print this message to `stderr`.  Even in the case of `python run_glue.py > log_file` only `stdout` is being redirected and so the message is always shown.


Windows Permission Error (most recent version of datasets)
 Hi everyone,
Can anyone help me with why the dataset loading script below raises a Windows Permission Error? I stuck quite closely to https://github.com/huggingface/datasets/blob/master/datasets/conll2003/conll2003.py , only I want to load the data from three local three-column tsv-files (id\ttokens\tpos_tags\n). I am using the most recent version of datasets. Thank you in advance!
Luisa

My script:
```
import datasets
import csv

logger = datasets.logging.get_logger(__name__)

class SampleConfig(datasets.BuilderConfig):

    def __init__(self, **kwargs):
        super(SampleConfig, self).__init__(**kwargs)

class Sample(datasets.GeneratorBasedBuilder):
    BUILDER_CONFIGS = [
        SampleConfig(name="conll2003", version=datasets.Version("1.0.0"), description="Conll2003 dataset"),
    ]

    def _info(self):
        return datasets.DatasetInfo(
            description="Dataset with words and their POS-Tags",
            features=datasets.Features(
                {
                    "id": datasets.Value("string"),
                    "tokens": datasets.Sequence(datasets.Value("string")),
                    "pos_tags": datasets.Sequence(
                        datasets.features.ClassLabel(
                            names=[
                                "''",
                                ",",
                                "-LRB-",
                                "-RRB-",
                                ".",
                                ":",
                                "CC",
                                "CD",
                                "DT",
                                "EX",
                                "FW",
                                "HYPH",
                                "IN",
                                "JJ",
                                "JJR",
                                "JJS",
                                "MD",
                                "NN",
                                "NNP",
                                "NNPS",
                                "NNS",
                                "PDT",
                                "POS",
                                "PRP",
                                "PRP$",
                                "RB",
                                "RBR",
                                "RBS",
                                "RP",
                                "TO",
                                "UH",
                                "VB",
                                "VBD",
                                "VBG",
                                "VBN",
                                "VBP",
                                "VBZ",
                                "WDT",
                                "WP",
                                "WRB",
                                "``"
                            ]
                        )
                    ),
                }
            ),
            supervised_keys=None,
            homepage="https://catalog.ldc.upenn.edu/LDC2011T03",
            citation="Weischedel, Ralph, et al. OntoNotes Release 4.0 LDC2011T03. Web Download. Philadelphia: Linguistic Data Consortium, 2011.",
        )

    def _split_generators(self, dl_manager):
        loaded_files = dl_manager.download_and_extract(self.config.data_files)
        return [
            datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={"filepath": loaded_files["train"]}),
            datasets.SplitGenerator(name=datasets.Split.TEST, gen_kwargs={"filepath": loaded_files["test"]}),
            datasets.SplitGenerator(name=datasets.Split.VALIDATION, gen_kwargs={"filepath": loaded_files["val"]})
        ]

    def _generate_examples(self, filepath):
        logger.info("generating examples from = %s", filepath)
        with open(filepath, encoding="cp1252") as f:
            data = csv.reader(f, delimiter="\t")
            ids = list()
            tokens = list()
            pos_tags = list()
            for id_, line in enumerate(data):
                #print(line)
                if len(line) == 1:
                    if tokens:
                        yield id_, {"id": ids, "tokens": tokens, "pos_tags": pos_tags}
                        ids = list()
                        tokens = list()
                        pos_tags = list()
                else:
                    ids.append(line[0])
                    tokens.append(line[1])
                    pos_tags.append(line[2])
            # last example
            yield id_, {"id": ids, "tokens": tokens, "pos_tags": pos_tags}

def main():
    dataset = datasets.load_dataset(
        "data_loading.py", data_files={
            "train": "train.tsv",
            "test": "test.tsv",
            "val": "val.tsv"
        }
    )

    #print(dataset)

if __name__=="__main__":
    main()
```

 Hi @itsLuisa !

Could you give us more information about the error you're getting, please?
A copy-paste of the Traceback would be nice to get a better understanding of what is wrong :)


Windows Permission Error (most recent version of datasets)
 Hi everyone,
Can anyone help me with why the dataset loading script below raises a Windows Permission Error? I stuck quite closely to https://github.com/huggingface/datasets/blob/master/datasets/conll2003/conll2003.py , only I want to load the data from three local three-column tsv-files (id\ttokens\tpos_tags\n). I am using the most recent version of datasets. Thank you in advance!
Luisa

My script:
```
import datasets
import csv

logger = datasets.logging.get_logger(__name__)

class SampleConfig(datasets.BuilderConfig):

    def __init__(self, **kwargs):
        super(SampleConfig, self).__init__(**kwargs)

class Sample(datasets.GeneratorBasedBuilder):
    BUILDER_CONFIGS = [
        SampleConfig(name="conll2003", version=datasets.Version("1.0.0"), description="Conll2003 dataset"),
    ]

    def _info(self):
        return datasets.DatasetInfo(
            description="Dataset with words and their POS-Tags",
            features=datasets.Features(
                {
                    "id": datasets.Value("string"),
                    "tokens": datasets.Sequence(datasets.Value("string")),
                    "pos_tags": datasets.Sequence(
                        datasets.features.ClassLabel(
                            names=[
                                "''",
                                ",",
                                "-LRB-",
                                "-RRB-",
                                ".",
                                ":",
                                "CC",
                                "CD",
                                "DT",
                                "EX",
                                "FW",
                                "HYPH",
                                "IN",
                                "JJ",
                                "JJR",
                                "JJS",
                                "MD",
                                "NN",
                                "NNP",
                                "NNPS",
                                "NNS",
                                "PDT",
                                "POS",
                                "PRP",
                                "PRP$",
                                "RB",
                                "RBR",
                                "RBS",
                                "RP",
                                "TO",
                                "UH",
                                "VB",
                                "VBD",
                                "VBG",
                                "VBN",
                                "VBP",
                                "VBZ",
                                "WDT",
                                "WP",
                                "WRB",
                                "``"
                            ]
                        )
                    ),
                }
            ),
            supervised_keys=None,
            homepage="https://catalog.ldc.upenn.edu/LDC2011T03",
            citation="Weischedel, Ralph, et al. OntoNotes Release 4.0 LDC2011T03. Web Download. Philadelphia: Linguistic Data Consortium, 2011.",
        )

    def _split_generators(self, dl_manager):
        loaded_files = dl_manager.download_and_extract(self.config.data_files)
        return [
            datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={"filepath": loaded_files["train"]}),
            datasets.SplitGenerator(name=datasets.Split.TEST, gen_kwargs={"filepath": loaded_files["test"]}),
            datasets.SplitGenerator(name=datasets.Split.VALIDATION, gen_kwargs={"filepath": loaded_files["val"]})
        ]

    def _generate_examples(self, filepath):
        logger.info("generating examples from = %s", filepath)
        with open(filepath, encoding="cp1252") as f:
            data = csv.reader(f, delimiter="\t")
            ids = list()
            tokens = list()
            pos_tags = list()
            for id_, line in enumerate(data):
                #print(line)
                if len(line) == 1:
                    if tokens:
                        yield id_, {"id": ids, "tokens": tokens, "pos_tags": pos_tags}
                        ids = list()
                        tokens = list()
                        pos_tags = list()
                else:
                    ids.append(line[0])
                    tokens.append(line[1])
                    pos_tags.append(line[2])
            # last example
            yield id_, {"id": ids, "tokens": tokens, "pos_tags": pos_tags}

def main():
    dataset = datasets.load_dataset(
        "data_loading.py", data_files={
            "train": "train.tsv",
            "test": "test.tsv",
            "val": "val.tsv"
        }
    )

    #print(dataset)

if __name__=="__main__":
    main()
```

 Hello @SBrandeis , this is it:
```
Traceback (most recent call last):
  File "C:\Users\Luisa\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\builder.py", line 537, in incomplete_dir
    yield tmp_dir
  File "C:\Users\Luisa\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\builder.py", line 578, in download_and_prepare
    self._download_and_prepare(
  File "C:\Users\Luisa\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\builder.py", line 656, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "C:\Users\Luisa\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\builder.py", line 982, in _prepare_split
    num_examples, num_bytes = writer.finalize()
  File "C:\Users\Luisa\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\arrow_writer.py", line 297, in finalize
    self.write_on_file()
  File "C:\Users\Luisa\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\arrow_writer.py", line 230, in write_on_file
    pa_array = pa.array(typed_sequence)
  File "pyarrow\array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow\array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "C:\Users\Luisa\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\arrow_writer.py", line 97, in __arrow_array__
    out = pa.array(self.data, type=type)
  File "pyarrow\array.pxi", line 305, in pyarrow.lib.array
  File "pyarrow\array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow\error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow\error.pxi", line 107, in pyarrow.lib.check_status
pyarrow.lib.ArrowTypeError: Expected bytes, got a 'list' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/Luisa/Documents/Uni/WS 2020,21/Neural Networks/Final_Project/NN_Project/data_loading.py", line 122, in <module>
    main()
  File "C:/Users/Luisa/Documents/Uni/WS 2020,21/Neural Networks/Final_Project/NN_Project/data_loading.py", line 111, in main
    dataset = datasets.load_dataset(
  File "C:\Users\Luisa\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\load.py", line 740, in load_dataset
    builder_instance.download_and_prepare(
  File "C:\Users\Luisa\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\builder.py", line 586, in download_and_prepare
    self._save_info()
  File "C:\Users\Luisa\AppData\Local\Programs\Python\Python38\lib\contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Users\Luisa\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\builder.py", line 543, in incomplete_dir
    shutil.rmtree(tmp_dir)
  File "C:\Users\Luisa\AppData\Local\Programs\Python\Python38\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\Luisa\AppData\Local\Programs\Python\Python38\lib\shutil.py", line 618, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\Luisa\AppData\Local\Programs\Python\Python38\lib\shutil.py", line 616, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird: 'C:\\Users\\Luisa\\.cache\\huggingface\\datasets\\sample\\default-20ee7d51a6a9454f\\0.0.0\\5fc4c3a355ea77ab446bd31fca5082437600b8364d29b2b95264048bd1f398b1.incomplete\\sample-train.arrow'

Process finished with exit code 1
```


Windows Permission Error (most recent version of datasets)
 Hi everyone,
Can anyone help me with why the dataset loading script below raises a Windows Permission Error? I stuck quite closely to https://github.com/huggingface/datasets/blob/master/datasets/conll2003/conll2003.py , only I want to load the data from three local three-column tsv-files (id\ttokens\tpos_tags\n). I am using the most recent version of datasets. Thank you in advance!
Luisa

My script:
```
import datasets
import csv

logger = datasets.logging.get_logger(__name__)

class SampleConfig(datasets.BuilderConfig):

    def __init__(self, **kwargs):
        super(SampleConfig, self).__init__(**kwargs)

class Sample(datasets.GeneratorBasedBuilder):
    BUILDER_CONFIGS = [
        SampleConfig(name="conll2003", version=datasets.Version("1.0.0"), description="Conll2003 dataset"),
    ]

    def _info(self):
        return datasets.DatasetInfo(
            description="Dataset with words and their POS-Tags",
            features=datasets.Features(
                {
                    "id": datasets.Value("string"),
                    "tokens": datasets.Sequence(datasets.Value("string")),
                    "pos_tags": datasets.Sequence(
                        datasets.features.ClassLabel(
                            names=[
                                "''",
                                ",",
                                "-LRB-",
                                "-RRB-",
                                ".",
                                ":",
                                "CC",
                                "CD",
                                "DT",
                                "EX",
                                "FW",
                                "HYPH",
                                "IN",
                                "JJ",
                                "JJR",
                                "JJS",
                                "MD",
                                "NN",
                                "NNP",
                                "NNPS",
                                "NNS",
                                "PDT",
                                "POS",
                                "PRP",
                                "PRP$",
                                "RB",
                                "RBR",
                                "RBS",
                                "RP",
                                "TO",
                                "UH",
                                "VB",
                                "VBD",
                                "VBG",
                                "VBN",
                                "VBP",
                                "VBZ",
                                "WDT",
                                "WP",
                                "WRB",
                                "``"
                            ]
                        )
                    ),
                }
            ),
            supervised_keys=None,
            homepage="https://catalog.ldc.upenn.edu/LDC2011T03",
            citation="Weischedel, Ralph, et al. OntoNotes Release 4.0 LDC2011T03. Web Download. Philadelphia: Linguistic Data Consortium, 2011.",
        )

    def _split_generators(self, dl_manager):
        loaded_files = dl_manager.download_and_extract(self.config.data_files)
        return [
            datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={"filepath": loaded_files["train"]}),
            datasets.SplitGenerator(name=datasets.Split.TEST, gen_kwargs={"filepath": loaded_files["test"]}),
            datasets.SplitGenerator(name=datasets.Split.VALIDATION, gen_kwargs={"filepath": loaded_files["val"]})
        ]

    def _generate_examples(self, filepath):
        logger.info("generating examples from = %s", filepath)
        with open(filepath, encoding="cp1252") as f:
            data = csv.reader(f, delimiter="\t")
            ids = list()
            tokens = list()
            pos_tags = list()
            for id_, line in enumerate(data):
                #print(line)
                if len(line) == 1:
                    if tokens:
                        yield id_, {"id": ids, "tokens": tokens, "pos_tags": pos_tags}
                        ids = list()
                        tokens = list()
                        pos_tags = list()
                else:
                    ids.append(line[0])
                    tokens.append(line[1])
                    pos_tags.append(line[2])
            # last example
            yield id_, {"id": ids, "tokens": tokens, "pos_tags": pos_tags}

def main():
    dataset = datasets.load_dataset(
        "data_loading.py", data_files={
            "train": "train.tsv",
            "test": "test.tsv",
            "val": "val.tsv"
        }
    )

    #print(dataset)

if __name__=="__main__":
    main()
```

 Hi @itsLuisa, thanks for sharing the Traceback.

You are defining the "id" field as a `string` feature:
```python
class Sample(datasets.GeneratorBasedBuilder):
    ...

    def _info(self):
        return datasets.DatasetInfo(
            features=datasets.Features(
                {
                    "id": datasets.Value("string"),
                    # ^^ here
                    "tokens": datasets.Sequence(datasets.Value("string")),
                    "pos_tags": datasets.Sequence(datasets.features.ClassLabel(names=[...])),
[...]
```

But in the `_generate_examples`, the "id" field is a list:
```python
ids = list()
```

Changing:
```python
"id": datasets.Value("string"),
```
Into:
```python
"id": datasets.Sequence(datasets.Value("string")),
```

Should fix your issue.

Let me know if this helps!


Error when exploring `arabic_speech_corpus`
 Navigate to https://huggingface.co/datasets/viewer/?dataset=arabic_speech_corpus

Error:
```
ImportError: To be able to use this dataset, you need to install the following dependencies['soundfile'] using 'pip install soundfile' for instance'
Traceback:
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/script_runner.py", line 332, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp-viewer/run.py", line 233, in <module>
    configs = get_confs(option)
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/caching.py", line 604, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/caching.py", line 588, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp-viewer/run.py", line 145, in get_confs
    module_path = nlp.load.prepare_module(path, dataset=True
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/datasets/load.py", line 342, in prepare_module
    f"To be able to use this {module_type}, you need to install the following dependencies"
```
 Actually soundfile is not a dependency of this dataset.
The error comes from a bug that was fixed in this commit: https://github.com/huggingface/datasets/pull/1767/commits/c304e63629f4453367de2fd42883a78768055532
Basically the library used to consider the `import soundfile` in the docstring as a dependency, while it's just here as a code example.

Updating the viewer to the latest version of `datasets` should fix this issue


not being able to get wikipedia es language
 Hi
I am trying to run a code with wikipedia of config 20200501.es, getting:

Traceback (most recent call last):
  File "run_mlm_t5.py", line 608, in <module>
    main()
  File "run_mlm_t5.py", line 359, in main
    datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name)
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/load.py", line 612, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/builder.py", line 527, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/builder.py", line 1050, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
datasets.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
	`load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

thanks @lhoestq  for any suggestion/help
 @lhoestq  I really appreciate if you could help me providiing processed datasets, I do not really have access to enough resources to run the apache-beam and need to run the codes on these datasets. Only en/de/fr currently works, but I need all the languages more or less. thanks


not being able to get wikipedia es language
 Hi
I am trying to run a code with wikipedia of config 20200501.es, getting:

Traceback (most recent call last):
  File "run_mlm_t5.py", line 608, in <module>
    main()
  File "run_mlm_t5.py", line 359, in main
    datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name)
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/load.py", line 612, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/builder.py", line 527, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/builder.py", line 1050, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
datasets.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
	`load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

thanks @lhoestq  for any suggestion/help
 Hi @dorost1234, I think I can help you a little. I’ve processed some Wikipedia datasets (Spanish inclusive) using the HF/datasets library during recent research.

@lhoestq Could you help me to upload these preprocessed datasets to Huggingface's repositories? To be more precise, I've built datasets from the following languages using the 20201201 dumps: Spanish, Portuguese, Russian, French, Japanese, Chinese, and Turkish. Process these datasets have high costs that most of the community can't afford. I think these preprocessed datasets I have could be helpful for someone without access to high-resource machines to process Wikipedia's dumps like @dorost1234


not being able to get wikipedia es language
 Hi
I am trying to run a code with wikipedia of config 20200501.es, getting:

Traceback (most recent call last):
  File "run_mlm_t5.py", line 608, in <module>
    main()
  File "run_mlm_t5.py", line 359, in main
    datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name)
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/load.py", line 612, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/builder.py", line 527, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/builder.py", line 1050, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
datasets.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
	`load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

thanks @lhoestq  for any suggestion/help
 Thank you so much @jonatasgrosman , I greatly appreciate your help with them.
Yes, I unfortunately does not have access to a good resource and need it for my
research. I greatly appreciate @lhoestq  your help with uploading the processed datasets in huggingface datasets. This would be really helpful for some users like me with not access to high-memory GPU resources.

thank you both so much again.

On Sat, Mar 6, 2021 at 12:55 AM Jonatas Grosman <notifications@github.com>
wrote:

> Hi @dorost1234 <https://github.com/dorost1234>, I think I can help you a
> little. I’ve processed some Wikipedia datasets (Spanish inclusive) using
> the HF/datasets library during recent research.
>
> @lhoestq <https://github.com/lhoestq> Could you help me to upload these
> preprocessed datasets to Huggingface's repositories? To be more precise,
> I've built datasets from the following languages using the 20201201 dumps:
> Spanish, Portuguese, Russian, French, Japanese, Chinese, and Turkish.
> Process these datasets have high costs that most of the community can't
> afford. I think these preprocessed datasets I have could be helpful for
> someone without access to high-resource machines to process Wikipedia's
> dumps like @dorost1234 <https://github.com/dorost1234>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/1994#issuecomment-791798195>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMWMK5GFJFU3ACCJFUDTCFVNZANCNFSM4YUZIF4A>
> .
>


not being able to get wikipedia es language
 Hi
I am trying to run a code with wikipedia of config 20200501.es, getting:

Traceback (most recent call last):
  File "run_mlm_t5.py", line 608, in <module>
    main()
  File "run_mlm_t5.py", line 359, in main
    datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name)
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/load.py", line 612, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/builder.py", line 527, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/builder.py", line 1050, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
datasets.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
	`load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

thanks @lhoestq  for any suggestion/help
 Hi @dorost1234, so sorry, but looking at my files here, I figure out that I've preprocessed files using the HF/datasets for all the languages previously listed by me (Portuguese, Russian, French, Japanese, Chinese, and Turkish) except the Spanish (on my tests I've used the [wikicorpus](https://www.cs.upc.edu/~nlp/wikicorpus/) instead).

Only with the Spanish Wikipedia's dump, I had the same `KeyError: '000nbsp'` problem already reported here https://github.com/huggingface/datasets/issues/577

So nowadays, even with access to a high resource machine, you couldn't be able to get Wikipedia's Spanish data using the HF/datasets :(


not being able to get wikipedia es language
 Hi
I am trying to run a code with wikipedia of config 20200501.es, getting:

Traceback (most recent call last):
  File "run_mlm_t5.py", line 608, in <module>
    main()
  File "run_mlm_t5.py", line 359, in main
    datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name)
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/load.py", line 612, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/builder.py", line 527, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/builder.py", line 1050, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
datasets.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
	`load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

thanks @lhoestq  for any suggestion/help
 Thanks a lot for the information and help. This would be great to have
these datasets.
@lhoestq <https://github.com/lhoestq>  Do you know a way I could get
smaller amount of these data like 1 GBtype of each language to deal with
computatioanl requirements? thanks

On Sat, Mar 6, 2021 at 5:36 PM Jonatas Grosman <notifications@github.com>
wrote:

> Hi @dorost1234 <https://github.com/dorost1234>, so sorry, but looking at
> my files here, I figure out that I've preprocessed files using the
> HF/datasets for all the languages previously listed by me (Portuguese,
> Russian, French, Japanese, Chinese, and Turkish) except the Spanish (on my
> tests I've used the wikicorpus <https://www.cs.upc.edu/~nlp/wikicorpus/>
> instead).
>
> Only with the Spanish Wikipedia's dump, I had the same KeyError: '000nbsp'
> problem already reported here #577
> <https://github.com/huggingface/datasets/issues/577>
>
> So nowadays, even with access to a high resource machine, you couldn't be
> able to get Wikipedia's Spanish data using the HF/datasets :(
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/1994#issuecomment-791985546>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMWMO7WOHWLOROPD6Q3TCJKXPANCNFSM4YUZIF4A>
> .
>


not being able to get wikipedia es language
 Hi
I am trying to run a code with wikipedia of config 20200501.es, getting:

Traceback (most recent call last):
  File "run_mlm_t5.py", line 608, in <module>
    main()
  File "run_mlm_t5.py", line 359, in main
    datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name)
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/load.py", line 612, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/builder.py", line 527, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/builder.py", line 1050, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
datasets.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
	`load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

thanks @lhoestq  for any suggestion/help
 Hi ! As mentioned above the Spanish configuration have parsing issues from `mwparserfromhell`. I haven't tested with the latest `mwparserfromhell` >=0.6 though. Which version of `mwparserfromhell` are you using ?

> @lhoestq Could you help me to upload these preprocessed datasets to Huggingface's repositories? To be more precise, I've built datasets from the following languages using the 20201201 dumps: Spanish, Portuguese, Russian, French, Japanese, Chinese, and Turkish. Process these datasets have high costs that most of the community can't afford. I think these preprocessed datasets I have could be helpful for someone without access to high-resource machines to process Wikipedia's dumps like @dorost1234

That would be awesome ! Feel free to ping me on slack so we can put the processed wikipedia files on google storage with the other ones we've already preprocessed.

> Do you know a way I could get smaller amount of these data like 1 GBtype of each language to deal with computatioanl requirements? thanks

I'd suggest to copy the [wikipedia.py](https://github.com/huggingface/datasets/blob/master/datasets/wikipedia/wikipedia.py) to a new script `custom_wikipedia.py` and modify it to only download and process only a subset of the raw data files.
You can for example replace [this line](https://github.com/huggingface/datasets/blob/64e59fc45ca2134218b3e42e83fddddbe840ff74/datasets/wikipedia/wikipedia.py#L446) by:
```python
            if total_bytes >= (1 << 30):  # stop if the total amount of data is >= 1GB
                break
            else:
                xml_urls.append(_base_url(lang) + fname)
```

Then you can load your custom wikipedia dataset with
```python
load_dataset("path/to/my/custom_wikipedia.py", f"{date}.{language}")
```