imagenet = load_dataset("imagenet-1k", split="train", streaming=True)  # will start loading the data when iterated over
for example in imagenet:
    print(example)
    break
```

Streaming can read online data without writing any file to disk.
For example, you can stream datasets made out of multiple shards, each of which is hundreds of gigabytes like [C4](https://huggingface.co/datasets/c4), [OSCAR](https://huggingface.co/datasets/oscar) or [LAION-2B](https://huggingface.co/datasets/laion/laion2B-en).
Learn more about how to stream a dataset in the [Dataset Streaming Guide](./stream).

This is not the only difference though, because the "lazy" behavior of an `IterableDataset` is also present when it comes to dataset creation and processing.

## Creating map-style datasets and iterable datasets

You can create a [`Dataset`] using lists or dictionaries, and the data is entirely converted to Arrow so you can easily access any row:
```python
my_dataset = Dataset.from_dict({"col_1": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]})
print(my_dataset[0])
```

To create an `IterableDataset` on the other hand, you must provide a "lazy" way to load the data.
In Python, we generally use generator functions. These functions `yield` one example at a time, which means you can't access a row by slicing it like a regular `Dataset`:
```python
def my_generator(n):
    for i in range(n):
        yield {"col_1": i}

my_iterable_dataset = IterableDataset.from_generator(my_generator, gen_kwargs={"n": 10})
for example in my_iterable_dataset:
    print(example)
    break
```

## Loading local files entirely and progressively

It is possible to convert local or remote data files to an Arrow [`Dataset`] using [`load_dataset`]:
```python
data_files = {"train": ["path/to/data.csv"]}
my_dataset = load_dataset("csv", data_files=data_files, split="train")
print(my_dataset[0])
```

However, this requires a conversion step from CSV to Arrow format, which takes time and disk space if your dataset is big.
The context provides a clear explanation of how to create an `IterableDataset` in Python. It specifies that to create an `IterableDataset`, you must provide a "lazy" way to load the data, typically using generator functions that yield one example at a time. The context includes a code example demonstrating the use of a generator function to create an `IterableDataset`. This directly answers the question about what is required to create an `IterableDataset`.


The question about creating an `IterableDataset` in Python is quite useful for machine learning developers, especially those working with NLP applications in the Hugging Face ecosystem. `IterableDataset` is a fundamental concept in PyTorch, which is often used in conjunction with Hugging Face's Transformers library for handling large datasets that do not fit into memory. Understanding how to create and use an `IterableDataset` can help developers efficiently manage data loading and preprocessing, which are critical steps in training NLP models. Therefore, this question is relevant and beneficial for developers looking to optimize their data handling processes.


The question "What is required to create an `IterableDataset` in Python?" is context-independent and can be understood without additional information. It refers to a specific concept in Python programming, and anyone familiar with Python or with access to relevant documentation can understand and answer it. The question does not rely on any external context or specific document to make sense.

