### Descriptive statistics worker

Set environment variables to configure the `descriptive-statistics` worker (`DESCRIPTIVE_STATISTICS_` prefix):

- `DESCRIPTIVE_STATISTICS_CACHE_DIRECTORY`: directory to which a dataset in parquet format is downloaded. Defaults to empty.
- `DESCRIPTIVE_STATISTICS_HISTOGRAM_NUM_BINS`: number of histogram bins (see examples below for more info).
- `DESCRIPTIVE_STATISTICS_MAX_PARQUET_SIZE_BYTES`: maximum size in bytes of the dataset's parquet files to compute statistics. Datasets with bigger size are ignored. Defaults to `100_000_000`.

#### How descriptive statistics are computed 

Descriptive statistics are currently computed for the following data types: strings, floats, and ints (including `ClassLabel` int). 
Response has two fields: `num_examples` and `statistics`. `statistics` field is a list of dicts with three keys: `column_name`, `column_type`, and `column_statistics`.

`column_type` is one of the following values:
* `class_label` - for `datasets.ClassLabel` feature
* `float` - for float dtypes ("float16", "float32", "float64")
* `int` - for integer dtypes ("int8", "int16", "int32", "int64", "uint8", "uint16", "uint32", "uint64")
* `string_label` - for string dtypes ("string", "large_string") - if there are less than or equal to `MAX_NUM_STRING_LABELS` unique values (hardcoded in worker's code, for now it's 30)
* `string_text` - for string dtypes ("string", "large_string") - if there are more than `MAX_NUM_STRING_LABELS` unique values
* `bool` - for boolean dtype ("bool")

`column_statistics` content depends on the feature type, see examples below.
##### class_label

<details><summary>example: </summary>
<p>
The context provides a clear and specific value for the default maximum size in bytes for a dataset's parquet files to compute statistics. It states that the `DESCRIPTIVE_STATISTICS_MAX_PARQUET_SIZE_BYTES` defaults to `100_000_000`. This directly answers the question without ambiguity.


This question is quite specific and technical, focusing on the default maximum size for parquet files when computing statistics in the Hugging Face ecosystem. It is useful for developers who are working with datasets and need to understand the limitations and configurations of the system, particularly when dealing with large datasets and performance optimization. Understanding these defaults can help in fine-tuning the data processing pipeline, which is a common task in NLP application development.


The question is specific and technical, but it is clear and self-contained. It asks about the default maximum size in bytes for a dataset's parquet files in a particular context (the descriptive statistics worker). An operator with access to relevant documentation or knowledge about the system should be able to understand and answer it without needing additional context.

