Transformer Class on dataset
Hi ! Do you have an example in mind that shows how this could be useful ? Example:

Merge 2 datasets into one datasets

Label extraction from dataset

dataset(text, label)
   —> dataset(text, newlabel)

TextCleaning.


For image dataset, 
Transformation are easier (ie linear algebra).






> On Jul 6, 2021, at 17:39, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> Hi ! Do you have an example in mind that shows how this could be useful ?
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
 There are already a few transformations that you can apply on a dataset using methods like `dataset.map()`.
You can find examples in the documentation here:
https://huggingface.co/docs/datasets/processing.html

You can merge two datasets with `concatenate_datasets()` or do label extraction with `dataset.map()` for example Ok, sure.

Thanks for pointing on functional part.
My question is more
“Philosophical”/Design perspective.

There are 2 perspetive:
  Add transformation methods to 
     Dataset Class


 OR Create a Transformer Class
   which operates on Dataset Class.

T(Dataset) —> Dataset

datasetnew = MyTransform.transform(dataset)
datasetNew.save(path)


What would be the difficulty
of implementing a Transformer Class
operating at dataset level ?


thanks









> On Jul 6, 2021, at 22:00, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> There are already a few transformations that you can apply on a dataset using methods like dataset.map().
> You can find examples in the documentation here:
> https://huggingface.co/docs/datasets/processing.html
> 
> You can merge two datasets with concatenate_datasets() or do label extraction with dataset.map() for example
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
 I can imagine that this would be a useful API to implement processing pipelines as transforms. They could be used to perform higher level transforms compared to the atomic transforms allowed by methods like map, filter, etc.

I guess if you find any transform that could be useful for text dataset processing, image dataset processing etc. we could definitely start having such transforms :) Thanks for reply.

What would be the constraints
to have
Dataset —> Dataset consistency ?

Main issue would be
larger than memory dataset and
serialization on disk.

Technically,
one still process at atomic level
and try to wrap the full results
into Dataset…. (!)

What would you think ?








> On Jul 7, 2021, at 16:51, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> I can imagine that this would be a useful API to implement processing pipelines as transforms. They could be used to perform higher level transforms compared to the atomic transforms allowed by methods like map, filter, etc.
> 
> I guess if you find any transform that could be useful for text dataset processing, image dataset processing etc. we could definitely start having such transforms :)
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
 We can be pretty flexible and not impose any constraints for transforms.

Moreover, this library is designed to support datasets bigger than memory. The datasets are loaded from the disk via memory mapping, without filling up RAM. Even processing functions like `map` work in a batched fashion to not fill up your RAM. So this shouldn't be an issue Ok thanks.

But, Dataset has various flavors.
In current design of Dataset,
   how the serialization on disk is done (?)


The main issue is serialization 
of   newdataset= Transform(Dataset)
 (ie thats why am referring to Out Of memory dataset…):

   Should be part of Transform or part of dataset ?




Maybe, not, since the output is aimed to feed model in memory (?)








> On Jul 7, 2021, at 18:04, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> We can be pretty flexible and not impose any constraints for transforms.
> 
> Moreover, this library is designed to support datasets bigger than memory. The datasets are loaded from the disk via memory mapping, without filling up RAM. Even processing functions like map work in a batched fashion to not fill up your RAM. So this shouldn't be an issue
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
 I'm not sure I understand, could you elaborate a bit more please ?

Each dataset is a wrapper of a PyArrow Table that contains all the data. The table is loaded from an arrow file on the disk.
We have an ArrowWriter and ArrowReader class to write/read arrow tables on disk or in in-memory buffers.
Just wondering if you have intenttion to create

TransformerClass :
    dataset --> dataset

and make determnistic transformation (ie not fit).





