pyarrow.lib.ArrowInvalid: Column 1 named input_ids expected length 599 but got length 1500
Hi !
The error you have is due to the `input_ids` column not having the same number of examples as the other columns.
Indeed you're concatenating the `input_ids` at this line:

https://github.com/LuCeHe/GenericTools/blob/431835d8e13ec24dceb5ee4dc4ae58f0e873b091/KerasTools/lm_preprocessing.py#L134

However the other columns are kept unchanged, and therefore you end up with an `input_ids` column with 599 elements while the others columns like `attention_mask` have 1500.

To fix that you can instead concatenate them all using
```python
concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
```

Also you may need to drop the "text" column before applying `group_texts` since strings can't be concatenated with lists. You can drop it at the tokenization step:
```python
dset = dset.map(
    tokenize_function,
    batched=True,
    remove_columns=["text"]
)
``` You saved my life.
I am trying to preprocess any dataset in this package with GPT-2 tokenizer, so I need to structure the datasets as long sequences of text without padding. I've been following a couple of your tutorials and here you can find the script that is failing right at the end

https://github.com/LuCeHe/GenericTools/blob/master/KerasTools/lm_preprocessing.py

In the last iteration of the last dset.map, it gives the error that I copied in the title. Another issue that I have, if I leave the batch_size set as 1000 in the last .map, I'm afraid it's going to lose most text, so I'm considering setting both writer_batch_size and batch_size to 300 K, but I'm not sure it's the best way to go.

Can you help me?
Thanks!