Token Alignment for input and output data over train and test batch/dataset.
`data`
> DatasetDict({
    train: Dataset({
        features: ['input', 'output'],
        num_rows: 4500
    })
    test: Dataset({
        features: ['input', 'output'],
        num_rows: 500
    })
})

**# input (in-correct sentence)**
`data['train'][0]['input']`
**>>** 'We are meet sunday 10am12pmET in Crown Heights Brooklyn New York'
**# output (correct sentence)**
`data['train'][0]['output']`
**>>** 'We meet Sundays 10am-12pmET in Crown Heights, Brooklyn, New York.'

**I Want to align the output tokens with input**

```
`# tokenize both inputs and targets
def tokenize_fn(batch):
    # tokenize the input sequence first
    # this populates input_ids, attention_mask, etc.

    tokenized_inputs = tokenizer(
    batch['input']
    )
    
    labels_batch = tokenizer.tokenize(batch['output']) # original targets

    aligned_labels_batch = []
    for i, labels in enumerate(labels_batch):
        word_ids = tokenized_inputs[i].word_ids()
        aligned_labels_batch.append(align_targets(labels, word_ids))  # align_targets is another user defined function which is been called here

    # recall: the 'target' must be stored in key called 'labels'
    tokenized_inputs['labels'] = aligned_labels_batch

    return tokenized_inputs`
```
```
data.map(
  tokenize_fn,
  batched=True,
  remove_columns=data['train'].column_names,
)
```

When this user defined function is mapped to every records of train and test batch am getting following error:

**1.** **raise DatasetTransformationNotAllowedError(
   3457         "Using `.map` in batched mode on a dataset with attached indexes is allowed only if it doesn't create or remove existing examples. You can first run `.drop_index() to remove your index and then re-add it."**

**2.** **TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]**