When building a `Tokenizer`, you can
customize its normalizer by just changing the corresponding attribute:

<tokenizerslangcontent>
<python>
<literalinclude>
{"path": "../../bindings/python/tests/documentation/test_pipeline.py",
"language": "python",
"start-after": "START replace_normalizer",
"end-before": "END replace_normalizer",
"dedent": 8}
</literalinclude>
</python>
<rust>
<literalinclude>
{"path": "../../tokenizers/tests/documentation.rs",
"language": "rust",
"start-after": "START pipeline_replace_normalizer",
"end-before": "END pipeline_replace_normalizer",
"dedent": 4}
</literalinclude>
</rust>
<node>
<literalinclude>
{"path": "../../bindings/node/examples/documentation/pipeline.test.ts",
"language": "js",
"start-after": "START replace_normalizer",
"end-before": "END replace_normalizer",
"dedent": 8}
</literalinclude>
</node>
</tokenizerslangcontent>

Of course, if you change the way a tokenizer applies normalization, you
should probably retrain it from scratch afterward.

## Pre-Tokenization

Pre-tokenization is the act of splitting a text into smaller objects
that give an upper bound to what your tokens will be at the end of
training. A good way to think of this is that the pre-tokenizer will
split your text into "words" and then, your final tokens will be parts
of those words.

An easy way to pre-tokenize inputs is to split on spaces and
punctuations, which is done by the
`pre_tokenizers.Whitespace`
pre-tokenizer:
The context provides a clear explanation of what pre-tokenization is and explicitly mentions an easy method to pre-tokenize inputs, which is to split on spaces and punctuations using the `pre_tokenizers.Whitespace` pre-tokenizer. This directly answers the question posed, making the context highly relevant and unambiguous in providing the necessary information.


The question "What is an easy way to pre-tokenize inputs?" is quite useful for machine learning developers working with NLP applications, especially within the Hugging Face ecosystem. Pre-tokenization is a crucial step in preparing text data for models, and understanding efficient methods can significantly impact the performance and accuracy of NLP models. This question is relevant for developers looking to optimize their data preprocessing pipeline and can lead to discussions about various tokenization libraries and techniques, including those provided by Hugging Face, such as the `transformers` library.


The question "What is an easy way to pre-tokenize inputs?" is relatively context-independent. It asks for a method or approach to pre-tokenize inputs, which is a common task in natural language processing. While the term "pre-tokenize" might be technical, it is a standard term in the field, and the question does not rely on any specific context or additional information to be understood. An operator with access to relevant documentation or knowledge in NLP would understand what is being asked.

