README.md

metadata

title: Chunking
emoji: 🐠
colorFrom: pink
colorTo: pink
sdk: gradio
sdk_version: 4.0.2
app_file: app.py
pinned: false
license: apache-2.0

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

#LLMlocal's Text Tokenization Tool

##Introduction

LLMlocal's Text Tokenization Tool is a user-friendly application designed to tokenize large bodies of text using various methods. This tool is built using Gradio, a Python library that allows the easy creation of web interfaces for machine learning models.

##Features Tokenization Method Selection: Users can select different text tokenization methods. The current version supports the RecursiveCharacterTextSplitter method. Customizable Parameters: Users have the flexibility to set parameters like chunk size, chunk overlap, and the number of chunks to display. Interactive Interface: The tool features an intuitive interface with dropdowns, textboxes, and number inputs for easy interaction. Installation Before using the tool, ensure you have Python and the necessary packages installed. You can install the required packages using the following command:

bash' pip install gradio pandas langchain'

##Usage To use the tool, follow these simple steps:

Launch the Tool: Run the provided Python script. This will launch the Gradio interface in your default web browser. Select a Tokenization Method: Choose the desired method from the dropdown menu. Enter Text: Type or paste the text you want to tokenize in the textbox. Set Parameters: Adjust the chunk size, chunk overlap, and the number of chunks as per your requirements. View Results: The tokenized text will be displayed in a table format with details such as chunk number, text chunk, character count, and token count. Contributing Feedback and contributions to enhance the tool are always welcome. Please feel free to raise issues or submit pull requests on the repository.

##License This tool is open-source and available under apache-2.0 .

Acknowledgments Thanks to the developers of Gradio, Pandas, and Langchain for providing the libraries that made this tool possible.