Text splitter that uses HuggingFace tokenizer to count length.
classmethod from_tiktoken_encoder(encoding_name: str = 'gpt2', model_name: Optional[str] = None, allowed_special: Union[Literal['all'], AbstractSet[str]] = {}, disallowed_special: Union[Literal['all'], Collection[str]] = 'all', **kwargs: Any) → TS¶
Text splitter that uses tiktoken encoder to count length.
split_documents(documents: Iterable[Document]) → List[Document]¶
Split documents.
split_text(text: str) → List[str][source]¶
Split incoming text and return chunks.
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]¶
Transform sequence of documents by splitting them.
Examples using CharacterTextSplitter¶
Hugging Face
OpenAI
Vectara Text Generation
Document Comparison
Vectorstore Agent
LanceDB
Weaviate
Activeloop’s Deep Lake
Vectara
Redis
PGVector
Rockset
Zilliz
SingleStoreDB
Annoy
Typesense
Tair
Chroma
Alibaba Cloud OpenSearch
StarRocks
Clarifai
scikit-learn
DocArrayHnswSearch
MyScale
ClickHouse Vector Search
Qdrant
Tigris
AwaDB
Supabase (Postgres)
OpenSearch
Pinecone
Azure Cognitive Search
Cassandra
Milvus
ElasticSearch
Marqo
DocArrayInMemorySearch
pg_embedding
FAISS
AnalyticDB
Hologres
MongoDB Atlas
Meilisearch
Figma
Psychic
Manifest
Caching integrations
Data Augmented Question Answering
Question answering over a group chat messages using Activeloop’s DeepLake
Retrieve from vector stores directly
Improve document indexing with HyDE