glob – The glob pattern to use to find documents.
suffixes – The suffixes to use to filter documents. If None, all files
matching the glob will be loaded.
exclude – A list of patterns to exclude from the loader.
show_progress – Whether to show a progress bar or not (requires tqdm).
Proxies to the file system loader.
parser – A blob parser which knows how to parse blobs into documents
num_workers – Max number of concurrent workers to use.
parser_kwargs – Keyword arguments to pass to the parser.
static get_parser(**kwargs: Any) → BaseBlobParser¶
Override this method to associate a default parser with the class.
lazy_load() → Iterator[Document][source]¶
Load documents lazily with concurrent parsing.
load() → List[Document]¶
Load all documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load all documents and split them into sentences.
Examples using ConcurrentLoader¶
Concurrent Loader