matching the glob will be loaded.
exclude – A list of patterns to exclude from the loader.
show_progress – Whether to show a progress bar or not (requires tqdm).
Proxies to the file system loader.
parser – A blob parser which knows how to parse blobs into documents,
will instantiate a default parser if not provided.
The default can be overridden by either passing a parser or
setting the class attribute blob_parser (the latter
should be used with inheritance).
parser_kwargs – Keyword arguments to pass to the parser.
Returns
A generic document loader.
static get_parser(**kwargs: Any) → BaseBlobParser[source]¶
Override this method to associate a default parser with the class.
lazy_load() → Iterator[Document][source]¶
Load documents lazily. Use this when working at a large scale.
load() → List[Document][source]¶
Load all documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document][source]¶
Load all documents and split them into sentences.
Examples using GenericLoader¶
Grobid
Loading documents from a YouTube url
Source Code
Set env var OPENAI_API_KEY or load from a .env file