Initialize with URL to crawl and any subdirectories to exclude.
:param url: The URL to crawl.
:param exclude_dirs: A list of subdirectories to exclude.
:param use_async: Whether to use asynchronous loading,
:param if use_async is true:
:param this function will not be lazy:
:param :
:param but it will still work in the expected way:
:param just not lazy.:
:param extractor: A function to extract the text from the html,
:param when extract function returns empty string:
:param the document will be ignored.:
:param max_depth: The max depth of the recursive loading.
:param timeout: The timeout for the requests, in the unit of seconds.
lazy_load() → Iterator[Document][source]¶
Lazy load web pages.
When use_async is True, this function will not be lazy,
but it will still work in the expected way, just not lazy.
load() → List[Document][source]¶
Load web pages.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using RecursiveUrlLoader¶
Recursive URL Loader