langchain.document_loaders.recursive_url_loader.RecursiveUrlLoader¶
class langchain.document_loaders.recursive_url_loader.RecursiveUrlLoader(url: str, max_depth: Optional[int] = None, use_async: Optional[bool] = None, extractor: Optional[Callable[[str], str]] = None, exclude_dirs: Optional[str] = None, timeout: Optional[int] = None, prevent_outside: Optional[bool] = None)[source]¶
Loads all child links from a given url.
Initialize with URL to crawl and any subdirectories to exclude.
:param url: The URL to crawl.
:param exclude_dirs: A list of subdirectories to exclude.
:param use_async: Whether to use asynchronous loading,
:param if use_async is true:
:param this function will not be lazy:
:param :
:param but it will still work in the expected way:
:param just not lazy.:
:param extractor: A function to extract the text from the html,
:param when extract function returns empty string:
:param the document will be ignored.:
:param max_depth: The max depth of the recursive loading.
:param timeout: The timeout for the requests, in the unit of seconds.
Methods
__init__(url[, max_depth, use_async, ...])
Initialize with URL to crawl and any subdirectories to exclude.
lazy_load()
Lazy load web pages.
load()
Load web pages.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(url: str, max_depth: Optional[int] = None, use_async: Optional[bool] = None, extractor: Optional[Callable[[str], str]] = None, exclude_dirs: Optional[str] = None, timeout: Optional[int] = None, prevent_outside: Optional[bool] = None) → None[source]¶