langchain_community.document_loaders.docusaurus.DocusaurusLoader¶
class langchain_community.document_loaders.docusaurus.DocusaurusLoader(url: str, custom_html_tags: Optional[List[str]] = None, **kwargs: Any)[source]¶
Loader that leverages the SitemapLoader to loop through the generated pages of a
Docusaurus Documentation website and extracts the content by looking for specific
HTML tags. By default, the parser searches for the main content of the Docusaurus
page, which is normally the <article>. You also have the option to define your own
custom HTML tags by providing them as a list, for example: [“div”, “.main”, “a”].
Initialize DocusaurusLoader
:param url: The base URL of the Docusaurus website.
:param custom_html_tags: Optional custom html tags to extract content from pages.
:param kwargs: Additional args to extend the underlying SitemapLoader, for example:
filter_urls, blocksize, meta_function, is_local, continue_on_failure
Attributes
web_path
Methods
__init__(url[, custom_html_tags])
Initialize DocusaurusLoader :param url: The base URL of the Docusaurus website. :param custom_html_tags: Optional custom html tags to extract content from pages. :param kwargs: Additional args to extend the underlying SitemapLoader, for example:                filter_urls, blocksize, meta_function, is_local, continue_on_failure.
aload()
Load text from the urls in web_path async into Documents.
fetch_all(urls)
Fetch all urls concurrently with rate limiting.
lazy_load()
Lazy load text from the url(s) in web_path.
load()
Load sitemap.
load_and_split([text_splitter])
Load Documents and split into chunks.
parse_sitemap(soup)
Parse sitemap xml and load into a list of dicts.