human
What is a “web crawler”?
assistant
A web crawler is a program or script that automatically browses the World Wide Web, typically following links from one page to another. It can be used for many purposes, such as searching for information, indexing websites, and monitoring changes in websites over time.