What are some issues building an internet scale web crawler?
https://news.ycombinator.com/item?id=16775863
I used to work at a vertically-focused web search engine and ran the operational...
2018-04-06
None
<div><p></p></div>||||I|||| Hacker News new | past | comments | ask | show | jobs | submit login      
       

 hcoyote on April 6, 2018 | parent | context | favorite | on: How would you build an internet scale web crawler?     
       
 I used to work at a vertically-focused web search engine and ran the operational side of the crawler.     
       
 Also missing from this discussion would be a mechanism to rate limit (and determine adequate rate limits, based on your error rates) the crawl.     
       
 Also, detecting that you've been blocked and backing off so as not to further hammer the site you're crawling with requests. Related:     
       
 IP management is an issue here as well: lots of places just carte blanche block whole ranges from crawling activity. And will you be honoring robots.txt or not?    
       
 Be prepared for people to block you in new and stupid ways: once got blocked from hitting the site's name servers to even do lookups against them. They blackholed our packets. So what should have been a ~500ms DNS query at each http request turned into a 15s pause while the DNS request timed out ... eventually this stacked up across all threads, backing the overall crawling infrastructure to deadlock.
       
 The Wayback Machine architecture is probably a good, public implementation of a large scale crawling mechanism. This post[1] about it may be a bit dated, but it's probably still accurate.    
       
 [1] http://highscalability.com/blog/2014/5/19/a-short-on-how-the...      
       
       
       
       
       
       
       
       
       
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact      
       
Search:       

Hacker News new | past | comments | ask | show | jobs | submit login      
       
 hcoyote on April 6, 2018 | parent | context | favorite | on: How would you build an internet scale web crawler?     
       
 I used to work at a vertically-focused web search engine and ran the operational side of the crawler.     
       
 Also missing from this discussion would be a mechanism to rate limit (and determine adequate rate limits, based on your error rates) the crawl.     
       
 Also, detecting that you've been blocked and backing off so as not to further hammer the site you're crawling with requests. Related:     
       
 IP management is an issue here as well: lots of places just carte blanche block whole ranges from crawling activity. And will you be honoring robots.txt or not?    
       
 Be prepared for people to block you in new and stupid ways: once got blocked from hitting the site's name servers to even do lookups against them. They blackholed our packets. So what should have been a ~500ms DNS query at each http request turned into a 15s pause while the DNS request timed out ... eventually this stacked up across all threads, backing the overall crawling infrastructure to deadlock.
       
 The Wayback Machine architecture is probably a good, public implementation of a large scale crawling mechanism. This post[1] about it may be a bit dated, but it's probably still accurate.    
       
 [1] http://highscalability.com/blog/2014/5/19/a-short-on-how-the...      
       
       
       
       
       
       
       
       
       
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact      
       
Search:
I used to work at a vertically-focused web search engine and ran the operational...
https://news.ycombinator.com/item?id=16775863
https://xetera.dev/building-a-scalable-scraper/
🔭 Building a scalable scraper
2021-12-25
None
<div><article><a href="https://github.com/xetera/jiu"></a><hr /><p>For the past year, I’ve been working on a website for Kpop image aggregation and labeling called <a href="https://kiyomi.io/">Kiyomi</a>. I’m interested in imageboards like <a href="https://safebooru.org/">Safebooru</a> that have an active community of fans who label every anime image to make searching easy for everyone. I love seeing how members come together to create a massive database of carefully curated content, and it’s only possible because of how much diverse content there is for the community to go through. For that reason, one of the challenges in working on Kiyomi has been to make sure that I can get fresh content to users automatically. Thankfully, there’s nothing Kpop idols love more than posting images of themselves online, so there’s an endless source of relevant material to work with.</p><p>I decided to solve this with what I know best, a scraper that I named <strong>Jiu</strong> after one of the top 7 <a href="https://www.youtube.com/watch?v=PEKkdIT8JPM">Dreamcatcher</a> members. In case you’re not familiar, scraping is a way of programmatically going to a website and extracting some kind of information from it. In my case, this information is image URLs and their associated metadata.</p><p>Normally this kind of image retrieval task is trivial to work with if all you need to check is a handful of pages on a single site. It’s nothing more than a for loop, some database lookups, and a bunch of HTTP requests. But I was planning on scraping many websites, which meant the solution had to be easily extendable to support new sites. I also had to consider another thing, which is that I don’t wanna like… be an asshole? I don’t like it when devs carelessly hammer my website with requests. So given the nature of what I’m doing, it would be pretty hypocritical of me to try to do the same for others. This meant that while working on this service, I first had to exhaust all of the “good guy” workarounds before I moved on to less-friendly measures like rotating IPs and pretending to be a real user with puppeteer.</p><figcaption>Also, proxies that work well are expensive, and I'm cheap</figcaption><p>This project was originally meant to be a part of Kiyomi, but I figured the problem of finding images online was generic enough that it could be <a href="https://github.com/xetera/jiu">its own standalone thing</a>. Solving some of these problems has been incredibly fun and a huge learning process for me, so I’d like to document at least some of the ways I dealt with them.</p><h2><a href="#the-problem-space">The problem space</a></h2><p>This isn’t crucial to understanding the problem, but I want to share the infrastructure Jiu sits on top of because it could provide important context. Also, I spent a lot of time working on this and drawing the diagram for it, so I’m going to need someone to look at it.</p><figure><div><p>
 <a href="/static/4bd614220cb04bd6a05d667b6d5ca3e5/2dc7d/kiyomi_arch.png">
 
 
 
 
 
 
 </a>
 </p><figcaption>Kiyomi architecture map</figcaption></div></figure><p>New images Jiu finds on supported sites are stored in a database isolated from every other service and get published to RabbitMQ for consumers to do whatever with. Jiu also runs its own webserver to allow dynamically adding and removing new providers, querying the stats of scrapes, and displaying its schedule/history.</p><p>
 <a href="/static/edc3932389f1440c61a6dd715a7ea45e/0940f/scrape_schedule.png">
 
 
 
 
 
 
 </a>
 </p><p>The requirements for building this scraper are:</p><ul><li>Creating a “plugin” system capable of supporting multiple social media sites</li><li>Turning polling into an event-based system with webhooks/AMQP</li><li>Automatically logging in to sites that require authentication</li><li>Being able to handle failure from APIs gracefully</li><li>Finding a high volume of images from different sites</li><li>Being nice and not sending too many requests to any specific site (if possible)</li></ul><br /><h2><a href="#i-am-scraping-respectfully">I am scraping respectfully</a></h2><p>The easiest way to prevent spamming a server with requests is to just not send it many requests, duh. This is an obvious one, but it’s something that works surprisingly well for what we’re building. Because the point of Jiu is to eventually detect all images posted on social media and not as soon as possible. We can use this to our advantage when deciding the architecture of the scraper.</p><p>A common way of implementing these scrapers is something commonly referred to as a monitor. A monitor takes one or a handful of critical pages and bombards it with requests to detect a change as soon as possible.</p><p>
 <a href="/static/ff466a38eeaf13efa4fd8e55ff5d2115/db783/monitor.png">
 
 
 
 
 
 
 </a>
 </p><p>This is useful for reacting to changes made in a website quickly, but it’s not that important for this specific use case. Instead, the goal here is to detect changes across every social media account that posts about Kpop, but in a sustainable way (aka slowly). So we want the same volume of requests, just spread out thinner over time and across significantly more pages.</p><p>
 <a href="/static/d5926b2a60c2214ac04c63dc9d0ce656/db783/slow_monitor.png">
 
 
 
 
 
 
 </a>
 </p><h2><a href="#priority">Priority</a></h2><p>Something important to recognize is that not all pages are created the same. There are accounts that consistently post good content, usually ran by companies behind groups; these should be checked frequently. There are also accounts that post infrequently, or post second-hand content (like with fan accounts); these should be checked less frequently. This requirement calls for implementing a priority system where different pages can automatically adjust their priority depending on the post frequency, on top of a manual multiplier if needed.</p><p>Originally, I went for a dynamic priority system where tasks would get re-scheduled based on how many images they found as they moved up and down on a discrete priority scale. The highest priority scrapes would get scheduled for 4 hours after a request, and the lowest would get two weeks. This worked well for changing scrape frequencies dynamically, but it created an issue with scrape distribution. It allowed multiple scheduled scrapes to fall within the same hour and get the bot rate limited by accident. This is unlikely with only a few endpoints but would become more and more likely as the app scaled, and that’s a problem.</p><p>I played around with some numbers and re-scheduling details, but the problem here was scheduling itself. I had to completely re-think how I approached it. In the end, I went with something similar to a <a href="https://en.wikipedia.org/wiki/Token_bucket">token bucket</a>, often used for rate limiting algorithms. Every page/account has a bucket of tokens and a rate in which it gains new tokens (called priority). At the start of every day, their tokens get updated by their priority. If a page has more than one token, they get scheduled one scrape per token sometime within the next 24 hours, and that amount gets subtracted from their tokens.</p><p>At the start of the day, a database that looks like the following will get these tasks scheduled.</p><div><table><thead><tr><th>Provider Type</th><th>Priority</th><th>Tokens</th></tr></thead><tbody><tr><td>A</td><td>1.0</td><td>2.0</td></tr><tr><td>B</td><td>1.0</td><td>2.2</td></tr><tr><td>C</td><td>0.5</td><td>2.5</td></tr></tbody></table></div><figure><div><p>
 <a href="/static/6bbe0ca5953d5c3f2aeec0eb9d496b79/e2bb0/bad_distribution.png">
 
 
 
 
 
 
 </a>
 </p><figcaption>Example list of scheduled scrapes</figcaption></div></figure><p>After every successful scrape, the next priority of the provider is calculated by getting the latest scrape result + the past 30 scrapes, and summing up the amount of images it found per check, weighed by how long ago the scrape was from the current day.</p><figure><div><p>
 <a href="/static/72253fab2999eae65fa39f5df72423d6/2ed34/priority_graph.png">
 
 
 
 
 
 
 </a>
 </p><figcaption>Example list of scheduled scrapes</figcaption></div></figure><p>This makes sure that an inactive account that starts to post images gets a higher bump in priority, and an active account that stops posting images gets knocked down an equivalent amount.</p><p>When the tokens for a provider is under 1.0, it gets compounded by its priority every day even if it doesn’t get scraped. A priority can’t go below 0.07 (meaning a scrape once every 2 weeks) and can’t go above 1.75 (3 times every 2 days).</p><h2><a href="#distribution">Distribution</a></h2><p>Once all the tasks that need to run within the next 24 hours are identified, they need to be scheduled for an exact time with an even distribution. We can’t just blindly start going through all pages that need to be sent requests immediately, otherwise we run into the exact same rate limiting problem from before.</p><p>That day’s worth of pending requests need to be grouped by their provider type —think Instagram, Twitter, etc.— and scheduled in an evenly-spaced manner using a linear interpolation between NOW() and NOW() + 24 hours</p><p>
 <a href="/static/f0cadfa0433c2e64669dd500b04cb2ac/d1d24/task_distribution.png">
 
 
 
 
 
 
 </a>
 </p><pre><div><pre><p>.pending-requests {</p><p> display: flex;</p><p> justify-content: space-evenly;</p><p>}</p></pre></div></pre><p>This method of scheduling ensures that pages on different domains that don’t share rate limits are processed concurrently, while pages that do share a rate limit get processed sequentially.</p><p>With what we know now, let’s take another look at the scheduled tasks from last time and imagine each letter is an individual Twitter page, each scheduled to get processed at an even interval after the other.</p><p>
 <a href="/static/6bbe0ca5953d5c3f2aeec0eb9d496b79/e2bb0/bad_distribution.png">
 
 
 
 
 
 
 </a>
 </p><figcaption>Can you spot the inefficiency here?</figcaption><p>We’re scheduling the same page to be scraped multiple times right after each other. We’re making sure to space the scrapes within the same provider evenly but leaving uneven spaces between repeated scrapes.</p><p>The correct order of the tasks should be more like this:</p><p>
 <a href="/static/c47786eca1bdcc33a8518da6394472fe/6b26f/good_distribution.png">
 
 
 
 
 
 
 </a>
 </p><p>For this reason, Jiu tries its best to maximize the spacing between requests going to the same endpoint so that there’s enough time for content to update.</p><p>“Tries its best” because sadly <a href="https://stackoverflow.com/questions/12375831/algorithm-to-separate-items-of-the-same-type">the solution</a> to this problem is not simple or deterministic.</p><h2><a href="#authentication">Authentication</a></h2><p>Another unfortunate problem that must be dealt with is that not every single website lets us view images without being logged in. To be able to do this, the login flow of the site needs to be <a href="https://gist.github.com/Xetera/aa59e84f3959a37c16a3309b5d9ab5a0">reverse engineered</a> in order to grab the necessary images. This also sometimes comes with the downside of needing to fake a user agent (or possibly other fingerprints).</p><p>Now, I know that reverse engineering and using a private API for fetching data and going outside the boundaries of what a site thinks is acceptable kind of contradicts wanting to be nice to websites and not spamming them. But using official APIs can get us rate-limited quickly, even with large gaps in requests. I’m going out of my way to be careful with the sites I’m scraping, so I deserve this. Besides, I know what’s best for Twitter more than Twitter knows what’s best for herself, and I promise I will treat her like the queen she is if she just gives me a chance.</p><figcaption><a href="https://vm.tiktok.com/ZM83u526w/">context</a></figcaption><h2><a href="#architecture-and-code">Architecture and Code</a></h2><p>For the more technical part, instead of copy-pasting functions from the codebase, I’m going to walk you through building a mini-scraper logic so you can maybe get a feel for what it would’ve been like to come up with some of the abstractions I haven’t mentioned yet on your own.</p><p>I wrote this program in Rust, but I’m going to use Typescript examples because it’s probably way more accessible for my audience. Feel free to look at the Rust source if you’re interested In case you’re interested, do not feel free to look through my Rust code. Please just keep reading the examples. I will take legal action against anyone I catch browsing through the repo.</p><p>Let’s start with the most fundamental idea. A requirement for this project is that we need a solution that allows fetching different forms of media from many sites. One assumption we can make is that every image/video should conform to one shared interface no matter its source. At this point, one could go for a simple class that turns a URL into an array of images.</p><pre><div><div><p>Typescript</p><div></div></div><pre><p>class Provider {</p><p> constructor(public siteName: string) {}</p><p> async scrape(url: string): Promise&lt;Image[]&gt; {</p><p> const images = await this.request(url)</p><p> return this.extractAllImages(images)</p><p> }</p><p>}</p></pre></div></pre><figcaption>Try to use your imagination for the functions I haven't defined</figcaption><p>We should now account for the possibility that this is the first time we’re gathering data from that page and haven’t seen any images. We need to add a pagination concept so our scraper can go through the page history and grab everything.</p><pre><div><div><p>Typescript</p><div></div></div><pre><p>class Provider {</p><p> async scrape(url: string): Promise&lt;Image[]&gt; {</p><p> let target = url</p><p> const images: Image[] = []</p><p> while (true) {</p><p> const response = await this.request(target)</p><p> const pagination = this.getPagination(response)</p><p> images.push(this.extractAllImages(response))</p><p> if (pagination.hasNextPage) {</p><p> target = this.getNextPage(target)</p><p> } else {</p><p> break</p><p> }</p><p> }</p><p> return images</p><p> }</p><p>}</p></pre></div></pre><p>Now we have a way of getting all images from a website. Except, with this approach, we always paginate through the full resource for every check. We should be able to stop when we find an image we’ve seen before and only “gather” the ones we haven’t seen in a previous run. This can be solved by passing an array of previously seen images to the function to test against.</p><pre><div><div><p>Typescript</p><div></div></div><pre><p>class Provider {</p><p> async scrape(url: string, seen: string[]): Promise&lt;Image[]&gt; {</p><p> let target = url</p><p> const images: Image[] = []</p><p> while (true) {</p><p> const response = await this.request(target)</p><p> const pagination = this.getPagination(response)</p><p> const found = this.extractAllImages(response)</p><p> const unseen = this.allUnseenImages(found, seen)</p><p> images.push(unseen)</p><p> if (unseen.length !== found.length) {</p><p> break</p><p> }</p><p> if (pagination.hasNextPage) {</p><p> target = this.getNextPage(target)</p><p> } else {</p><p> break</p><p> }</p><p> }</p><p> return images</p><p> }</p><p>}</p></pre></div></pre><p>Great, it looks like this is working fine for one provider like Twitter, but it’s still not scalable. The entire point of the project was to implement a solution that can have multiple sites plugged into it, and every provider is going to require a different way of fetching and processing data. There’s a <code>Provider</code> class here, but it’s not extensible in any way. We can’t configure the parts that need to be swapped out for different sites because everything gets done in a single function. Perhaps OOP is the right fix? Maybe all we need to do is extend <code>Provider</code> and reuse the logic somehow.</p><pre><div><div><p>Typescript</p><div></div></div><pre><p>class WeverseProvider extends Provider {</p><p> override async scrape(url: string, seen: string[]): Promise&lt;Image[]&gt; {</p><p> }</p><p>}</p></pre></div></pre><p>But what exactly goes in this function now, literally everything we had in the previous class? Aside from <code>this.request</code>, every single method on here is custom (mostly) per-provider and can’t just be overridden. It would be nice to have a way to almost… “template” a class method so that the <code>if</code> checks —like the part where we look to see if we’ve reached the end of everything we’ve scraped— and <code>break</code> statements stay, but everything else is customized.</p><p>As you may have guessed, there’s a way to do that. Well, not with templating, god no. That was a terrible call; what are you crazy? The issue is <code>Provider</code> is trying to do way too much here, and it’s getting in our way. As a rule of thumb, when you run into a problem where you need to copy-paste logic, creating reusable functions is usually the correct approach. However, when you run into problems where you need to copy-paste control flow, the solution is getting rid of tight coupling.</p><p>I mean, why does this class need to care at all about previously seen images? A provider’s scrape function only needs to scrape, and that’s it, no? It’s time to turn our single class method into a producer/consumer pair. Ideally, we need a producer that knows nothing about the outside world. And a consumer that only knows about the promise that a provider makes about what it’s capable of doing through an interface.</p><p>Let’s call the new class that will decouple that logic <code>Scraper</code>, and clean up the function names to reflect the new responsibilities of the two.</p><pre><div><div><p>Typescript</p><div></div></div><pre><p>class Scraper {</p><p> async scrape(provider: Provider, url: string): Promise&lt;Image[]&gt; {</p><p> const seen = await this.getLatestResultsFor(provider)</p><p> const found = await provider.process(url)</p><p> return provider.allUnseenImages(found, seen)</p><p> }</p><p>}</p><p>class Provider {</p><p> async process(url: string): Promise&lt;Image[]&gt; {</p><p> const response = await this.request(target)</p><p> return this.extractAllImages(response)</p><p> }</p><p>}</p></pre></div></pre><p>This works well for checking images. We can do something similar to re-implement pagination. Except, this time, the while loop can’t be inside the provider; it has to be inside <code>Scraper</code>. The problem with that is the scraper doesn’t understand what pagination means. Most sites (like Weverse) use page-based pagination, where you increase page numbers by 1 to go to the next page. But there are also sites (like Pinterest &amp; Twitter) that use a cursor-based pagination technique, where you pass the server some opaque identifier from request one that represents the next page for request two.</p><p><code>Scraper</code> shouldn’t have to know anything about how a provider’s pagination works, only that pagination is a thing that exists and needs to be passed back to a provider. <code>Provider</code> needs to accept and return information about pagination so it can decide what page to scrape by itself.</p><pre><div><div><p>Typescript</p><div></div></div><pre><p>class Scraper {</p><p> async scrape(provider: Provider, url: string): Promise&lt;Image[]&gt; {</p><p> const seen = await this.getLatestResultsFor(provider)</p><p> let pagination: Pagination | undefined</p><p> let images: Image[] = []</p><p> while (true) {</p><p> const result = await provider.process(url, pagination)</p><p> const unseen = provider.allUnseenImages(found, seen)</p><p> images.push(unseen)</p><p> if (unseen.length !== result.images.length) {</p><p> break</p><p> }</p><p> if (result.nextPage) {</p><p> pagination = result.nextPage</p><p> } else {</p><p> break</p><p> }</p><p> }</p><p> return images</p><p> }</p><p>}</p><p>type Pagination =</p><p> | { type: "cursor"; id: string }</p><p> | { type: "page"; index: number }</p><p>type ProviderResult = { images: Image[]; nextPage?: Pagination }</p><p>class Provider {</p><p> async process(url: string, pagination?: Pagination): Promise&lt;ProviderResult&gt; {</p><p> const target = this.buildUrlWithPagination(url, pagination)</p><p> const response = await this.request(target)</p><p> const images = this.extractAllImages(response)</p><p> const nextPage = this.extractNextPage(response)</p><p> return { images, nextPage }</p><p> }</p><p>}</p></pre></div></pre><p>This approach works perfectly! Though, the observant functional programmers may have noticed that the thing we’re doing here looks awfully similar to array.reduce. We’re going through a loop and feeding the result of a function call in one iteration (in our case, pagination) back into the function called in the next iteration, much like reduce. It could be useful to re-package iteration logic with a functional helper if it makes things easier to reason about.</p><p>As a refresher, <code>reduce</code> (also referred to as <code>fold</code> in some languages) works by doing something like the following, where it takes an A’, an array of values, and produces E’ as an output.</p><p>
 <a href="/static/971f39e1666f714a45498d14df8711f1/5d36b/reduce.png">
 
 
 
 
 
 
 </a>
 </p><p>But this doesn’t quite work for us. We have an initial seed value of a URL, yes. But we don’t have an array of things we can go over. Instead we want to create that array of things (image URLs) as a result of repeatedly applying a function (scrape) on something (pagination). Almost like a fold, but the other way around. Thankfully the thing we’re looking for here is a well-known concept like <code>fold</code> and is unsurprisingly called <code>unfold</code>.</p><p>
 <a href="/static/9545389ec01196b7a8e16f74f37caad4/c655d/unfold.png">
 
 
 
 
 
 
 </a>
 </p><p>We start with an initial value A’ and produce two outputs, one as the value that will be “pushed” into an array and the other which will get passed as an argument into the next iteration.</p><pre><div><div><p>Typescript</p><div></div></div><pre><p>class Scraper {</p><p> scrape(provider: Provider, url: string): AsyncGenerator&lt;Image[]&gt; {</p><p> return unfold(undefined, async pagination =&gt; {</p><p> const result = await provider.process(url, pagination)</p><p> if (result.nextPage) {</p><p> return next(result.images, result.nextPage)</p><p> } else {</p><p> return stop(result.images)</p><p> }</p><p> })</p><p> }</p><p>}</p></pre></div></pre><p>The hypothetical stop function here lets us signal the unfold to stop early if we’ve exhausted all resources. Typically, this would be done by returning an <code>Option</code> data type in a language like Rust, but it’s hard to represent that as clearly in JS.</p><p>This approach to iteration lets us decouple logic even further. Unlike <code>fold</code> which reduces an iterable into a single value synchronously, <code>unfold</code> generates an array of values. If the operation that is being run to generate values is asynchronous, that array can also be represented as an infinite stream of values that a consumer can stop generating at any point. <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol/asyncIterator">Async iterators</a> are great for doing this in Javascript.</p><pre><div><div><p>Typescript</p><div></div></div><pre><p>const seen = await getLatestResultsFor(provider)</p><p>for await (const images of scraper.scrape(provider, url)) {</p><p> const unseen = scraper.getUnseenImages(images, seen)</p><p> if (unseen.length !== images.length) {</p><p> break</p><p> }</p><p> await processImages(unseen)</p><p> await sleep(1000)</p><p>}</p></pre></div></pre><p>Since we’re using generators now, the scraper doesn’t need to know the conditions to look for to stop doing its job anymore; it stops when we tell it to stop inside the consumer (or runs out of pages). This means that if we wanted to, we could conveniently get the scraper to keep going even if it finds known images —like when re-fetching all images posted by a specific account— without touching the scraping logic.</p><p>Error handling and other related things can also be done now by enriching the return value of <code>process</code> to include more information about what happened. A provider can accept an error <em>that it itself created in a different function</em> and figure out how to deal with it without actually dealing with it to further decouple the process of handing off errors.</p><pre><div><div><p>Typescript</p><div></div></div><pre><p>type ProcessResult =</p><p> | { type: "success"; result: ProviderResult }</p><p> | { type: "error"; response: Response }</p><p>type ErrorHandle =</p><p> | { type: "tryAgain" }</p><p> | { type: "tryAuth" }</p><p> | { type: "cantContinue"; reason: Error }</p><p>async function iterate(pagination?: Pagination) {</p><p> const process = await provider.process(url, pagination)</p><p> if (process.error) {</p><p> const decision: ErrorHandle = await provider.onError(process.response)</p><p> switch (decision.type) {</p><p> case "tryAgain":</p><p> return iterate(pagination)</p><p> case "tryAuth":</p><p> await provider.authenticate()</p><p> return iterate(pagination)</p><p> case "cantContinue":</p><p> return stop([])</p><p> }</p><p> }</p><p> const { images, nextPage } = process.result</p><p> if (nextPage) {</p><p> return next(images, nextPage)</p><p> } else {</p><p> return stop(images)</p><p> }</p><p>}</p><p>return unfold(undefined, iterate)</p></pre></div></pre><p>Now, adding new providers is simple. It requires no duplication of control flow and not even a class. All that’s needed is some object that satisfies an interface. Notice how even with a function like <code>onError</code>, we are simply taking a response as input and returning a decision as a response, not performing an action. This framework allows the gears of <code>Scraper</code> and <code>Provider</code> to turn together without stepping over each other’s toes. Providers only know how to go, and scrapers only know how to tell providers to go.</p><pre><div><div><p>Typescript</p><div></div></div><pre><p>interface Provider {</p><p> process(url: string, pagination?: Pagination): Promise&lt;ProviderResult&gt;</p><p> onError(response: Response): Promise&lt;ErrorHandle&gt;</p><p>}</p></pre></div></pre><hr /><p>I’ve skipped over how further implementation details of providers, persistence, publishing data, and other shenanigans work in Jiu in the interest of not boring people to death. But hopefully, this helps get a basic overview of implementing a generic scraper and decoupling business logic.</p><p>There are a couple more intriguing things I’ve come across while working on Kiyomi that I want to write about, like perceptual hashing and face recognition, but those will have to wait until I actually have any idea what I’m doing with them.</p><p>Here’s <a href="https://github.com/Xetera/jiu">Jiu</a> if you’re interested, and <a href="https://kiyomi.io/">Kiyomi</a>.</p></article>||||I|||| Article — 20 min read

 🔭 Building a scalable scraper

 My Rust project for (politely) checking thousands of social media pages every day

 Programming

 Personal Project

 Rust

 December 25, 2021

 about 1 year ago

 For the past year, I’ve been working on a website for Kpop image aggregation and labeling called Kiyomi. I’m interested in imageboards like Safebooru that have an active community of fans who label every anime image to make searching easy for everyone. I love seeing how members come together to create a massive database of carefully curated content, and it’s only possible because of how much diverse content there is for the community to go through. For that reason, one of the challenges in working on Kiyomi has been to make sure that I can get fresh content to users automatically. Thankfully, there’s nothing Kpop idols love more than posting images of themselves online, so there’s an endless source of relevant material to work with.

 I decided to solve this with what I know best, a scraper that I named Jiu after one of the top 7 Dreamcatcher members. In case you’re not familiar, scraping is a way of programmatically going to a website and extracting some kind of information from it. In my case, this information is image URLs and their associated metadata.

 Normally this kind of image retrieval task is trivial to work with if all you need to check is a handful of pages on a single site. It’s nothing more than a for loop, some database lookups, and a bunch of HTTP requests. But I was planning on scraping many websites, which meant the solution had to be easily extendable to support new sites. I also had to consider another thing, which is that I don’t wanna like… be an asshole? I don’t like it when devs carelessly hammer my website with requests. So given the nature of what I’m doing, it would be pretty hypocritical of me to try to do the same for others. This meant that while working on this service, I first had to exhaust all of the “good guy” workarounds before I moved on to less-friendly measures like rotating IPs and pretending to be a real user with puppeteer.

 Also, proxies that work well are expensive, and I'm cheap

 This project was originally meant to be a part of Kiyomi, but I figured the problem of finding images online was generic enough that it could be its own standalone thing. Solving some of these problems has been incredibly fun and a huge learning process for me, so I’d like to document at least some of the ways I dealt with them.

 The problem space

 This isn’t crucial to understanding the problem, but I want to share the infrastructure Jiu sits on top of because it could provide important context. Also, I spent a lot of time working on this and drawing the diagram for it, so I’m going to need someone to look at it.

 Kiyomi architecture map

 New images Jiu finds on supported sites are stored in a database isolated from every other service and get published to RabbitMQ for consumers to do whatever with. Jiu also runs its own webserver to allow dynamically adding and removing new providers, querying the stats of scrapes, and displaying its schedule/history.

 The requirements for building this scraper are:

 * Creating a “plugin” system capable of supporting multiple social media sites
 * Turning polling into an event-based system with webhooks/AMQP
 * Automatically logging in to sites that require authentication
 * Being able to handle failure from APIs gracefully
 * Finding a high volume of images from different sites
 * Being nice and not sending too many requests to any specific site (if possible)


 I am scraping respectfully

 The easiest way to prevent spamming a server with requests is to just not send it many requests, duh. This is an obvious one, but it’s something that works surprisingly well for what we’re building. Because the point of Jiu is to eventually detect all images posted on social media and not as soon as possible. We can use this to our advantage when deciding the architecture of the scraper.

 A common way of implementing these scrapers is something commonly referred to as a monitor. A monitor takes one or a handful of critical pages and bombards it with requests to detect a change as soon as possible.

 This is useful for reacting to changes made in a website quickly, but it’s not that important for this specific use case. Instead, the goal here is to detect changes across every social media account that posts about Kpop, but in a sustainable way (aka slowly). So we want the same volume of requests, just spread out thinner over time and across significantly more pages.

 Priority

 Something important to recognize is that not all pages are created the same. There are accounts that consistently post good content, usually ran by companies behind groups; these should be checked frequently. There are also accounts that post infrequently, or post second-hand content (like with fan accounts); these should be checked less frequently. This requirement calls for implementing a priority system where different pages can automatically adjust their priority depending on the post frequency, on top of a manual multiplier if needed.

 Originally, I went for a dynamic priority system where tasks would get re-scheduled based on how many images they found as they moved up and down on a discrete priority scale. The highest priority scrapes would get scheduled for 4 hours after a request, and the lowest would get two weeks. This worked well for changing scrape frequencies dynamically, but it created an issue with scrape distribution. It allowed multiple scheduled scrapes to fall within the same hour and get the bot rate limited by accident. This is unlikely with only a few endpoints but would become more and more likely as the app scaled, and that’s a problem.

 I played around with some numbers and re-scheduling details, but the problem here was scheduling itself. I had to completely re-think how I approached it. In the end, I went with something similar to a token bucket, often used for rate limiting algorithms. Every page/account has a bucket of tokens and a rate in which it gains new tokens (called priority ). At the start of every day, their tokens get updated by their priority. If a page has more than one token, they get scheduled one scrape per token sometime within the next 24 hours, and that amount gets subtracted from their tokens.

 At the start of the day, a database that looks like the following will get these tasks scheduled.

 Provider Type Priority Tokens
 A 1.0 2.0 
 B 1.0 2.2 
 C 0.5 2.5 
 

 Example list of scheduled scrapes

 After every successful scrape, the next priority of the provider is calculated by getting the latest scrape result + the past 30 scrapes, and summing up the amount of images it found per check, weighed by how long ago the scrape was from the current day.

 Example list of scheduled scrapes

 This makes sure that an inactive account that starts to post images gets a higher bump in priority, and an active account that stops posting images gets knocked down an equivalent amount.

 When the tokens for a provider is under 1.0, it gets compounded by its priority every day even if it doesn’t get scraped. A priority can’t go below 0.07 (meaning a scrape once every 2 weeks) and can’t go above 1.75 (3 times every 2 days) .

 Distribution

 Once all the tasks that need to run within the next 24 hours are identified, they need to be scheduled for an exact time with an even distribution. We can’t just blindly start going through all pages that need to be sent requests immediately, otherwise we run into the exact same rate limiting problem from before.

 That day’s worth of pending requests need to be grouped by their provider type —think Instagram, Twitter, etc.— and scheduled in an evenly-spaced manner using a linear interpolation between NOW() and NOW() + 24 hours

 /* scheduling algorithms be like */
 .pending-requests {
 display: flex;
 justify-content: space-evenly;
 }

 This method of scheduling ensures that pages on different domains that don’t share rate limits are processed concurrently, while pages that do share a rate limit get processed sequentially.

 With what we know now, let’s take another look at the scheduled tasks from last time and imagine each letter is an individual Twitter page, each scheduled to get processed at an even interval after the other.

 Can you spot the inefficiency here?

 We’re scheduling the same page to be scraped multiple times right after each other. We’re making sure to space the scrapes within the same provider evenly but leaving uneven spaces between repeated scrapes.

 The correct order of the tasks should be more like this:

 For this reason, Jiu tries its best to maximize the spacing between requests going to the same endpoint so that there’s enough time for content to update.

 “Tries its best” because sadly the solution to this problem is not simple or deterministic.

 Authentication

 Another unfortunate problem that must be dealt with is that not every single website lets us view images without being logged in. To be able to do this, the login flow of the site needs to be reverse engineered in order to grab the necessary images. This also sometimes comes with the downside of needing to fake a user agent (or possibly other fingerprints).

 Now, I know that reverse engineering and using a private API for fetching data and going outside the boundaries of what a site thinks is acceptable kind of contradicts wanting to be nice to websites and not spamming them. But using official APIs can get us rate-limited quickly, even with large gaps in requests. I’m going out of my way to be careful with the sites I’m scraping, so I deserve this. Besides, I know what’s best for Twitter more than Twitter knows what’s best for herself, and I promise I will treat her like the queen she is if she just gives me a chance.

 context

 Architecture and Code

 For the more technical part, instead of copy-pasting functions from the codebase, I’m going to walk you through building a mini-scraper logic so you can maybe get a feel for what it would’ve been like to come up with some of the abstractions I haven’t mentioned yet on your own.

 I wrote this program in Rust, but I’m going to use Typescript examples because it’s probably way more accessible for my audience. Feel free to look at the Rust source if you’re interested In case you’re interested, do not feel free to look through my Rust code. Please just keep reading the examples. I will take legal action against anyone I catch browsing through the repo.

 Let’s start with the most fundamental idea. A requirement for this project is that we need a solution that allows fetching different forms of media from many sites. One assumption we can make is that every image/video should conform to one shared interface no matter its source. At this point, one could go for a simple class that turns a URL into an array of images.

 Typescript

 class Provider {
 constructor(public siteName: string) {}
 
 
 async scrape(url: string): Promise<Image[]> {
 const images = await this.request(url)
 return this.extractAllImages(images)
 }
 }
 Try to use your imagination for the functions I haven't defined

 We should now account for the possibility that this is the first time we’re gathering data from that page and haven’t seen any images. We need to add a pagination concept so our scraper can go through the page history and grab everything.

 Typescript

 class Provider {
 async scrape(url: string): Promise<Image[]> {
 let target = url
 const images: Image[] = []
 while (true) {
 const response = await this.request(target)
 const pagination = this.getPagination(response)
 images.push(this.extractAllImages(response))
 if (pagination.hasNextPage) {
 // turns ?page=1 into ?page=2
 target = this.getNextPage(target)
 } else {
 break
 }
 }
 return images
 }
 }

 Now we have a way of getting all images from a website. Except, with this approach, we always paginate through the full resource for every check. We should be able to stop when we find an image we’ve seen before and only “gather” the ones we haven’t seen in a previous run. This can be solved by passing an array of previously seen images to the function to test against.

 Typescript

 class Provider {
 async scrape(url: string, seen: string[]): Promise<Image[]> {
 let target = url
 const images: Image[] = []
 while (true) {
 const response = await this.request(target)
 const pagination = this.getPagination(response)
 const found = this.extractAllImages(response)
 const unseen = this.allUnseenImages(found, seen)
 images.push(unseen)
 if (unseen.length !== found.length) {
 // some of the images we scraped were seen before
 break
 }
 if (pagination.hasNextPage) {
 // turns ?page=1 into ?page=2
 target = this.getNextPage(target)
 } else {
 break
 }
 }
 return images
 }
 }

 Great, it looks like this is working fine for one provider like Twitter, but it’s still not scalable. The entire point of the project was to implement a solution that can have multiple sites plugged into it, and every provider is going to require a different way of fetching and processing data. There’s a Provider class here, but it’s not extensible in any way. We can’t configure the parts that need to be swapped out for different sites because everything gets done in a single function. Perhaps OOP is the right fix? Maybe all we need to do is extend Provider and reuse the logic somehow.

 Typescript

 class WeverseProvider extends Provider {
 override async scrape(url: string, seen: string[]): Promise<Image[]> {
 // ???
 }
 }

 But what exactly goes in this function now, literally everything we had in the previous class? Aside from this.request, every single method on here is custom (mostly) per-provider and can’t just be overridden. It would be nice to have a way to almost… “template” a class method so that the if checks —like the part where we look to see if we’ve reached the end of everything we’ve scraped— and break statements stay, but everything else is customized.

 As you may have guessed, there’s a way to do that. Well, not with templating, god no. That was a terrible call; what are you crazy? The issue is Provider is trying to do way too much here, and it’s getting in our way. As a rule of thumb, when you run into a problem where you need to copy-paste logic , creating reusable functions is usually the correct approach. However, when you run into problems where you need to copy-paste control flow , the solution is getting rid of tight coupling.

 I mean, why does this class need to care at all about previously seen images? A provider’s scrape function only needs to scrape, and that’s it, no? It’s time to turn our single class method into a producer/consumer pair. Ideally, we need a producer that knows nothing about the outside world. And a consumer that only knows about the promise that a provider makes about what it’s capable of doing through an interface.

 Let’s call the new class that will decouple that logic Scraper, and clean up the function names to reflect the new responsibilities of the two.

 Typescript

 class Scraper {
 async scrape(provider: Provider, url: string): Promise<Image[]> {
 const seen = await this.getLatestResultsFor(provider)
 const found = await provider.process(url)
 return provider.allUnseenImages(found, seen)
 }
 }
 
 
 class Provider {
 async process(url: string): Promise<Image[]> {
 const response = await this.request(target)
 return this.extractAllImages(response)
 }
 }

 This works well for checking images. We can do something similar to re-implement pagination. Except, this time, the while loop can’t be inside the provider; it has to be inside Scraper. The problem with that is the scraper doesn’t understand what pagination means. Most sites (like Weverse) use page-based pagination, where you increase page numbers by 1 to go to the next page. But there are also sites (like Pinterest & Twitter) that use a cursor-based pagination technique, where you pass the server some opaque identifier from request one that represents the next page for request two.

 Scraper shouldn’t have to know anything about how a provider’s pagination works, only that pagination is a thing that exists and needs to be passed back to a provider. Provider needs to accept and return information about pagination so it can decide what page to scrape by itself.

 Typescript

 class Scraper {
 async scrape(provider: Provider, url: string): Promise<Image[]> {
 const seen = await this.getLatestResultsFor(provider)
 // undefined in the initial iteration
 let pagination: Pagination | undefined
 let images: Image[] = []
 while (true) {
 const result = await provider.process(url, pagination)
 const unseen = provider.allUnseenImages(found, seen)
 images.push(unseen)
 if (unseen.length !== result.images.length) {
 break
 }
 if (result.nextPage) {
 pagination = result.nextPage
 } else {
 break
 }
 }
 return images
 }
 }
 
 
 type Pagination =
 | { type: "cursor"; id: string }
 | { type: "page"; index: number }
 
 
 type ProviderResult = { images: Image[]; nextPage?: Pagination }
 
 
 class Provider {
 async process(url: string, pagination?: Pagination): Promise<ProviderResult> {
 const target = this.buildUrlWithPagination(url, pagination)
 const response = await this.request(target)
 const images = this.extractAllImages(response)
 const nextPage = this.extractNextPage(response)
 
 
 return { images, nextPage }
 }
 }

 This approach works perfectly! Though, the observant functional programmers may have noticed that the thing we’re doing here looks awfully similar to array.reduce . We’re going through a loop and feeding the result of a function call in one iteration (in our case, pagination) back into the function called in the next iteration, much like reduce. It could be useful to re-package iteration logic with a functional helper if it makes things easier to reason about.

 As a refresher, reduce (also referred to as fold in some languages) works by doing something like the following, where it takes an A’ , an array of values, and produces E’ as an output.

 But this doesn’t quite work for us. We have an initial seed value of a URL, yes. But we don’t have an array of things we can go over. Instead we want to create that array of things (image URLs) as a result of repeatedly applying a function (scrape) on something (pagination) . Almost like a fold, but the other way around. Thankfully the thing we’re looking for here is a well-known concept like fold and is unsurprisingly called unfold.

 We start with an initial value A’ and produce two outputs, one as the value that will be “pushed” into an array and the other which will get passed as an argument into the next iteration.

 Typescript

 class Scraper {
 scrape(provider: Provider, url: string): AsyncGenerator<Image[]> {
 // initial value is `undefined` beceause
 // the first scrape has no pagination
 return unfold(undefined, async pagination => {
 const result = await provider.process(url, pagination)
 if (result.nextPage) {
 return next(result.images, result.nextPage)
 } else {
 return stop(result.images)
 }
 })
 }
 }

 The hypothetical stop function here lets us signal the unfold to stop early if we’ve exhausted all resources. Typically, this would be done by returning an Option data type in a language like Rust, but it’s hard to represent that as clearly in JS.

 This approach to iteration lets us decouple logic even further. Unlike fold which reduces an iterable into a single value synchronously, unfold generates an array of values. If the operation that is being run to generate values is asynchronous, that array can also be represented as an infinite stream of values that a consumer can stop generating at any point. Async iterators are great for doing this in Javascript.

 Typescript

 const seen = await getLatestResultsFor(provider)
 for await (const images of scraper.scrape(provider, url)) {
 const unseen = scraper.getUnseenImages(images, seen)
 if (unseen.length !== images.length) {
 // scraping stops here because the generator
 // is no longer being consumed
 break
 }
 await processImages(unseen)
 // delay the next iteration of the scrape
 // for 1 second
 await sleep(1000)
 }

 Since we’re using generators now, the scraper doesn’t need to know the conditions to look for to stop doing its job anymore; it stops when we tell it to stop inside the consumer (or runs out of pages). This means that if we wanted to, we could conveniently get the scraper to keep going even if it finds known images —like when re-fetching all images posted by a specific account— without touching the scraping logic.

 Error handling and other related things can also be done now by enriching the return value of process to include more information about what happened. A provider can accept an error that it itself created in a different function and figure out how to deal with it without actually dealing with it to further decouple the process of handing off errors.

 Typescript

 type ProcessResult =
 | { type: "success"; result: ProviderResult }
 | { type: "error"; response: Response }
 
 
 type ErrorHandle =
 | { type: "tryAgain" }
 | { type: "tryAuth" }
 | { type: "cantContinue"; reason: Error }
 
 
 async function iterate(pagination?: Pagination) {
 const process = await provider.process(url, pagination)
 if (process.error) {
 const decision: ErrorHandle = await provider.onError(process.response)
 switch (decision.type) {
 case "tryAgain":
 // retry logic with an unfold is nothing
 // more than recursion
 return iterate(pagination)
 case "tryAuth":
 await provider.authenticate()
 return iterate(pagination)
 case "cantContinue":
 // handle errors here :)
 return stop([])
 }
 }
 const { images, nextPage } = process.result
 if (nextPage) {
 return next(images, nextPage)
 } else {
 return stop(images)
 }
 }
 
 
 // inside scraper
 return unfold(undefined, iterate)

 Now, adding new providers is simple. It requires no duplication of control flow and not even a class. All that’s needed is some object that satisfies an interface. Notice how even with a function like onError, we are simply taking a response as input and returning a decision as a response, not performing an action. This framework allows the gears of Scraper and Provider to turn together without stepping over each other’s toes. Providers only know how to go, and scrapers only know how to tell providers to go.

 Typescript

 interface Provider {
 process(url: string, pagination?: Pagination): Promise<ProviderResult>
 onError(response: Response): Promise<ErrorHandle>
 // any other functionality that could be customizable per site
 }

 I’ve skipped over how further implementation details of providers, persistence, publishing data, and other shenanigans work in Jiu in the interest of not boring people to death. But hopefully, this helps get a basic overview of implementing a generic scraper and decoupling business logic.

 There are a couple more intriguing things I’ve come across while working on Kiyomi that I want to write about, like perceptual hashing and face recognition, but those will have to wait until I actually have any idea what I’m doing with them.

 Here’s Jiu if you’re interested, and Kiyomi.

 Previous Article

 🐿 Golang is not a good language

 Next Article

 Sheeesh, you just read the last post
For the past year, I’ve been working on a website for Kpop image aggregation and labeling called Kiyomi. I’m interested in imageboards like Safebooru that have an active community of fans who label every anime image to make searching easy for everyone. I love seeing how members come together to create a massive database of carefully curated content, and it’s only possible because of how much diverse content there is for the community to go through. For that reason, one of the challenges in working on Kiyomi has been to make sure that I can get fresh content to users automatically. Thankfully, there’s nothing Kpop idols love more than posting images of themselves online, so there’s an endless source of relevant material to work with. I decided to solve this with what I know best, a scraper that I named Jiu after one of the top 7 Dreamcatcher members. In case you’re not familiar, scraping is a way of programmatically going to a website and extracting some kind of information from it. In my case, this information is image URLs and their associated metadata. Normally this kind of image retrieval task is trivial to work with if all you need to check is a handful of pages on a single site. It’s nothing more than a for loop, some database lookups, and a bunch of HTTP requests. But I was planning on scraping many websites, which meant the solution had to be easily extendable to support new sites. I also had to consider another thing, which is that I don’t wanna like… be an asshole? I don’t like it when devs carelessly hammer my website with requests. So given the nature of what I’m doing, it would be pretty hypocritical of me to try to do the same for others. This meant that while working on this service, I first had to exhaust all of the “good guy” workarounds before I moved on to less-friendly measures like rotating IPs and pretending to be a real user with puppeteer. Also, proxies that work well are expensive, and I'm cheap This project was originally meant to be a part of Kiyomi, but I figured the problem of finding images online was generic enough that it could be its own standalone thing. Solving some of these problems has been incredibly fun and a huge learning process for me, so I’d like to document at least some of the ways I dealt with them.  The problem space  This isn’t crucial to understanding the problem, but I want to share the infrastructure Jiu sits on top of because it could provide important context. Also, I spent a lot of time working on this and drawing the diagram for it, so I’m going to need someone to look at it.   
 
 
 
 
 
 
 
 
  Kiyomi architecture map   New images Jiu finds on supported sites are stored in a database isolated from every other service and get published to RabbitMQ for consumers to do whatever with. Jiu also runs its own webserver to allow dynamically adding and removing new providers, querying the stats of scrapes, and displaying its schedule/history. 
 
 
 
 
 
 
 
 
  The requirements for building this scraper are:  Creating a “plugin” system capable of supporting multiple social media sites Turning polling into an event-based system with webhooks/AMQP Automatically logging in to sites that require authentication Being able to handle failure from APIs gracefully Finding a high volume of images from different sites Being nice and not sending too many requests to any specific site (if possible)    I am scraping respectfully  The easiest way to prevent spamming a server with requests is to just not send it many requests, duh. This is an obvious one, but it’s something that works surprisingly well for what we’re building. Because the point of Jiu is to eventually detect all images posted on social media and not as soon as possible. We can use this to our advantage when deciding the architecture of the scraper. A common way of implementing these scrapers is something commonly referred to as a monitor. A monitor takes one or a handful of critical pages and bombards it with requests to detect a change as soon as possible. 
 
 
 
 
 
 
 
 
  This is useful for reacting to changes made in a website quickly, but it’s not that important for this specific use case. Instead, the goal here is to detect changes across every social media account that posts about Kpop, but in a sustainable way (aka slowly). So we want the same volume of requests, just spread out thinner over time and across significantly more pages. 
 
 
 
 
 
 
 
 
   Priority  Something important to recognize is that not all pages are created the same. There are accounts that consistently post good content, usually ran by companies behind groups; these should be checked frequently. There are also accounts that post infrequently, or post second-hand content (like with fan accounts); these should be checked less frequently. This requirement calls for implementing a priority system where different pages can automatically adjust their priority depending on the post frequency, on top of a manual multiplier if needed. Originally, I went for a dynamic priority system where tasks would get re-scheduled based on how many images they found as they moved up and down on a discrete priority scale. The highest priority scrapes would get scheduled for 4 hours after a request, and the lowest would get two weeks. This worked well for changing scrape frequencies dynamically, but it created an issue with scrape distribution. It allowed multiple scheduled scrapes to fall within the same hour and get the bot rate limited by accident. This is unlikely with only a few endpoints but would become more and more likely as the app scaled, and that’s a problem. I played around with some numbers and re-scheduling details, but the problem here was scheduling itself. I had to completely re-think how I approached it. In the end, I went with something similar to a token bucket, often used for rate limiting algorithms. Every page/account has a bucket of tokens and a rate in which it gains new tokens (called priority). At the start of every day, their tokens get updated by their priority. If a page has more than one token, they get scheduled one scrape per token sometime within the next 24 hours, and that amount gets subtracted from their tokens. At the start of the day, a database that looks like the following will get these tasks scheduled.     Provider Type Priority Tokens     A 1.0 2.0   B 1.0 2.2   C 0.5 2.5       
 
 
 
 
 
 
 
 
  Example list of scheduled scrapes   After every successful scrape, the next priority of the provider is calculated by getting the latest scrape result + the past 30 scrapes, and summing up the amount of images it found per check, weighed by how long ago the scrape was from the current day.   
 
 
 
 
 
 
 
 
  Example list of scheduled scrapes   This makes sure that an inactive account that starts to post images gets a higher bump in priority, and an active account that stops posting images gets knocked down an equivalent amount. When the tokens for a provider is under 1.0, it gets compounded by its priority every day even if it doesn’t get scraped. A priority can’t go below 0.07 (meaning a scrape once every 2 weeks) and can’t go above 1.75 (3 times every 2 days).  Distribution  Once all the tasks that need to run within the next 24 hours are identified, they need to be scheduled for an exact time with an even distribution. We can’t just blindly start going through all pages that need to be sent requests immediately, otherwise we run into the exact same rate limiting problem from before. That day’s worth of pending requests need to be grouped by their provider type —think Instagram, Twitter, etc.— and scheduled in an evenly-spaced manner using a linear interpolation between NOW() and NOW() + 24 hours 
 
 
 
 
 
 
 
 
     .pending-requests {  display: flex;  justify-content: space-evenly; }    This method of scheduling ensures that pages on different domains that don’t share rate limits are processed concurrently, while pages that do share a rate limit get processed sequentially. With what we know now, let’s take another look at the scheduled tasks from last time and imagine each letter is an individual Twitter page, each scheduled to get processed at an even interval after the other. 
 
 
 
 
 
 
 
 
  Can you spot the inefficiency here? We’re scheduling the same page to be scraped multiple times right after each other. We’re making sure to space the scrapes within the same provider evenly but leaving uneven spaces between repeated scrapes. The correct order of the tasks should be more like this: 
 
 
 
 
 
 
 
 
  For this reason, Jiu tries its best to maximize the spacing between requests going to the same endpoint so that there’s enough time for content to update. “Tries its best” because sadly the solution to this problem is not simple or deterministic.  Authentication  Another unfortunate problem that must be dealt with is that not every single website lets us view images without being logged in. To be able to do this, the login flow of the site needs to be reverse engineered in order to grab the necessary images. This also sometimes comes with the downside of needing to fake a user agent (or possibly other fingerprints). Now, I know that reverse engineering and using a private API for fetching data and going outside the boundaries of what a site thinks is acceptable kind of contradicts wanting to be nice to websites and not spamming them. But using official APIs can get us rate-limited quickly, even with large gaps in requests. I’m going out of my way to be careful with the sites I’m scraping, so I deserve this. Besides, I know what’s best for Twitter more than Twitter knows what’s best for herself, and I promise I will treat her like the queen she is if she just gives me a chance.  context   Architecture and Code  For the more technical part, instead of copy-pasting functions from the codebase, I’m going to walk you through building a mini-scraper logic so you can maybe get a feel for what it would’ve been like to come up with some of the abstractions I haven’t mentioned yet on your own. I wrote this program in Rust, but I’m going to use Typescript examples because it’s probably way more accessible for my audience. Feel free to look at the Rust source if you’re interested In case you’re interested, do not feel free to look through my Rust code. Please just keep reading the examples. I will take legal action against anyone I catch browsing through the repo. Let’s start with the most fundamental idea. A requirement for this project is that we need a solution that allows fetching different forms of media from many sites. One assumption we can make is that every image/video should conform to one shared interface no matter its source. At this point, one could go for a simple class that turns a URL into an array of images.    Typescript     class Provider {  constructor(public siteName: string) {}  async scrape(url: string): Promise&lt;Image[]&gt; {  const images = await this.request(url)  return this.extractAllImages(images)  } }    Try to use your imagination for the functions I haven't defined We should now account for the possibility that this is the first time we’re gathering data from that page and haven’t seen any images. We need to add a pagination concept so our scraper can go through the page history and grab everything.    Typescript     class Provider {  async scrape(url: string): Promise&lt;Image[]&gt; {  let target = url  const images: Image[] = []  while (true) {  const response = await this.request(target)  const pagination = this.getPagination(response)  images.push(this.extractAllImages(response))  if (pagination.hasNextPage) {  target = this.getNextPage(target)  } else {  break  }  }  return images  } }    Now we have a way of getting all images from a website. Except, with this approach, we always paginate through the full resource for every check. We should be able to stop when we find an image we’ve seen before and only “gather” the ones we haven’t seen in a previous run. This can be solved by passing an array of previously seen images to the function to test against.    Typescript     class Provider {  async scrape(url: string, seen: string[]): Promise&lt;Image[]&gt; {  let target = url  const images: Image[] = []  while (true) {  const response = await this.request(target)  const pagination = this.getPagination(response)  const found = this.extractAllImages(response)  const unseen = this.allUnseenImages(found, seen)  images.push(unseen)  if (unseen.length !== found.length) {  break  }  if (pagination.hasNextPage) {  target = this.getNextPage(target)  } else {  break  }  }  return images  } }    Great, it looks like this is working fine for one provider like Twitter, but it’s still not scalable. The entire point of the project was to implement a solution that can have multiple sites plugged into it, and every provider is going to require a different way of fetching and processing data. There’s a Provider class here, but it’s not extensible in any way. We can’t configure the parts that need to be swapped out for different sites because everything gets done in a single function. Perhaps OOP is the right fix? Maybe all we need to do is extend Provider and reuse the logic somehow.    Typescript     class WeverseProvider extends Provider {  override async scrape(url: string, seen: string[]): Promise&lt;Image[]&gt; {  } }    But what exactly goes in this function now, literally everything we had in the previous class? Aside from this.request, every single method on here is custom (mostly) per-provider and can’t just be overridden. It would be nice to have a way to almost… “template” a class method so that the if checks —like the part where we look to see if we’ve reached the end of everything we’ve scraped— and break statements stay, but everything else is customized. As you may have guessed, there’s a way to do that. Well, not with templating, god no. That was a terrible call; what are you crazy? The issue is Provider is trying to do way too much here, and it’s getting in our way. As a rule of thumb, when you run into a problem where you need to copy-paste logic, creating reusable functions is usually the correct approach. However, when you run into problems where you need to copy-paste control flow, the solution is getting rid of tight coupling. I mean, why does this class need to care at all about previously seen images? A provider’s scrape function only needs to scrape, and that’s it, no? It’s time to turn our single class method into a producer/consumer pair. Ideally, we need a producer that knows nothing about the outside world. And a consumer that only knows about the promise that a provider makes about what it’s capable of doing through an interface. Let’s call the new class that will decouple that logic Scraper, and clean up the function names to reflect the new responsibilities of the two.    Typescript     class Scraper {  async scrape(provider: Provider, url: string): Promise&lt;Image[]&gt; {  const seen = await this.getLatestResultsFor(provider)  const found = await provider.process(url)  return provider.allUnseenImages(found, seen)  } } class Provider {  async process(url: string): Promise&lt;Image[]&gt; {  const response = await this.request(target)  return this.extractAllImages(response)  } }    This works well for checking images. We can do something similar to re-implement pagination. Except, this time, the while loop can’t be inside the provider; it has to be inside Scraper. The problem with that is the scraper doesn’t understand what pagination means. Most sites (like Weverse) use page-based pagination, where you increase page numbers by 1 to go to the next page. But there are also sites (like Pinterest &amp; Twitter) that use a cursor-based pagination technique, where you pass the server some opaque identifier from request one that represents the next page for request two.  Scraper shouldn’t have to know anything about how a provider’s pagination works, only that pagination is a thing that exists and needs to be passed back to a provider. Provider needs to accept and return information about pagination so it can decide what page to scrape by itself.    Typescript     class Scraper {  async scrape(provider: Provider, url: string): Promise&lt;Image[]&gt; {  const seen = await this.getLatestResultsFor(provider)  let pagination: Pagination | undefined  let images: Image[] = []  while (true) {  const result = await provider.process(url, pagination)  const unseen = provider.allUnseenImages(found, seen)  images.push(unseen)  if (unseen.length !== result.images.length) {  break  }  if (result.nextPage) {  pagination = result.nextPage  } else {  break  }  }  return images  } } type Pagination =  | { type: "cursor"; id: string }  | { type: "page"; index: number } type ProviderResult = { images: Image[]; nextPage?: Pagination } class Provider {  async process(url: string, pagination?: Pagination): Promise&lt;ProviderResult&gt; {  const target = this.buildUrlWithPagination(url, pagination)  const response = await this.request(target)  const images = this.extractAllImages(response)  const nextPage = this.extractNextPage(response)  return { images, nextPage }  } }    This approach works perfectly! Though, the observant functional programmers may have noticed that the thing we’re doing here looks awfully similar to array.reduce. We’re going through a loop and feeding the result of a function call in one iteration (in our case, pagination) back into the function called in the next iteration, much like reduce. It could be useful to re-package iteration logic with a functional helper if it makes things easier to reason about. As a refresher, reduce (also referred to as fold in some languages) works by doing something like the following, where it takes an A’, an array of values, and produces E’ as an output. 
 
 
 
 
 
 
 
 
  But this doesn’t quite work for us. We have an initial seed value of a URL, yes. But we don’t have an array of things we can go over. Instead we want to create that array of things (image URLs) as a result of repeatedly applying a function (scrape) on something (pagination). Almost like a fold, but the other way around. Thankfully the thing we’re looking for here is a well-known concept like fold and is unsurprisingly called unfold. 
 
 
 
 
 
 
 
 
  We start with an initial value A’ and produce two outputs, one as the value that will be “pushed” into an array and the other which will get passed as an argument into the next iteration.    Typescript     class Scraper {  scrape(provider: Provider, url: string): AsyncGenerator&lt;Image[]&gt; {  return unfold(undefined, async pagination =&gt; {  const result = await provider.process(url, pagination)  if (result.nextPage) {  return next(result.images, result.nextPage)  } else {  return stop(result.images)  }  })  } }    The hypothetical stop function here lets us signal the unfold to stop early if we’ve exhausted all resources. Typically, this would be done by returning an Option data type in a language like Rust, but it’s hard to represent that as clearly in JS. This approach to iteration lets us decouple logic even further. Unlike fold which reduces an iterable into a single value synchronously, unfold generates an array of values. If the operation that is being run to generate values is asynchronous, that array can also be represented as an infinite stream of values that a consumer can stop generating at any point. Async iterators are great for doing this in Javascript.    Typescript     const seen = await getLatestResultsFor(provider) for await (const images of scraper.scrape(provider, url)) {  const unseen = scraper.getUnseenImages(images, seen)  if (unseen.length !== images.length) {  break  }  await processImages(unseen)  await sleep(1000) }    Since we’re using generators now, the scraper doesn’t need to know the conditions to look for to stop doing its job anymore; it stops when we tell it to stop inside the consumer (or runs out of pages). This means that if we wanted to, we could conveniently get the scraper to keep going even if it finds known images —like when re-fetching all images posted by a specific account— without touching the scraping logic. Error handling and other related things can also be done now by enriching the return value of process to include more information about what happened. A provider can accept an error that it itself created in a different function and figure out how to deal with it without actually dealing with it to further decouple the process of handing off errors.    Typescript     type ProcessResult =  | { type: "success"; result: ProviderResult }  | { type: "error"; response: Response } type ErrorHandle =  | { type: "tryAgain" }  | { type: "tryAuth" }  | { type: "cantContinue"; reason: Error } async function iterate(pagination?: Pagination) {  const process = await provider.process(url, pagination)  if (process.error) {  const decision: ErrorHandle = await provider.onError(process.response)  switch (decision.type) {  case "tryAgain":  return iterate(pagination)  case "tryAuth":  await provider.authenticate()  return iterate(pagination)  case "cantContinue":  return stop([])  }  }  const { images, nextPage } = process.result  if (nextPage) {  return next(images, nextPage)  } else {  return stop(images)  } } return unfold(undefined, iterate)    Now, adding new providers is simple. It requires no duplication of control flow and not even a class. All that’s needed is some object that satisfies an interface. Notice how even with a function like onError, we are simply taking a response as input and returning a decision as a response, not performing an action. This framework allows the gears of Scraper and Provider to turn together without stepping over each other’s toes. Providers only know how to go, and scrapers only know how to tell providers to go.    Typescript     interface Provider {  process(url: string, pagination?: Pagination): Promise&lt;ProviderResult&gt;  onError(response: Response): Promise&lt;ErrorHandle&gt; }     I’ve skipped over how further implementation details of providers, persistence, publishing data, and other shenanigans work in Jiu in the interest of not boring people to death. But hopefully, this helps get a basic overview of implementing a generic scraper and decoupling business logic. There are a couple more intriguing things I’ve come across while working on Kiyomi that I want to write about, like perceptual hashing and face recognition, but those will have to wait until I actually have any idea what I’m doing with them. Here’s Jiu if you’re interested, and Kiyomi.
🔭 Building a scalable scraper
https://xetera.dev/building-a-scalable-scraper/
https://en.wikipedia.org/wiki/Web_crawler
Web crawler - Wikipedia
2001-04-27
Authority control National libraries Germany
<div><div><p>This article is about the internet bot. For the search engine, see <a href="/wiki/WebCrawler">WebCrawler</a>. "Web spider" redirects here. Not to be confused with <a href="/wiki/Spider_web">Spider web</a>. "Spiderbot" redirects here. For the video game, see <a href="/wiki/Arac_(video_game)">Arac (video game)</a>.</p>
<div><p><a href="/wiki/File:WebCrawlerArchitecture.svg"></a></p><div><p>Architecture of a Web crawler</p></div></div>
<p>A <b>Web crawler</b>, sometimes called a <b>spider</b> or <b>spiderbot</b> and often shortened to <b>crawler</b>, is an <a href="/wiki/Internet_bot">Internet bot</a> that systematically browses the <a href="/wiki/World_Wide_Web">World Wide Web</a> and that is typically operated by search engines for the purpose of <a href="/wiki/Web_indexing">Web indexing</a> (<i>web spidering</i>).<sup><a href="#cite_note-1">[1]</a></sup>
</p><p>Web <a href="/wiki/Search_engine">search engines</a> and some other <a href="/wiki/Website">websites</a> use Web crawling or spidering <a href="/wiki/Software">software</a> to update their <a href="/wiki/Web_content">web content</a> or indices of other sites' web content. Web crawlers copy pages for processing by a search engine, which <a href="/wiki/Index_(search_engine)">indexes</a> the downloaded pages so that users can search more efficiently.
</p><p>Crawlers consume resources on visited systems and often visit sites unprompted. Issues of schedule, load, and "politeness" come into play when large collections of pages are accessed. Mechanisms exist for public sites not wishing to be crawled to make this known to the crawling agent. For example, including a <code><a href="/wiki/Robots.txt">robots.txt</a></code> file can request <a href="/wiki/Software_agent">bots</a> to index only parts of a website, or nothing at all.
</p><p>The number of Internet pages is extremely large; even the largest crawlers fall short of making a complete index. For this reason, search engines struggled to give relevant search results in the early years of the World Wide Web, before 2000. Today, relevant results are given almost instantly.
</p><p>Crawlers can validate <a href="/wiki/Hyperlink">hyperlinks</a> and <a href="/wiki/HTML">HTML</a> code. They can also be used for <a href="/wiki/Web_scraping">web scraping</a> and <a href="/wiki/Data-driven_programming">data-driven programming</a>.
</p>
<h2>Nomenclature[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=1">edit</a>]</h2>
<p>A web crawler is also known as a <i>spider</i>,<sup><a href="#cite_note-spekta-2">[2]</a></sup> an <i>ant</i>, an <i>automatic indexer</i>,<sup><a href="#cite_note-3">[3]</a></sup> or (in the <a href="/wiki/FOAF_(software)">FOAF</a> software context) a <i>Web scutter</i>.<sup><a href="#cite_note-4">[4]</a></sup>
</p>
<h2>Overview[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=2">edit</a>]</h2>
<p>A Web crawler starts with a list of <a href="/wiki/Uniform_Resource_Locator">URLs</a> to visit. Those first URLs are called the <i>seeds</i>. As the crawler visits these URLs, by communicating with <a href="/wiki/Web_server">web servers</a> that respond to those URLs, it identifies all the <a href="/wiki/Hyperlink">hyperlinks</a> in the retrieved web pages and adds them to the list of URLs to visit, called the <i><a href="/wiki/Crawl_frontier">crawl frontier</a></i>. URLs from the frontier are <a href="/wiki/Recursion">recursively</a> visited according to a set of policies. If the crawler is performing archiving of <a href="/wiki/Website">websites</a> (or <a href="/wiki/Web_archiving">web archiving</a>), it copies and saves the information as it goes. The archives are usually stored in such a way they can be viewed, read and navigated as if they were on the live web, but are preserved as 'snapshots'.<sup><a href="#cite_note-GoogleBooks-9237380-5">[5]</a></sup>
</p><p>The archive is known as the <i>repository</i> and is designed to store and manage the collection of <a href="/wiki/Web_page">web pages</a>. The <a href="/wiki/Repository_(version_control)">repository</a> only stores <a href="/wiki/HTML">HTML</a> pages and these pages are stored as distinct files. A repository is similar to any other system that stores data, like a modern-day database. The only difference is that a repository does not need all the functionality offered by a database system. The repository stores the most recent version of the web page retrieved by the crawler.<sup>[<i><a href="/wiki/Wikipedia:Citation_needed">citation needed</a></i>]</sup>
</p><p>The large volume implies the crawler can only download a limited number of the Web pages within a given time, so it needs to prioritize its downloads. The high rate of change can imply the pages might have already been updated or even deleted.
</p><p>The number of possible URLs crawled being generated by server-side software has also made it difficult for web crawlers to avoid retrieving <a href="/wiki/Duplicate_content">duplicate content</a>. Endless combinations of <a href="/wiki/HTTP">HTTP</a> GET (URL-based) parameters exist, of which only a small selection will actually return unique content. For example, a simple online photo gallery may offer three options to users, as specified through HTTP GET parameters in the URL. If there exist four ways to sort images, three choices of <a href="/wiki/Thumbnail">thumbnail</a> size, two file formats, and an option to disable user-provided content, then the same set of content can be accessed with 48 different URLs, all of which may be linked on the site. This <a href="/wiki/Mathematical_combination">mathematical combination</a> creates a problem for crawlers, as they must sort through endless combinations of relatively minor scripted changes in order to retrieve unique content.
</p><p>As Edwards <i>et al.</i> noted, "Given that the <a href="/wiki/Bandwidth_(computing)">bandwidth</a> for conducting crawls is neither infinite nor free, it is becoming essential to crawl the Web in not only a scalable, but efficient way, if some reasonable measure of quality or freshness is to be maintained."<sup><a href="#cite_note-edwards2001-6">[6]</a></sup> A crawler must carefully choose at each step which pages to visit next.
</p>
<h2>Crawling policy[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=3">edit</a>]</h2>
<p>The behavior of a Web crawler is the outcome of a combination of policies:<sup><a href="#cite_note-7">[7]</a></sup>
</p>
<ul><li>a <i>selection policy</i> which states the pages to download,</li>
<li>a <i>re-visit policy</i> which states when to check for changes to the pages,</li>
<li>a <i>politeness policy</i> that states how to avoid overloading <a href="/wiki/Web_sites">Web sites</a>.</li>
<li>a <i>parallelization policy</i> that states how to coordinate distributed web crawlers.</li></ul>
<h3>Selection policy[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=4">edit</a>]</h3>
<p>Given the current size of the Web, even large search engines cover only a portion of the publicly available part. A 2009 study showed even large-scale <a href="/wiki/Search_engines">search engines</a> index no more than 40–70% of the indexable Web;<sup><a href="#cite_note-8">[8]</a></sup> a previous study by <a href="/wiki/Steve_Lawrence_(computer_scientist)">Steve Lawrence</a> and <a href="/wiki/Lee_Giles">Lee Giles</a> showed that no <a href="/wiki/Search_engine_indexing">search engine indexed</a> more than 16% of the Web in 1999.<sup><a href="#cite_note-9">[9]</a></sup> As a crawler always downloads just a fraction of the <a href="/wiki/Web_pages">Web pages</a>, it is highly desirable for the downloaded fraction to contain the most relevant pages and not just a random sample of the Web.
</p><p>This requires a metric of importance for prioritizing Web pages. The importance of a page is a function of its <a href="/wiki/Intrinsic_and_extrinsic_properties_(philosophy)">intrinsic</a> quality, its popularity in terms of links or visits, and even of its URL (the latter is the case of <a href="/wiki/Vertical_search">vertical search engines</a> restricted to a single <a href="/wiki/Top-level_domain">top-level domain</a>, or search engines restricted to a fixed Web site). Designing a good selection policy has an added difficulty: it must work with partial information, as the complete set of Web pages is not known during crawling.
</p><p>Junghoo Cho <i>et al.</i> made the first study on policies for crawling scheduling. Their data set was a 180,000-pages crawl from the <kbd>stanford.edu</kbd> domain, in which a crawling simulation was done with different strategies.<sup><a href="#cite_note-10">[10]</a></sup> The ordering metrics tested were <a href="/wiki/Breadth-first_search">breadth-first</a>, <a href="/wiki/Backlink">backlink</a> count and partial <a href="/wiki/PageRank">PageRank</a> calculations. One of the conclusions was that if the crawler wants to download pages with high Pagerank early during the crawling process, then the partial Pagerank strategy is the better, followed by breadth-first and backlink-count. However, these results are for just a single domain. Cho also wrote his PhD dissertation at Stanford on web crawling.<sup><a href="#cite_note-11">[11]</a></sup>
</p><p>Najork and Wiener performed an actual crawl on 328 million pages, using breadth-first ordering.<sup><a href="#cite_note-12">[12]</a></sup> They found that a breadth-first crawl captures pages with high Pagerank early in the crawl (but they did not compare this strategy against other strategies). The explanation given by the authors for this result is that "the most important pages have many links to them from numerous hosts, and those links will be found early, regardless of on which host or page the crawl originates."
</p><p>Abiteboul designed a crawling strategy based on an <a href="/wiki/Algorithm">algorithm</a> called OPIC (On-line Page Importance Computation).<sup><a href="#cite_note-13">[13]</a></sup> In OPIC, each page is given an initial sum of "cash" that is distributed equally among the pages it points to. It is similar to a PageRank computation, but it is faster and is only done in one step. An OPIC-driven crawler downloads first the pages in the crawling frontier with higher amounts of "cash". Experiments were carried in a 100,000-pages synthetic graph with a power-law distribution of in-links. However, there was no comparison with other strategies nor experiments in the real Web.
</p><p>Boldi <i>et al.</i> used simulation on subsets of the Web of 40 million pages from the <kbd>.it</kbd> domain and 100 million pages from the WebBase crawl, testing breadth-first against depth-first, random ordering and an omniscient strategy. The comparison was based on how well PageRank computed on a partial crawl approximates the true PageRank value. Surprisingly, some visits that accumulate PageRank very quickly (most notably, breadth-first and the omniscient visit) provide very poor progressive approximations.<sup><a href="#cite_note-14">[14]</a></sup><sup><a href="#cite_note-15">[15]</a></sup>
</p><p>Baeza-Yates <i>et al.</i> used simulation on two subsets of the Web of 3 million pages from the <kbd>.gr</kbd> and <kbd>.cl</kbd> domain, testing several crawling strategies.<sup><a href="#cite_note-baeza2005-16">[16]</a></sup> They showed that both the OPIC strategy and a strategy that uses the length of the per-site queues are better than <a href="/wiki/Breadth-first_search">breadth-first</a> crawling, and that it is also very effective to use a previous crawl, when it is available, to guide the current one.
</p><p>Daneshpajouh <i>et al.</i> designed a community based algorithm for discovering good seeds.<sup><a href="#cite_note-17">[17]</a></sup> Their method crawls web pages with high PageRank from different communities in less iteration in comparison with crawl starting from random seeds. One can extract good seed from a previously-crawled-Web graph using this new method. Using these seeds, a new crawl can be very effective.
</p>
<h4>Restricting followed links[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=5">edit</a>]</h4>
<p>A crawler may only want to seek out HTML pages and avoid all other <a href="/wiki/Internet_media_type">MIME types</a>. In order to request only HTML resources, a crawler may make an HTTP HEAD request to determine a Web resource's MIME type before requesting the entire resource with a GET request. To avoid making numerous HEAD requests, a crawler may examine the URL and only request a resource if the URL ends with certain characters such as .html, .htm, .asp, .aspx, .php, .jsp, .jspx or a slash. This strategy may cause numerous HTML Web resources to be unintentionally skipped.
</p><p>Some crawlers may also avoid requesting any resources that have a <a href="/wiki/Query_string">"?"</a> in them (are dynamically produced) in order to avoid <a href="/wiki/Spider_trap">spider traps</a> that may cause the crawler to download an infinite number of URLs from a Web site. This strategy is unreliable if the site uses <a href="/wiki/URL_rewriting">URL rewriting</a> to simplify its URLs.
</p>
<h4>URL normalization[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=6">edit</a>]</h4>
<p>Crawlers usually perform some type of <a href="/wiki/URL_normalization">URL normalization</a> in order to avoid crawling the same resource more than once. The term <i>URL normalization</i>, also called <i>URL canonicalization</i>, refers to the process of modifying and standardizing a URL in a consistent manner. There are several types of normalization that may be performed including conversion of URLs to lowercase, removal of "." and ".." segments, and adding trailing slashes to the non-empty path component.<sup><a href="#cite_note-18">[18]</a></sup>
</p>
<h4>Path-ascending crawling[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=7">edit</a>]</h4>
<p>Some crawlers intend to download/upload as many resources as possible from a particular web site. So <i>path-ascending crawler</i> was introduced that would ascend to every path in each URL that it intends to crawl.<sup><a href="#cite_note-19">[19]</a></sup> For example, when given a seed URL of http://llama.org/hamster/monkey/page.html, it will attempt to crawl /hamster/monkey/, /hamster/, and /. Cothey found that a path-ascending crawler was very effective in finding isolated resources, or resources for which no inbound link would have been found in regular crawling.
</p>
<h4>Focused crawling[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=8">edit</a>]</h4>
<p>The importance of a page for a crawler can also be expressed as a function of the similarity of a page to a given query. Web crawlers that attempt to download pages that are similar to each other are called <b>focused crawler</b> or <b>topical crawlers</b>. The concepts of topical and focused crawling were first introduced by <a href="/wiki/Filippo_Menczer">Filippo Menczer</a><sup><a href="#cite_note-20">[20]</a></sup><sup><a href="#cite_note-21">[21]</a></sup> and by Soumen Chakrabarti <i>et al.</i><sup><a href="#cite_note-22">[22]</a></sup>
</p><p>The main problem in focused crawling is that in the context of a Web crawler, we would like to be able to predict the similarity of the text of a given page to the query before actually downloading the page. A possible predictor is the anchor text of links; this was the approach taken by Pinkerton<sup><a href="#cite_note-pinkerton1994-23">[23]</a></sup> in the first web crawler of the early days of the Web. Diligenti <i>et al.</i><sup><a href="#cite_note-24">[24]</a></sup> propose using the complete content of the pages already visited to infer the similarity between the driving query and the pages that have not been visited yet. The performance of a focused crawling depends mostly on the richness of links in the specific topic being searched, and a focused crawling usually relies on a general Web search engine for providing starting points.
</p>
<h5>Academic focused crawler[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=9">edit</a>]</h5>
<p>An example of the <a href="/wiki/Focused_crawlers">focused crawlers</a> are academic crawlers, which crawls free-access academic related documents, such as the <i>citeseerxbot</i>, which is the crawler of <a href="/wiki/CiteSeer">CiteSeer</a><sup>X</sup> search engine. Other academic search engines are <a href="/wiki/Google_Scholar">Google Scholar</a> and <a href="/wiki/Microsoft_Academic_Search">Microsoft Academic Search</a> etc. Because most academic papers are published in <a href="/wiki/PDF">PDF</a> formats, such kind of crawler is particularly interested in crawling PDF, <a href="/wiki/PostScript">PostScript</a> files, <a href="/wiki/Microsoft_Word">Microsoft Word</a> including their <a href="/wiki/Zipped_file">zipped</a> formats. Because of this, general open-source crawlers, such as <a href="/wiki/Heritrix">Heritrix</a>, must be customized to filter out other <a href="/wiki/MIME_types">MIME types</a>, or a <a href="/wiki/Middleware">middleware</a> is used to extract these documents out and import them to the focused crawl database and repository.<sup><a href="#cite_note-25">[25]</a></sup> Identifying whether these documents are academic or not is challenging and can add a significant overhead to the crawling process, so this is performed as a post crawling process using <a href="/wiki/Machine_learning">machine learning</a> or <a href="/wiki/Regular_expression">regular expression</a> algorithms. These academic documents are usually obtained from home pages of faculties and students or from publication page of research institutes. Because academic documents make up only a small fraction of all web pages, a good seed selection is important in boosting the efficiencies of these web crawlers.<sup><a href="#cite_note-26">[26]</a></sup> Other academic crawlers may download plain text and <a href="/wiki/HTML">HTML</a> files, that contains <a href="/wiki/Metadata">metadata</a> of academic papers, such as titles, papers, and abstracts. This increases the overall number of papers, but a significant fraction may not provide free PDF downloads.
</p>
<h5>Semantic focused crawler[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=10">edit</a>]</h5>
<p>Another type of focused crawlers is semantic focused crawler, which makes use of <a href="/wiki/Domain_ontology">domain ontologies</a> to represent topical maps and link Web pages with relevant ontological concepts for the selection and categorization purposes.<sup><a href="#cite_note-27">[27]</a></sup> In addition, ontologies can be automatically updated in the crawling process. Dong et al.<sup><a href="#cite_note-28">[28]</a></sup> introduced such an ontology-learning-based crawler using a <a href="/wiki/Support-vector_machine">support-vector machine</a> to update the content of ontological concepts when crawling Web pages.
</p>
<h3>Re-visit policy[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=11">edit</a>]</h3>
<p>The Web has a very dynamic nature, and crawling a fraction of the Web can take weeks or months. By the time a Web crawler has finished its crawl, many events could have happened, including creations, updates, and deletions.
</p><p>From the search engine's point of view, there is a cost associated with not detecting an event, and thus having an outdated copy of a resource. The most-used cost functions are freshness and age.<sup><a href="#cite_note-29">[29]</a></sup>
</p><p><b>Freshness</b>: This is a binary measure that indicates whether the local copy is accurate or not. The freshness of a page <i>p</i> in the repository at time <i>t</i> is defined as:
</p>
<dl><dd></dd></dl>
<p><b>Age</b>: This is a measure that indicates how outdated the local copy is. The age of a page <i>p</i> in the repository, at time <i>t</i> is defined as:
</p>
<dl><dd></dd></dl>
<p><a href="/wiki/Edward_G._Coffman,_Jr.">Coffman</a> <i>et al.</i> worked with a definition of the objective of a Web crawler that is equivalent to freshness, but use a different wording: they propose that a crawler must minimize the fraction of time pages remain outdated. They also noted that the problem of Web crawling can be modeled as a multiple-queue, single-server polling system, on which the Web crawler is the server and the Web sites are the queues. Page modifications are the arrival of the customers, and switch-over times are the interval between page accesses to a single Web site. Under this model, mean waiting time for a customer in the polling system is equivalent to the average age for the Web crawler.<sup><a href="#cite_note-coffman-30">[30]</a></sup>
</p><p>The objective of the crawler is to keep the average freshness of pages in its collection as high as possible, or to keep the average age of pages as low as possible. These objectives are not equivalent: in the first case, the crawler is just concerned with how many pages are out-dated, while in the second case, the crawler is concerned with how old the local copies of pages are.
</p>
<div><p><a href="/wiki/File:Web_Crawling_Freshness_Age.png"></a></p><div><p>Evolution of Freshness and Age in a web crawler</p></div></div>
<p>Two simple re-visiting policies were studied by Cho and Garcia-Molina:<sup><a href="#cite_note-cho2003-31">[31]</a></sup>
</p>
<ul><li>Uniform policy: This involves re-visiting all pages in the collection with the same frequency, regardless of their rates of change.</li>
<li>Proportional policy: This involves re-visiting more often the pages that change more frequently. The visiting frequency is directly proportional to the (estimated) change frequency.</li></ul>
<p>In both cases, the repeated crawling order of pages can be done either in a random or a fixed order.
</p><p>Cho and Garcia-Molina proved the surprising result that, in terms of average freshness, the uniform policy outperforms the proportional policy in both a simulated Web and a real Web crawl. Intuitively, the reasoning is that, as web crawlers have a limit to how many pages they can crawl in a given time frame, (1) they will allocate too many new crawls to rapidly changing pages at the expense of less frequently updating pages, and (2) the freshness of rapidly changing pages lasts for shorter period than that of less frequently changing pages. In other words, a proportional policy allocates more resources to crawling frequently updating pages, but experiences less overall freshness time from them.
</p><p>To improve freshness, the crawler should penalize the elements that change too often.<sup><a href="#cite_note-cho2003a-32">[32]</a></sup> The optimal re-visiting policy is neither the uniform policy nor the proportional policy. The optimal method for keeping average freshness high includes ignoring the pages that change too often, and the optimal for keeping average age low is to use access frequencies that monotonically (and sub-linearly) increase with the rate of change of each page. In both cases, the optimal is closer to the uniform policy than to the proportional policy: as <a href="/wiki/Edward_G._Coffman,_Jr.">Coffman</a> <i>et al.</i> note, "in order to minimize the expected obsolescence time, the accesses to any particular page should be kept as evenly spaced as possible".<sup><a href="#cite_note-coffman-30">[30]</a></sup> Explicit formulas for the re-visit policy are not attainable in general, but they are obtained numerically, as they depend on the distribution of page changes. Cho and Garcia-Molina show that the exponential distribution is a good fit for describing page changes,<sup><a href="#cite_note-cho2003a-32">[32]</a></sup> while <a href="/wiki/Panos_Ipeirotis">Ipeirotis</a> <i>et al.</i> show how to use statistical tools to discover parameters that affect this distribution.<sup><a href="#cite_note-33">[33]</a></sup> Note that the re-visiting policies considered here regard all pages as homogeneous in terms of quality ("all pages on the Web are worth the same"), something that is not a realistic scenario, so further information about the Web page quality should be included to achieve a better crawling policy.
</p>
<h3>Politeness policy[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=12">edit</a>]</h3>
<p>Crawlers can retrieve data much quicker and in greater depth than human searchers, so they can have a crippling impact on the performance of a site. If a single crawler is performing multiple requests per second and/or downloading large files, a server can have a hard time keeping up with requests from multiple crawlers.
</p><p>As noted by Koster, the use of Web crawlers is useful for a number of tasks, but comes with a price for the general community.<sup><a href="#cite_note-34">[34]</a></sup> The costs of using Web crawlers include:
</p>
<ul><li>network resources, as crawlers require considerable bandwidth and operate with a high degree of parallelism during a long period of time;</li>
<li>server overload, especially if the frequency of accesses to a given server is too high;</li>
<li>poorly written crawlers, which can crash servers or routers, or which download pages they cannot handle; and</li>
<li>personal crawlers that, if deployed by too many users, can disrupt networks and Web servers.</li></ul>
<p>A partial solution to these problems is the <a href="/wiki/Robots_Exclusion_Standard">robots exclusion protocol</a>, also known as the robots.txt protocol that is a standard for administrators to indicate which parts of their Web servers should not be accessed by crawlers.<sup><a href="#cite_note-35">[35]</a></sup> This standard does not include a suggestion for the interval of visits to the same server, even though this interval is the most effective way of avoiding server overload. Recently commercial search engines like <a href="/wiki/Google.com">Google</a>, <a href="/wiki/Ask.com">Ask Jeeves</a>, <a href="/wiki/Bing_(search_engine)">MSN</a> and <a href="/wiki/Yahoo!_Search">Yahoo! Search</a> are able to use an extra "Crawl-delay:" parameter in the <a href="/wiki/Robots.txt">robots.txt</a> file to indicate the number of seconds to delay between requests.
</p><p>The first proposed interval between successive pageloads was 60 seconds.<sup><a href="#cite_note-36">[36]</a></sup> However, if pages were downloaded at this rate from a website with more than 100,000 pages over a perfect connection with zero latency and infinite bandwidth, it would take more than 2 months to download only that entire Web site; also, only a fraction of the resources from that Web server would be used. This does not seem acceptable.
</p><p>Cho uses 10 seconds as an interval for accesses,<sup><a href="#cite_note-cho2003-31">[31]</a></sup> and the WIRE crawler uses 15 seconds as the default.<sup><a href="#cite_note-baeza2002-37">[37]</a></sup> The MercatorWeb crawler follows an adaptive politeness policy: if it took <i>t</i> seconds to download a document from a given server, the crawler waits for 10<i>t</i> seconds before downloading the next page.<sup><a href="#cite_note-38">[38]</a></sup> Dill <i>et al.</i> use 1 second.<sup><a href="#cite_note-39">[39]</a></sup>
</p><p>For those using Web crawlers for research purposes, a more detailed cost-benefit analysis is needed and ethical considerations should be taken into account when deciding where to crawl and how fast to crawl.<sup><a href="#cite_note-40">[40]</a></sup>
</p><p>Anecdotal evidence from access logs shows that access intervals from known crawlers vary between 20 seconds and 3–4 minutes. It is worth noticing that even when being very polite, and taking all the safeguards to avoid overloading Web servers, some complaints from Web server administrators are received. <a href="/wiki/Sergey_Brin">Brin</a> and <a href="/wiki/Larry_Page">Page</a> note that: "... running a crawler which connects to more than half a million servers ... generates a fair amount of e-mail and phone calls. Because of the vast number of people coming on line, there are always those who do not know what a crawler is, because this is the first one they have seen."<sup><a href="#cite_note-brin1998-41">[41]</a></sup>
</p>
<h3>Parallelization policy[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=13">edit</a>]</h3>
<p>A <a href="/wiki/Parallel_computing">parallel</a> crawler is a crawler that runs multiple processes in parallel. The goal is to maximize the download rate while minimizing the overhead from parallelization and to avoid repeated downloads of the same page. To avoid downloading the same page more than once, the crawling system requires a policy for assigning the new URLs discovered during the crawling process, as the same URL can be found by two different crawling processes.
</p>
<h2>Architectures[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=14">edit</a>]</h2>
<div><p><a href="/wiki/File:WebCrawlerArchitecture.svg"></a></p><div><p>High-level architecture of a standard Web crawler</p></div></div>
<p>A crawler must not only have a good crawling strategy, as noted in the previous sections, but it should also have a highly optimized architecture.
</p><p>Shkapenyuk and Suel noted that:<sup><a href="#cite_note-shkapenyuk2002-42">[42]</a></sup>
</p>
<blockquote><p>While it is fairly easy to build a slow crawler that downloads a few pages per second for a short period of time, building a high-performance system that can download hundreds of millions of pages over several weeks presents a number of challenges in system design, I/O and network efficiency, and robustness and manageability.</p></blockquote>
<p>Web crawlers are a central part of search engines, and details on their algorithms and architecture are kept as business secrets. When crawler designs are published, there is often an important lack of detail that prevents others from reproducing the work. There are also emerging concerns about "<a href="/wiki/Spamdexing">search engine spamming</a>", which prevent major search engines from publishing their ranking algorithms.
</p>
<h2>Security[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=15">edit</a>]</h2>
<p>While most of the website owners are keen to have their pages indexed as broadly as possible to have strong presence in <a href="/wiki/Web_search_engine">search engines</a>, web crawling can also have <a href="/wiki/Unintended_consequences">unintended consequences</a> and lead to a <a href="/wiki/Web_application_security">compromise</a> or <a href="/wiki/Data_breach">data breach</a> if a search engine indexes resources that shouldn't be publicly available, or pages revealing potentially vulnerable versions of software.
</p>
<p>Apart from standard <a href="/wiki/Web_application_security">web application security</a> recommendations website owners can reduce their exposure to opportunistic hacking by only allowing search engines to index the public parts of their websites (with <a href="/wiki/Robots_exclusion_standard">robots.txt</a>) and explicitly blocking them from indexing transactional parts (login pages, private pages, etc.).
</p>
<h2>Crawler identification[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=16">edit</a>]</h2>
<p>Web crawlers typically identify themselves to a Web server by using the <a href="/wiki/User_agent">User-agent</a> field of an <a href="/wiki/HTTP">HTTP</a> request. Web site administrators typically examine their <a href="/wiki/Web_server">Web servers</a>' log and use the user agent field to determine which crawlers have visited the web server and how often. The user agent field may include a <a href="/wiki/Uniform_Resource_Locator">URL</a> where the Web site administrator may find out more information about the crawler. Examining Web server log is tedious task, and therefore some administrators use tools to identify, track and verify Web crawlers. <a href="/wiki/Spambots">Spambots</a> and other malicious Web crawlers are unlikely to place identifying information in the user agent field, or they may mask their identity as a browser or other well-known crawler.
</p><p>Web site administrators prefer Web crawlers to identify themselves so that they can contact the owner if needed. In some cases, crawlers may be accidentally trapped in a <a href="/wiki/Spider_trap">crawler trap</a> or they may be overloading a Web server with requests, and the owner needs to stop the crawler. Identification is also useful for administrators that are interested in knowing when they may expect their Web pages to be indexed by a particular <a href="/wiki/Web_search_engine">search engine</a>.
</p>
<h2>Crawling the deep web[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=17">edit</a>]</h2>
<p>A vast amount of web pages lie in the <a href="/wiki/Deep_Web_(search_indexing)">deep or invisible web</a>.<sup><a href="#cite_note-43">[43]</a></sup> These pages are typically only accessible by submitting queries to a database, and regular crawlers are unable to find these pages if there are no links that point to them. Google's <a href="/wiki/Sitemaps">Sitemaps</a> protocol and <a href="/wiki/Mod_oai">mod oai</a><sup><a href="#cite_note-44">[44]</a></sup> are intended to allow discovery of these deep-Web resources.
</p><p>Deep web crawling also multiplies the number of web links to be crawled. Some crawlers only take some of the URLs in <code>&lt;a href="URL"&gt;</code> form. In some cases, such as the <a href="/wiki/Googlebot">Googlebot</a>, Web crawling is done on all text contained inside the hypertext content, tags, or text.
</p><p>Strategic approaches may be taken to target deep Web content. With a technique called <a href="/wiki/Screen_scraping">screen scraping</a>, specialized software may be customized to automatically and repeatedly query a given Web form with the intention of aggregating the resulting data. Such software can be used to span multiple Web forms across multiple Websites. Data extracted from the results of one Web form submission can be taken and applied as input to another Web form thus establishing continuity across the Deep Web in a way not possible with traditional web crawlers.<sup><a href="#cite_note-45">[45]</a></sup>
</p><p>Pages built on <a href="/wiki/AJAX">AJAX</a> are among those causing problems to web crawlers. Google has proposed a format of AJAX calls that their bot can recognize and index.<sup><a href="#cite_note-46">[46]</a></sup>
</p>
<h2>Visual vs programmatic crawlers[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=18">edit</a>]</h2>
<p>There are a number of "visual web scraper/crawler" products available on the web which will crawl pages and structure data into columns and rows based on the users requirements. One of the main difference between a classic and a visual crawler is the level of programming ability required to set up a crawler. The latest generation of "visual scrapers" remove the majority of the programming skill needed to be able to program and start a crawl to scrape web data.
</p><p>The visual scraping/crawling method relies on the user "teaching" a piece of crawler technology, which then follows patterns in semi-structured data sources. The dominant method for teaching a visual crawler is by highlighting data in a browser and training columns and rows. While the technology is not new, for example it was the basis of Needlebase which has been bought by Google (as part of a larger acquisition of ITA Labs<sup><a href="#cite_note-47">[47]</a></sup>), there is continued growth and investment in this area by investors and end-users.<sup>[<i><a href="/wiki/Wikipedia:Citation_needed">citation needed</a></i>]</sup>
</p>
<h2>List of web crawlers[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=19">edit</a>]</h2>
<p>The following is a list of published crawler architectures for general-purpose crawlers (excluding focused web crawlers), with a brief description that includes the names given to the different components and outstanding features:
</p>
<h3>Historical web crawlers[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=20">edit</a>]</h3>
<ul><li><a href="/wiki/World_Wide_Web_Worm">World Wide Web Worm</a> was a crawler used to build a simple index of document titles and URLs. The index could be searched by using the <kbd><a href="/wiki/Grep">grep</a></kbd> <a href="/wiki/Unix">Unix</a> command.</li>
<li>Yahoo! Slurp was the name of the <a href="/wiki/Yahoo!">Yahoo!</a> Search crawler until Yahoo! contracted with <a href="/wiki/Microsoft">Microsoft</a> to use <a href="/wiki/Bingbot">Bingbot</a> instead.</li></ul>
<h3>In-house web crawlers[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=21">edit</a>]</h3>
<ul><li>Applebot is <a href="/wiki/Apple_(company)">Apple</a>'s web crawler. It supports <a href="/wiki/Siri">Siri</a> and other products.<sup><a href="#cite_note-48">[48]</a></sup></li>
<li><a href="/wiki/Bingbot">Bingbot</a> is the name of Microsoft's <a href="/wiki/Bing_(search_engine)">Bing</a> webcrawler. It replaced <i><a href="/wiki/Msnbot">Msnbot</a></i>.</li>
<li>Baiduspider is <a href="/wiki/Baidu">Baidu</a>'s web crawler.</li>
<li><a href="/wiki/Googlebot">Googlebot</a> is described in some detail, but the reference is only about an early version of its architecture, which was written in C++ and <a href="/wiki/Python_(programming_language)">Python</a>. The crawler was integrated with the indexing process, because text parsing was done for full-text indexing and also for URL extraction. There is a URL server that sends lists of URLs to be fetched by several crawling processes. During parsing, the URLs found were passed to a URL server that checked if the URL have been previously seen. If not, the URL was added to the queue of the URL server.</li>
<li><a href="/wiki/WebCrawler">WebCrawler</a> was used to build the first publicly available full-text index of a subset of the Web. It was based on <a href="/wiki/Libwww">lib-WWW</a> to download pages, and another program to parse and order URLs for breadth-first exploration of the Web graph. It also included a real-time crawler that followed links based on the similarity of the anchor text with the provided query.</li>
<li><a href="/wiki/WebFountain">WebFountain</a> is a distributed, modular crawler similar to Mercator but written in C++.</li>
<li><a href="/wiki/Xenon_(program)">Xenon</a> is a web crawler used by government tax authorities to detect fraud.<sup><a href="#cite_note-Norton-2007-01-25-49">[49]</a></sup><sup><a href="#cite_note-Canada-2017-04-11-50">[50]</a></sup></li></ul>
<h3>Commercial web crawlers[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=22">edit</a>]</h3>
<p>The following web crawlers are available, for a price::
</p>
<ul><li><a href="/wiki/Diffbot">Diffbot</a> - programmatic general web crawler, available as an <a href="/wiki/API">API</a></li>
<li><a href="/wiki/SortSite">SortSite</a> - crawler for analyzing websites, available for <a href="/wiki/Microsoft_Windows">Windows</a> and <a href="/wiki/Macintosh_operating_systems">Mac OS</a></li>
<li>Swiftbot - <a href="/wiki/Swiftype">Swiftype</a>'s web crawler, available as <a href="/wiki/Software_as_a_service">software as a service</a></li></ul>
<h3>Open-source crawlers[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=23">edit</a>]</h3>
<ul><li><a href="/wiki/Wget">GNU Wget</a> is a <a href="/wiki/Command_line_interface">command-line</a>-operated crawler written in <a href="/wiki/C_(programming_language)">C</a> and released under the <a href="/wiki/GNU_General_Public_License">GPL</a>. It is typically used to mirror Web and FTP sites.</li>
<li><a href="/wiki/Grub_(search_engine)">GRUB</a> was an open source distributed search crawler that <a href="/wiki/Wikia_Search">Wikia Search</a> used to crawl the web.</li>
<li><a href="/wiki/Heritrix">Heritrix</a> is the <a href="/wiki/Internet_Archive">Internet Archive</a>'s archival-quality crawler, designed for archiving periodic snapshots of a large portion of the Web. It was written in <a href="/wiki/Java_(programming_language)">Java</a>.</li>
<li><a href="/wiki/Ht-//dig">ht://Dig</a> includes a Web crawler in its indexing engine.</li>
<li><a href="/wiki/HTTrack">HTTrack</a> uses a Web crawler to create a mirror of a web site for off-line viewing. It is written in <a href="/wiki/C_(programming_language)">C</a> and released under the GPL.</li>
<li><a href="/wiki/MnoGoSearch">mnoGoSearch</a> is a crawler, indexer and a search engine written in C and licensed under the GPL (*NIX machines only)</li>
<li><a href="/wiki/Apache_Nutch">Apache Nutch</a> is a highly extensible and scalable web crawler written in Java and released under an <a href="/wiki/Apache_License">Apache License</a>. It is based on <a href="/wiki/Apache_Hadoop">Apache Hadoop</a> and can be used with <a href="/wiki/Apache_Solr">Apache Solr</a> or <a href="/wiki/Elasticsearch">Elasticsearch</a>.</li>
<li><a href="/wiki/Open_Search_Server">Open Search Server</a> is a search engine and web crawler software release under the GPL.</li>
<li><a href="/wiki/Scrapy">Scrapy</a>, an open source webcrawler framework, written in python (licensed under <a href="/wiki/BSD_License">BSD</a>).</li>
<li><a href="/wiki/Seeks">Seeks</a>, a free distributed search engine (licensed under <a href="/wiki/GNU_Affero_General_Public_License">AGPL</a>).</li>
<li><a href="/wiki/StormCrawler">StormCrawler</a>, a collection of resources for building low-latency, scalable web crawlers on <a href="/wiki/Storm_(event_processor)">Apache Storm</a> (Apache License).</li>
<li><a href="/wiki/TkWWW_Robot">tkWWW Robot</a>, a crawler based on the <a href="/wiki/TkWWW">tkWWW</a> web browser (licensed under GPL).</li>
<li><a href="/wiki/Xapian">Xapian</a>, a search crawler engine, written in c++.</li>
<li><a href="/wiki/YaCy">YaCy</a>, a free distributed search engine, built on principles of peer-to-peer networks (licensed under GPL).</li></ul>
<h2>See also[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=24">edit</a>]</h2>
<ul><li><a href="/wiki/Automatic_indexing">Automatic indexing</a></li>
<li><a href="/wiki/Gnutella_crawler">Gnutella crawler</a></li>
<li><a href="/wiki/Web_archiving">Web archiving</a></li>
<li><a href="/wiki/Webgraph">Webgraph</a></li>
<li><a href="/wiki/Website_mirroring_software">Website mirroring software</a></li>
<li><a href="/wiki/Search_Engine_Scraping">Search Engine Scraping</a></li>
<li><a href="/wiki/Web_scraping">Web scraping</a></li></ul>
<h2>References[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=25">edit</a>]</h2>
<div>
<ol>
<li><b><a href="#cite_ref-1">^</a></b> <cite><a href="https://web.archive.org/web/20211206205907/https://webbrowsersintroduction.com/">"Web Crawlers: Browsing the Web"</a>. Archived from <a href="https://webbrowsersintroduction.com/">the original</a> on 6 December 2021.</cite>
</li>
<li><b><a href="#cite_ref-spekta_2-0">^</a></b> <cite>Spetka, Scott. <a href="https://web.archive.org/web/20040903174942/http://archive.ncsa.uiuc.edu/SDG/IT94/Proceedings/Agents/spetka/spetka.html">"The TkWWW Robot: Beyond Browsing"</a>. <a href="/wiki/National_Center_for_Supercomputing_Applications">NCSA</a>. Archived from <a href="http://archive.ncsa.uiuc.edu/SDG/IT94/Proceedings/Agents/spetka/spetka.html">the original</a> on 3 September 2004. Retrieved 21 November 2010.</cite>
</li>
<li><b><a href="#cite_ref-3">^</a></b> <cite>Kobayashi, M. &amp; Takeda, K. (2000). "Information retrieval on the web". <i>ACM Computing Surveys</i>. <b>32</b> (2): 144–173. <a href="/wiki/CiteSeerX_(identifier)">CiteSeerX</a> <a href="//citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.126.6094">10.1.1.126.6094</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1145%2F358923.358934">10.1145/358923.358934</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:3710903">3710903</a>.</cite>
</li>
<li><b><a href="#cite_ref-4">^</a></b> See <a href="http://wiki.foaf-project.org/w/Scutter">definition of scutter on FOAF Project's wiki</a> <a href="https://web.archive.org/web/20091213213920/http://wiki.foaf-project.org/w/Scutter">Archived</a> 13 December 2009 at the <a href="/wiki/Wayback_Machine">Wayback Machine</a>
</li>
<li><b><a href="#cite_ref-GoogleBooks-9237380_5-0">^</a></b> <cite>Masanès, Julien (15 February 2007). <a href="https://www.springer.com/gp/book/9783540233381"><i>Web Archiving</i></a>. Springer. p. 1. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-3-54046332-0"><bdi>978-3-54046332-0</bdi></a>. Retrieved 24 April 2014.</cite>
</li>
<li><b><a href="#cite_ref-edwards2001_6-0">^</a></b> <cite>Edwards, J.; McCurley, K. S.; and Tomlin, J. A. (2001). "An adaptive model for optimizing performance of an incremental web crawler". <a href="http://www10.org/cdrom/papers/210/index.html"><i>Proceedings of the tenth international conference on World Wide Web - WWW '01</i></a>. <i>In Proceedings of the Tenth Conference on World Wide Web</i>. pp. 106–113. <a href="/wiki/CiteSeerX_(identifier)">CiteSeerX</a> <a href="//citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.1018.1506">10.1.1.1018.1506</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1145%2F371920.371960">10.1145/371920.371960</a>. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-1581133486"><bdi>978-1581133486</bdi></a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:10316730">10316730</a>.</cite><code>{{<a href="/wiki/Template:Cite_book">cite book</a>}}</code>: CS1 maint: multiple names: authors list (<a href="/wiki/Category:CS1_maint:_multiple_names:_authors_list">link</a>)
</li>
<li><b><a href="#cite_ref-7">^</a></b> <cite>Castillo, Carlos (2004). <a href="http://chato.cl/research/crawling_thesis"><i>Effective Web Crawling</i></a> (PhD thesis). University of Chile. Retrieved 3 August 2010.</cite>
</li>
<li><b><a href="#cite_ref-8">^</a></b> <cite>Gulls, A.; A. Signori (2005). "The indexable web is more than 11.5 billion pages". <i>Special interest tracks and posters of the 14th international conference on World Wide Web</i>. ACM Press. pp. 902–903. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1145%2F1062745.1062789">10.1145/1062745.1062789</a>.</cite>
</li>
<li><b><a href="#cite_ref-9">^</a></b> <cite>Lawrence, Steve; C. Lee Giles (8 July 1999). <a href="https://doi.org/10.1038%2F21987">"Accessibility of information on the web"</a>. <i>Nature</i>. <b>400</b> (6740): 107–9. <a href="/wiki/Bibcode_(identifier)">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/1999Natur.400..107L">1999Natur.400..107L</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1038%2F21987">10.1038/21987</a>. <a href="/wiki/PMID_(identifier)">PMID</a> <a href="//pubmed.ncbi.nlm.nih.gov/10428673">10428673</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:4347646">4347646</a>.</cite>
</li>
<li><b><a href="#cite_ref-10">^</a></b> <cite>Cho, J.; Garcia-Molina, H.; Page, L. (April 1998). <a href="http://ilpubs.stanford.edu:8090/347/">"Efficient Crawling Through URL Ordering"</a>. <i>Seventh International World-Wide Web Conference</i>. Brisbane, Australia. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1142%2F3725">10.1142/3725</a>. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-981-02-3400-3"><bdi>978-981-02-3400-3</bdi></a>. Retrieved 23 March 2009.</cite>
</li>
<li><b><a href="#cite_ref-11">^</a></b> Cho, Junghoo, <a href="http://oak.cs.ucla.edu/~cho/papers/cho-thesis.pdf">"Crawling the Web: Discovery and Maintenance of a Large-Scale Web Data"</a>, PhD dissertation, Department of Computer Science, Stanford University, November 2001.
</li>
<li><b><a href="#cite_ref-12">^</a></b> Najork, Marc and Janet L. Wiener. <a href="http://www10.org/cdrom/papers/pdf/p208.pdf">"Breadth-first crawling yields high-quality pages".</a> In: <i>Proceedings of the Tenth Conference on World Wide Web</i>, pages 114–118, Hong Kong, May 2001. Elsevier Science.
</li>
<li><b><a href="#cite_ref-13">^</a></b> <cite>Abiteboul, Serge; Mihai Preda; Gregory Cobena (2003). <a href="http://www2003.org/cdrom/papers/refereed/p007/p7-abiteboul.html">"Adaptive on-line page importance computation"</a>. <i>Proceedings of the 12th international conference on World Wide Web</i>. Budapest, Hungary: ACM. pp. 280–290. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1145%2F775152.775192">10.1145/775152.775192</a>. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/1-58113-680-3"><bdi>1-58113-680-3</bdi></a>. Retrieved 22 March 2009.</cite>
</li>
<li><b><a href="#cite_ref-14">^</a></b> <cite>Boldi, Paolo; Bruno Codenotti; Massimo Santini; Sebastiano Vigna (2004). <a href="https://web.archive.org/web/20090320030833/http://vigna.dsi.unimi.it/ftp/papers/UbiCrawler.pdf">"UbiCrawler: a scalable fully distributed Web crawler"</a> (PDF). <i>Software: Practice and Experience</i>. <b>34</b> (8): 711–726. <a href="/wiki/CiteSeerX_(identifier)">CiteSeerX</a> <a href="//citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.2.5538">10.1.1.2.5538</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1002%2Fspe.587">10.1002/spe.587</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:325714">325714</a>. Archived from <a href="http://vigna.dsi.unimi.it/ftp/papers/UbiCrawler.pdf">the original</a> (PDF) on 20 March 2009. Retrieved 23 March 2009.</cite>
</li>
<li><b><a href="#cite_ref-15">^</a></b> <cite>Boldi, Paolo; Massimo Santini; Sebastiano Vigna (2004). <a href="http://vigna.dsi.unimi.it/ftp/papers/ParadoxicalPageRank.pdf">"Do Your Worst to Make the Best: Paradoxical Effects in PageRank Incremental Computations"</a> (PDF). <i>Algorithms and Models for the Web-Graph</i>. Lecture Notes in Computer Science. Vol. 3243. pp. 168–180. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1007%2F978-3-540-30216-2_14">10.1007/978-3-540-30216-2_14</a>. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-3-540-23427-2"><bdi>978-3-540-23427-2</bdi></a>. Retrieved 23 March 2009.</cite>
</li>
<li><b><a href="#cite_ref-baeza2005_16-0">^</a></b> Baeza-Yates, R.; Castillo, C.; Marin, M. and Rodriguez, A. (2005). <a href="http://chato.cl/papers/baeza05_crawling_country_better_breadth_first_web_page_ordering.pdf">"Crawling a Country: Better Strategies than Breadth-First for Web Page Ordering."</a> In: <i>Proceedings of the Industrial and Practical Experience track of the 14th conference on World Wide Web</i>, pages 864–872, Chiba, Japan. ACM Press.
</li>
<li><b><a href="#cite_ref-17">^</a></b> Shervin Daneshpajouh, Mojtaba Mohammadi Nasiri, Mohammad Ghodsi, <a href="http://ce.sharif.edu/~daneshpajouh/publications/A%20Fast%20Community%20Based%20Algorithm%20for%20Generating%20Crawler%20Seeds%20Set.pdf">A Fast Community Based Algorithm for Generating Crawler Seeds Set</a>. In: <i>Proceedings of 4th International Conference on Web Information Systems and Technologies</i> (<i><a href="/wiki/Webist">Webist</a></i>-2008), Funchal, Portugal, May 2008.
</li>
<li><b><a href="#cite_ref-18">^</a></b> <cite>Pant, Gautam; Srinivasan, Padmini; Menczer, Filippo (2004). <a href="https://web.archive.org/web/20090320030832/http://dollar.biz.uiowa.edu/~pant/Papers/crawling.pdf">"Crawling the Web"</a> (PDF). In Levene, Mark; <a href="/wiki/Alexandra_Poulovassilis">Poulovassilis, Alexandra</a> (eds.). <i>Web Dynamics: Adapting to Change in Content, Size, Topology and Use</i>. Springer. pp. 153–178. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-3-540-40676-1"><bdi>978-3-540-40676-1</bdi></a>. Archived from <a href="http://dollar.biz.uiowa.edu/~pant/Papers/crawling.pdf">the original</a> (PDF) on 20 March 2009. Retrieved 9 May 2006.</cite>
</li>
<li><b><a href="#cite_ref-19">^</a></b> <cite>Cothey, Viv (2004). <a href="http://www.scit.wlv.ac.uk/~in7803/publications/cothey_2004.pdf">"Web-crawling reliability"</a> (PDF). <i>Journal of the American Society for Information Science and Technology</i>. <b>55</b> (14): 1228–1238. <a href="/wiki/CiteSeerX_(identifier)">CiteSeerX</a> <a href="//citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.117.185">10.1.1.117.185</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1002%2Fasi.20078">10.1002/asi.20078</a>.</cite>
</li>
<li><b><a href="#cite_ref-20">^</a></b> Menczer, F. (1997). <a href="http://informatics.indiana.edu/fil/Papers/ICML.ps">ARACHNID: Adaptive Retrieval Agents Choosing Heuristic Neighborhoods for Information Discovery</a> <a href="https://web.archive.org/web/20121221113620/http://informatics.indiana.edu/fil/Papers/ICML.ps">Archived</a> 21 December 2012 at the <a href="/wiki/Wayback_Machine">Wayback Machine</a>. In D. Fisher, ed., Machine Learning: Proceedings of the 14th International Conference (ICML97). Morgan Kaufmann
</li>
<li><b><a href="#cite_ref-21">^</a></b> Menczer, F. and Belew, R.K. (1998). <a href="http://informatics.indiana.edu/fil/Papers/AA98.ps">Adaptive Information Agents in Distributed Textual Environments</a> <a href="https://web.archive.org/web/20121221113630/http://informatics.indiana.edu/fil/Papers/AA98.ps">Archived</a> 21 December 2012 at the <a href="/wiki/Wayback_Machine">Wayback Machine</a>. In K. Sycara and M. Wooldridge (eds.) Proc. 2nd Intl. Conf. on Autonomous Agents (Agents '98). ACM Press
</li>
<li><b><a href="#cite_ref-22">^</a></b> <cite>Chakrabarti, Soumen; Van Den Berg, Martin; Dom, Byron (1999). <a href="https://web.archive.org/web/20040317210216/http://www.fxpal.com/people/vdberg/pubs/www8/www1999f.pdf">"Focused crawling: A new approach to topic-specific Web resource discovery"</a> (PDF). <i>Computer Networks</i>. <b>31</b> (11–16): 1623–1640. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1016%2Fs1389-1286%2899%2900052-3">10.1016/s1389-1286(99)00052-3</a>. Archived from <a href="http://www.fxpal.com/people/vdberg/pubs/www8/www1999f.pdf">the original</a> (PDF) on 17 March 2004.</cite>
</li>
<li><b><a href="#cite_ref-pinkerton1994_23-0">^</a></b> Pinkerton, B. (1994). <a href="https://web.archive.org/web/20010904075500/http://archive.ncsa.uiuc.edu/SDG/IT94/Proceedings/Searching/pinkerton/WebCrawler.html">Finding what people want: Experiences with the WebCrawler</a>. In Proceedings of the First World Wide Web Conference, Geneva, Switzerland.
</li>
<li><b><a href="#cite_ref-24">^</a></b> Diligenti, M., Coetzee, F., Lawrence, S., Giles, C. L., and Gori, M. (2000). <a href="http://clgiles.ist.psu.edu/papers/VLDB-2000-focused-crawling.pdf">Focused crawling using context graphs</a>. In Proceedings of 26th International Conference on Very Large Databases (VLDB), pages 527-534, Cairo, Egypt.
</li>
<li><b><a href="#cite_ref-25">^</a></b> <cite>Wu, Jian; Teregowda, Pradeep; Khabsa, Madian; Carman, Stephen; Jordan, Douglas; San Pedro Wandelmer, Jose; Lu, Xin; Mitra, Prasenjit; Giles, C. Lee (2012). "Web crawler middleware for search engine digital libraries". <i>Proceedings of the twelfth international workshop on Web information and data management - WIDM '12</i>. p. 57. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1145%2F2389936.2389949">10.1145/2389936.2389949</a>. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/9781450317207"><bdi>9781450317207</bdi></a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:18513666">18513666</a>.</cite>
</li>
<li><b><a href="#cite_ref-26">^</a></b> <cite>Wu, Jian; Teregowda, Pradeep; Ramírez, Juan Pablo Fernández; Mitra, Prasenjit; Zheng, Shuyi; Giles, C. Lee (2012). "The evolution of a crawling strategy for an academic document search engine". <i>Proceedings of the 3rd Annual ACM Web Science Conference on - Web </i>Sci<i> '12</i>. pp. 340–343. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1145%2F2380718.2380762">10.1145/2380718.2380762</a>. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/9781450312288"><bdi>9781450312288</bdi></a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:16718130">16718130</a>.</cite>
</li>
<li><b><a href="#cite_ref-27">^</a></b> <cite>Dong, Hai; Hussain, Farookh Khadeer; Chang, Elizabeth (2009). <a href="https://www.researchgate.net/publication/44241179">"State of the Art in Semantic Focused Crawlers"</a>. <i>Computational Science and Its Applications – ICCSA 2009</i>. Lecture Notes in Computer Science. Vol. 5593. pp. 910–924. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1007%2F978-3-642-02457-3_74">10.1007/978-3-642-02457-3_74</a>. <a href="/wiki/Hdl_(identifier)">hdl</a>:<a href="//hdl.handle.net/20.500.11937%2F48288">20.500.11937/48288</a>. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-3-642-02456-6"><bdi>978-3-642-02456-6</bdi></a>.</cite>
</li>
<li><b><a href="#cite_ref-28">^</a></b> <cite>Dong, Hai; Hussain, Farookh Khadeer (2013). <a href="https://www.researchgate.net/publication/264620349">"SOF: A semi-supervised ontology-learning-based focused crawler"</a>. <i>Concurrency and Computation: Practice and Experience</i>. <b>25</b> (12): 1755–1770. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1002%2Fcpe.2980">10.1002/cpe.2980</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:205690364">205690364</a>.</cite>
</li>
<li><b><a href="#cite_ref-29">^</a></b> <cite>Junghoo Cho; Hector Garcia-Molina (2000). <a href="http://www.cs.brown.edu/courses/cs227/2002/cache/Cho.pdf">"Synchronizing a database to improve freshness"</a> (PDF). <i>Proceedings of the 2000 ACM SIGMOD international conference on Management of data</i>. Dallas, Texas, United States: ACM. pp. 117–128. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1145%2F342009.335391">10.1145/342009.335391</a>. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/1-58113-217-4"><bdi>1-58113-217-4</bdi></a>. Retrieved 23 March 2009.</cite>
</li>
<li>^ <a href="#cite_ref-coffman_30-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-coffman_30-1"><sup><i><b>b</b></i></sup></a> <cite>E. G. Coffman Jr; Zhen Liu; Richard R. Weber (1998). "Optimal robot scheduling for Web search engines". <i>Journal of Scheduling</i>. <b>1</b> (1): 15–29. <a href="/wiki/CiteSeerX_(identifier)">CiteSeerX</a> <a href="//citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.36.6087">10.1.1.36.6087</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1002%2F%28SICI%291099-1425%28199806%291%3A1%3C15%3A%3AAID-JOS3%3E3.0.CO%3B2-K">10.1002/(SICI)1099-1425(199806)1:1&lt;15::AID-JOS3&gt;3.0.CO;2-K</a>.</cite>
</li>
<li>^ <a href="#cite_ref-cho2003_31-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-cho2003_31-1"><sup><i><b>b</b></i></sup></a> <cite>Cho, Junghoo; Garcia-Molina, Hector (2003). "Effective page refresh policies for Web crawlers". <i>ACM Transactions on Database Systems</i>. <b>28</b> (4): 390–426. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1145%2F958942.958945">10.1145/958942.958945</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:147958">147958</a>.</cite>
</li>
<li>^ <a href="#cite_ref-cho2003a_32-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-cho2003a_32-1"><sup><i><b>b</b></i></sup></a> <cite>Junghoo Cho; Hector Garcia-Molina (2003). "Estimating frequency of change". <i>ACM Transactions on Internet Technology</i>. <b>3</b> (3): 256–290. <a href="/wiki/CiteSeerX_(identifier)">CiteSeerX</a> <a href="//citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.59.5877">10.1.1.59.5877</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1145%2F857166.857170">10.1145/857166.857170</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:9362566">9362566</a>.</cite>
</li>
<li><b><a href="#cite_ref-33">^</a></b> Ipeirotis, P., Ntoulas, A., Cho, J., Gravano, L. (2005) <a href="http://pages.stern.nyu.edu/~panos/publications/icde2005.pdf">Modeling and managing content changes in text databases</a>. In Proceedings of the 21st IEEE International Conference on Data Engineering, pages 606-617, April 2005, Tokyo.
</li>
<li><b><a href="#cite_ref-34">^</a></b> Koster, M. (1995). Robots in the web: threat or treat? ConneXions, 9(4).
</li>
<li><b><a href="#cite_ref-35">^</a></b> Koster, M. (1996). <a href="http://www.robotstxt.org/wc/exclusion.html">A standard for robot exclusion</a> <a href="https://web.archive.org/web/20071107021800/http://www.robotstxt.org/wc/exclusion.html">Archived</a> 7 November 2007 at the <a href="/wiki/Wayback_Machine">Wayback Machine</a>.
</li>
<li><b><a href="#cite_ref-36">^</a></b> Koster, M. (1993). <a href="http://www.robotstxt.org/wc/guidelines.html">Guidelines for robots writers</a> <a href="https://web.archive.org/web/20050422045839/http://www.robotstxt.org/wc/guidelines.html">Archived</a> 22 April 2005 at the <a href="/wiki/Wayback_Machine">Wayback Machine</a>.
</li>
<li><b><a href="#cite_ref-baeza2002_37-0">^</a></b> Baeza-Yates, R. and Castillo, C. (2002). <a href="http://www.chato.cl/papers/baeza02balancing.pdf">Balancing volume, quality and freshness in Web crawling</a>. In Soft Computing Systems – Design, Management and Applications, pages 565–572, Santiago, Chile. IOS Press Amsterdam.
</li>
<li><b><a href="#cite_ref-38">^</a></b> <cite>Heydon, Allan; Najork, Marc (26 June 1999). <a href="https://web.archive.org/web/20060219085958/http://www.cindoc.csic.es/cybermetrics/pdf/68.pdf">"Mercator: A Scalable, Extensible Web Crawler"</a> (PDF). Archived from <a href="http://www.cindoc.csic.es/cybermetrics/pdf/68.pdf">the original</a> (PDF) on 19 February 2006. Retrieved 22 March 2009.</cite> 
</li>
<li><b><a href="#cite_ref-39">^</a></b> <cite>Dill, S.; Kumar, R.; Mccurley, K. S.; Rajagopalan, S.; Sivakumar, D.; Tomkins, A. (2002). <a href="http://www.mccurley.org/papers/fractal.pdf">"Self-similarity in the web"</a> (PDF). <i>ACM Transactions on Internet Technology</i>. <b>2</b> (3): 205–223. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1145%2F572326.572328">10.1145/572326.572328</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:6416041">6416041</a>.</cite>
</li>
<li><b><a href="#cite_ref-40">^</a></b> <cite>M. Thelwall; D. Stuart (2006). <a href="http://www.scit.wlv.ac.uk/%7Ecm1993/papers/Web_Crawling_Ethics_preprint.doc">"Web crawling ethics revisited: Cost, privacy and denial of service"</a>. <i>Journal of the American Society for Information Science and Technology</i>. <b>57</b> (13): 1771–1779. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1002%2Fasi.20388">10.1002/asi.20388</a>.</cite>
</li>
<li><b><a href="#cite_ref-brin1998_41-0">^</a></b> <cite>Brin, Sergey; Page, Lawrence (1998). <a href="http://infolab.stanford.edu/~backrub/google.html">"The anatomy of a large-scale hypertextual Web search engine"</a>. <i>Computer Networks and ISDN Systems</i>. <b>30</b> (1–7): 107–117. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1016%2Fs0169-7552%2898%2900110-x">10.1016/s0169-7552(98)00110-x</a>.</cite>
</li>
<li><b><a href="#cite_ref-shkapenyuk2002_42-0">^</a></b> Shkapenyuk, V. and <a href="/wiki/Torsten_Suel">Suel, T.</a> (2002). <a href="http://cis.poly.edu/tr/tr-cis-2001-03.pdf">Design and implementation of a high performance distributed web crawler</a>. In Proceedings of the 18th International Conference on Data Engineering (ICDE), pages 357-368, San Jose, California. IEEE CS Press.
</li>
<li><b><a href="#cite_ref-43">^</a></b> Shestakov, Denis (2008). <a href="https://oa.doria.fi/handle/10024/38506"><i>Search Interfaces on the Web: Querying and Characterizing</i></a> <a href="https://web.archive.org/web/20140706120641/https://oa.doria.fi/handle/10024/38506">Archived</a> 6 July 2014 at the <a href="/wiki/Wayback_Machine">Wayback Machine</a>. TUCS Doctoral Dissertations 104, University of Turku
</li>
<li><b><a href="#cite_ref-44">^</a></b> <cite>Michael L Nelson; Herbert Van de Sompel; Xiaoming Liu; Terry L Harrison; Nathan McFarland (24 March 2005). "mod_oai: An Apache Module for Metadata Harvesting": cs/0503069. <a href="/wiki/ArXiv_(identifier)">arXiv</a>:<a href="//arxiv.org/abs/cs/0503069">cs/0503069</a>. <a href="/wiki/Bibcode_(identifier)">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2005cs........3069N">2005cs........3069N</a>.</cite> 
</li>
<li><b><a href="#cite_ref-45">^</a></b> <cite>Shestakov, Denis; Bhowmick, Sourav S.; Lim, Ee-Peng (2005). <a href="http://www.mendeley.com/download/public/1423991/3893295922/dc0f7d824fd2a8fbbc84f6fdf9e4f337d343987d/dl.pdf">"DEQUE: Querying the Deep Web"</a> (PDF). <i>Data &amp; Knowledge Engineering</i>. <b>52</b> (3): 273–311. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1016%2Fs0169-023x%2804%2900107-7">10.1016/s0169-023x(04)00107-7</a>.</cite>
</li>
<li><b><a href="#cite_ref-46">^</a></b> <cite><a href="https://support.google.com/webmasters/bin/answer.py?hl=en&amp;answer=174992">"AJAX crawling: Guide for webmasters and developers"</a>. Retrieved 17 March 2013.</cite>
</li>
<li><b><a href="#cite_ref-47">^</a></b> ITA Labs <a href="http://semanticweb.com/more-semantics-for-google-ita-software-acquisition-comes-with-needlebase-too_b19329">"ITA Labs Acquisition"</a> 20 April 2011 1:28 AM
</li>
<li><b><a href="#cite_ref-48">^</a></b> <cite><a href="https://support.apple.com/en-us/HT204683">"About Applebot"</a>. Apple Inc. Retrieved 18 October 2021.</cite>
</li>
<li><b><a href="#cite_ref-Norton-2007-01-25_49-0">^</a></b> 
<cite>Norton, Quinn (25 January 2007). <a href="https://www.wired.com/2007/01/tax-takers-send-in-the-spiders/">"Tax takers send in the spiders"</a>. Business. <i><a href="/wiki/Wired_(magazine)">Wired</a></i>. <a href="https://web.archive.org/web/20161222185208/https://www.wired.com/2007/01/tax-takers-send-in-the-spiders/">Archived</a> from the original on 22 December 2016. Retrieved 13 October 2017.</cite>
</li>
<li><b><a href="#cite_ref-Canada-2017-04-11_50-0">^</a></b> 
<cite><a href="https://www.canada.ca/en/revenue-agency/services/about-canada-revenue-agency-cra/protecting-your-privacy/privacy-impact-assessment/xenon-web-crawling-initiative-privacy-impact-assessment-summary.html">"Xenon web crawling initiative: privacy impact assessment (PIA) summary"</a>. Ottawa: Government of Canada. 11 April 2017. <a href="https://web.archive.org/web/20170925203155/https://www.canada.ca/en/revenue-agency/services/about-canada-revenue-agency-cra/protecting-your-privacy/privacy-impact-assessment/xenon-web-crawling-initiative-privacy-impact-assessment-summary.html">Archived</a> from the original on 25 September 2017. Retrieved 13 October 2017.</cite>
</li>
</ol></div>
<h2>Further reading[<a href="/w/index.php?title=Web_crawler&amp;action=edit&amp;section=26">edit</a>]</h2>
<ul><li>Cho, Junghoo, <a href="http://oak.cs.ucla.edu/~cho/research/crawl.html">"Web Crawling Project"</a>, UCLA Computer Science Department.</li>
<li><a href="http://www.wiley.com/legacy/compbooks/sonnenreich/history.html">A History of Search Engines</a>, from <a href="/wiki/John_Wiley_%26_Sons">Wiley</a></li>
<li><a href="https://github.com/bedirhan/wivet">WIVET</a> is a benchmarking project by <a href="/wiki/OWASP">OWASP</a>, which aims to measure if a web crawler can identify all the hyperlinks in a target website.</li>
<li>Shestakov, Denis, <a href="https://www.slideshare.net/denshe/icwe13-tutorial-webcrawling">"Current Challenges in Web Crawling"</a> and <a href="https://www.slideshare.net/denshe/intelligent-crawling-shestakovwiiat13">"Intelligent Web Crawling"</a>, slides for tutorials given at ICWE'13 and WI-IAT'13.</li></ul>
</div>||||I|||| Jump to content
 Toggle sidebar
 Search
 * Create account
 Personal tools
 * Create account
 * Log in

 Pages for logged out editors learn more

 * Talk
 * Contributions
 Navigation
 * Main page
 * Contents
 * Current events
 * Random article
 * About Wikipedia
 * Contact us
 * Donate
 Contribute
 * Help
 * Learn to edit
 * Community portal
 * Recent changes
 * Upload file
 Tools
 * What links here
 * Related changes
 * Upload file
 * Special pages
 * Permanent link
 * Page information
 * Cite this page
 * Wikidata item
 Print/export
 * Download as PDF
 * Printable version
 Languages
 On this Wikipedia the language links are at the top of the page across from the article title. Go to top.

 Contents

 move to sidebar hide
 * (Top)
 * 1 Nomenclature
 * 2 Overview
 * 3 Crawling policy
 + 3.1 Selection policy
 o 3.1.1 Restricting followed links
 o 3.1.2 URL normalization
 o 3.1.3 Path-ascending crawling
 o 3.1.4 Focused crawling
 - 3.1.4.1 Academic focused crawler
 - 3.1.4.2 Semantic focused crawler
 + 3.2 Re-visit policy
 + 3.3 Politeness policy
 + 3.4 Parallelization policy
 * 4 Architectures
 * 5 Security
 * 6 Crawler identification
 * 7 Crawling the deep web
 * 8 Visual vs programmatic crawlers
 * 9 List of web crawlers
 + 9.1 Historical web crawlers
 + 9.2 In-house web crawlers
 + 9.3 Commercial web crawlers
 + 9.4 Open-source crawlers
 * 10 See also
 * 11 References
 * 12 Further reading
 Toggle the table of contents

 Web crawler

 42 languages
 * Afrikaans
 * العربية
 * Boarisch
 * Català
 * Čeština
 * Cymraeg
 * الدارجة
 * Deutsch
 * Ελληνικά
 * Español
 * Euskara
 * فارسی
 * Français
 * 한국어
 * Հայերեն
 * Hrvatski
 * Bahasa Indonesia
 * Italiano
 * עברית
 * Lietuvių
 * Magyar
 * Bahasa Melayu
 * Nederlands
 * Nedersaksies
 * 日本語
 * Norsk bokmål
 * Norsk nynorsk
 * Олык марий
 * Polski
 * Português
 * Română
 * Русский
 * Simple English
 * Српски / srpski
 * Suomi
 * Svenska
 * தமிழ்
 * ไทย
 * Türkçe
 * Українська
 * 文言
 * 中文
 Edit links
 * Article
 * Talk
 English
 * Read
 * Edit
 * View history
 More
 * Read
 * Edit
 * View history
 From Wikipedia, the free encyclopedia
 This article is about the internet bot. For the search engine, see WebCrawler . "Web spider" redirects here. Not to be confused with Spider web . "Spiderbot" redirects here. For the video game, see Arac (video game) .

 Architecture of a Web crawler

 A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).[1]

 Web search engines and some other websites use Web crawling or spidering software to update their web content or indices of other sites' web content. Web crawlers copy pages for processing by a search engine, which indexes the downloaded pages so that users can search more efficiently.

 Crawlers consume resources on visited systems and often visit sites unprompted. Issues of schedule, load, and "politeness" come into play when large collections of pages are accessed. Mechanisms exist for public sites not wishing to be crawled to make this known to the crawling agent. For example, including a robots.txt file can request bots to index only parts of a website, or nothing at all.

 The number of Internet pages is extremely large; even the largest crawlers fall short of making a complete index. For this reason, search engines struggled to give relevant search results in the early years of the World Wide Web, before 2000. Today, relevant results are given almost instantly.

 Crawlers can validate hyperlinks and HTML code. They can also be used for web scraping and data-driven programming.

 Nomenclature [ edit ]

 A web crawler is also known as a spider,[2] an ant, an automatic indexer,[3] or (in the FOAF software context) a Web scutter.[4]

 Overview [ edit ]

 A Web crawler starts with a list of URLs to visit. Those first URLs are called the seeds. As the crawler visits these URLs, by communicating with web servers that respond to those URLs, it identifies all the hyperlinks in the retrieved web pages and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies. If the crawler is performing archiving of websites (or web archiving), it copies and saves the information as it goes. The archives are usually stored in such a way they can be viewed, read and navigated as if they were on the live web, but are preserved as 'snapshots'.[5]

 The archive is known as the repository and is designed to store and manage the collection of web pages. The repository only stores HTML pages and these pages are stored as distinct files. A repository is similar to any other system that stores data, like a modern-day database. The only difference is that a repository does not need all the functionality offered by a database system. The repository stores the most recent version of the web page retrieved by the crawler.[ citation needed ]

 The large volume implies the crawler can only download a limited number of the Web pages within a given time, so it needs to prioritize its downloads. The high rate of change can imply the pages might have already been updated or even deleted.

 The number of possible URLs crawled being generated by server-side software has also made it difficult for web crawlers to avoid retrieving duplicate content. Endless combinations of HTTP GET (URL-based) parameters exist, of which only a small selection will actually return unique content. For example, a simple online photo gallery may offer three options to users, as specified through HTTP GET parameters in the URL. If there exist four ways to sort images, three choices of thumbnail size, two file formats, and an option to disable user-provided content, then the same set of content can be accessed with 48 different URLs, all of which may be linked on the site. This mathematical combination creates a problem for crawlers, as they must sort through endless combinations of relatively minor scripted changes in order to retrieve unique content.

 As Edwards et al. noted, "Given that the bandwidth for conducting crawls is neither infinite nor free, it is becoming essential to crawl the Web in not only a scalable, but efficient way, if some reasonable measure of quality or freshness is to be maintained."[6] A crawler must carefully choose at each step which pages to visit next.

 Crawling policy [ edit ]

 The behavior of a Web crawler is the outcome of a combination of policies:[7]

 * a selection policy which states the pages to download,
 * a re-visit policy which states when to check for changes to the pages,
 * a politeness policy that states how to avoid overloading Web sites.
 * a parallelization policy that states how to coordinate distributed web crawlers.

 Selection policy [ edit ]

 Given the current size of the Web, even large search engines cover only a portion of the publicly available part. A 2009 study showed even large-scale search engines index no more than 40–70% of the indexable Web;[8] a previous study by Steve Lawrence and Lee Giles showed that no search engine indexed more than 16% of the Web in 1999.[9] As a crawler always downloads just a fraction of the Web pages, it is highly desirable for the downloaded fraction to contain the most relevant pages and not just a random sample of the Web.

 This requires a metric of importance for prioritizing Web pages. The importance of a page is a function of its intrinsic quality, its popularity in terms of links or visits, and even of its URL (the latter is the case of vertical search engines restricted to a single top-level domain, or search engines restricted to a fixed Web site). Designing a good selection policy has an added difficulty: it must work with partial information, as the complete set of Web pages is not known during crawling.

 Junghoo Cho et al. made the first study on policies for crawling scheduling. Their data set was a 180,000-pages crawl from the stanford.edu domain, in which a crawling simulation was done with different strategies.[10] The ordering metrics tested were breadth-first, backlink count and partial PageRank calculations. One of the conclusions was that if the crawler wants to download pages with high Pagerank early during the crawling process, then the partial Pagerank strategy is the better, followed by breadth-first and backlink-count. However, these results are for just a single domain. Cho also wrote his PhD dissertation at Stanford on web crawling.[11]

 Najork and Wiener performed an actual crawl on 328 million pages, using breadth-first ordering.[12] They found that a breadth-first crawl captures pages with high Pagerank early in the crawl (but they did not compare this strategy against other strategies). The explanation given by the authors for this result is that "the most important pages have many links to them from numerous hosts, and those links will be found early, regardless of on which host or page the crawl originates."

 Abiteboul designed a crawling strategy based on an algorithm called OPIC (On-line Page Importance Computation).[13] In OPIC, each page is given an initial sum of "cash" that is distributed equally among the pages it points to. It is similar to a PageRank computation, but it is faster and is only done in one step. An OPIC-driven crawler downloads first the pages in the crawling frontier with higher amounts of "cash". Experiments were carried in a 100,000-pages synthetic graph with a power-law distribution of in-links. However, there was no comparison with other strategies nor experiments in the real Web.

 Boldi et al. used simulation on subsets of the Web of 40 million pages from the .it domain and 100 million pages from the WebBase crawl, testing breadth-first against depth-first, random ordering and an omniscient strategy. The comparison was based on how well PageRank computed on a partial crawl approximates the true PageRank value. Surprisingly, some visits that accumulate PageRank very quickly (most notably, breadth-first and the omniscient visit) provide very poor progressive approximations.[14][15]

 Baeza-Yates et al. used simulation on two subsets of the Web of 3 million pages from the .gr and .cl domain, testing several crawling strategies.[16] They showed that both the OPIC strategy and a strategy that uses the length of the per-site queues are better than breadth-first crawling, and that it is also very effective to use a previous crawl, when it is available, to guide the current one.

 Daneshpajouh et al. designed a community based algorithm for discovering good seeds.[17] Their method crawls web pages with high PageRank from different communities in less iteration in comparison with crawl starting from random seeds. One can extract good seed from a previously-crawled-Web graph using this new method. Using these seeds, a new crawl can be very effective.

 Restricting followed links [ edit ]

 A crawler may only want to seek out HTML pages and avoid all other MIME types. In order to request only HTML resources, a crawler may make an HTTP HEAD request to determine a Web resource's MIME type before requesting the entire resource with a GET request. To avoid making numerous HEAD requests, a crawler may examine the URL and only request a resource if the URL ends with certain characters such as .html, .htm, .asp, .aspx, .php, .jsp, .jspx or a slash. This strategy may cause numerous HTML Web resources to be unintentionally skipped.

 Some crawlers may also avoid requesting any resources that have a "?" in them (are dynamically produced) in order to avoid spider traps that may cause the crawler to download an infinite number of URLs from a Web site. This strategy is unreliable if the site uses URL rewriting to simplify its URLs.

 URL normalization [ edit ]

 Main article: URL normalization

 Crawlers usually perform some type of URL normalization in order to avoid crawling the same resource more than once. The term URL normalization, also called URL canonicalization, refers to the process of modifying and standardizing a URL in a consistent manner. There are several types of normalization that may be performed including conversion of URLs to lowercase, removal of "." and ".." segments, and adding trailing slashes to the non-empty path component.[18]

 Path-ascending crawling [ edit ]

 Some crawlers intend to download/upload as many resources as possible from a particular web site. So path-ascending crawler was introduced that would ascend to every path in each URL that it intends to crawl.[19] For example, when given a seed URL of http://llama.org/hamster/monkey/page.html, it will attempt to crawl /hamster/monkey/, /hamster/, and /. Cothey found that a path-ascending crawler was very effective in finding isolated resources, or resources for which no inbound link would have been found in regular crawling.

 Focused crawling [ edit ]

 Main article: Focused crawler

 The importance of a page for a crawler can also be expressed as a function of the similarity of a page to a given query. Web crawlers that attempt to download pages that are similar to each other are called focused crawler or topical crawlers. The concepts of topical and focused crawling were first introduced by Filippo Menczer[20][21] and by Soumen Chakrabarti et al.[22]

 The main problem in focused crawling is that in the context of a Web crawler, we would like to be able to predict the similarity of the text of a given page to the query before actually downloading the page. A possible predictor is the anchor text of links; this was the approach taken by Pinkerton[23] in the first web crawler of the early days of the Web. Diligenti et al.[24] propose using the complete content of the pages already visited to infer the similarity between the driving query and the pages that have not been visited yet. The performance of a focused crawling depends mostly on the richness of links in the specific topic being searched, and a focused crawling usually relies on a general Web search engine for providing starting points.

 Academic focused crawler [ edit ]

 An example of the focused crawlers are academic crawlers, which crawls free-access academic related documents, such as the citeseerxbot, which is the crawler of CiteSeerX search engine. Other academic search engines are Google Scholar and Microsoft Academic Search etc. Because most academic papers are published in PDF formats, such kind of crawler is particularly interested in crawling PDF, PostScript files, Microsoft Word including their zipped formats. Because of this, general open-source crawlers, such as Heritrix, must be customized to filter out other MIME types, or a middleware is used to extract these documents out and import them to the focused crawl database and repository.[25] Identifying whether these documents are academic or not is challenging and can add a significant overhead to the crawling process, so this is performed as a post crawling process using machine learning or regular expression algorithms. These academic documents are usually obtained from home pages of faculties and students or from publication page of research institutes. Because academic documents make up only a small fraction of all web pages, a good seed selection is important in boosting the efficiencies of these web crawlers.[26] Other academic crawlers may download plain text and HTML files, that contains metadata of academic papers, such as titles, papers, and abstracts. This increases the overall number of papers, but a significant fraction may not provide free PDF downloads.

 Semantic focused crawler [ edit ]

 Another type of focused crawlers is semantic focused crawler, which makes use of domain ontologies to represent topical maps and link Web pages with relevant ontological concepts for the selection and categorization purposes.[27] In addition, ontologies can be automatically updated in the crawling process. Dong et al.[28] introduced such an ontology-learning-based crawler using a support-vector machine to update the content of ontological concepts when crawling Web pages.

 Re-visit policy [ edit ]

 The Web has a very dynamic nature, and crawling a fraction of the Web can take weeks or months. By the time a Web crawler has finished its crawl, many events could have happened, including creations, updates, and deletions.

 From the search engine's point of view, there is a cost associated with not detecting an event, and thus having an outdated copy of a resource. The most-used cost functions are freshness and age.[29]

 Freshness: This is a binary measure that indicates whether the local copy is accurate or not. The freshness of a page p in the repository at time t is defined as:

 Age: This is a measure that indicates how outdated the local copy is. The age of a page p in the repository, at time t is defined as:

 Coffman et al. worked with a definition of the objective of a Web crawler that is equivalent to freshness, but use a different wording: they propose that a crawler must minimize the fraction of time pages remain outdated. They also noted that the problem of Web crawling can be modeled as a multiple-queue, single-server polling system, on which the Web crawler is the server and the Web sites are the queues. Page modifications are the arrival of the customers, and switch-over times are the interval between page accesses to a single Web site. Under this model, mean waiting time for a customer in the polling system is equivalent to the average age for the Web crawler.[30]

 The objective of the crawler is to keep the average freshness of pages in its collection as high as possible, or to keep the average age of pages as low as possible. These objectives are not equivalent: in the first case, the crawler is just concerned with how many pages are out-dated, while in the second case, the crawler is concerned with how old the local copies of pages are.

 Evolution of Freshness and Age in a web crawler

 Two simple re-visiting policies were studied by Cho and Garcia-Molina:[31]

 * Uniform policy: This involves re-visiting all pages in the collection with the same frequency, regardless of their rates of change.
 * Proportional policy: This involves re-visiting more often the pages that change more frequently. The visiting frequency is directly proportional to the (estimated) change frequency.

 In both cases, the repeated crawling order of pages can be done either in a random or a fixed order.

 Cho and Garcia-Molina proved the surprising result that, in terms of average freshness, the uniform policy outperforms the proportional policy in both a simulated Web and a real Web crawl. Intuitively, the reasoning is that, as web crawlers have a limit to how many pages they can crawl in a given time frame, (1) they will allocate too many new crawls to rapidly changing pages at the expense of less frequently updating pages, and (2) the freshness of rapidly changing pages lasts for shorter period than that of less frequently changing pages. In other words, a proportional policy allocates more resources to crawling frequently updating pages, but experiences less overall freshness time from them.

 To improve freshness, the crawler should penalize the elements that change too often.[32] The optimal re-visiting policy is neither the uniform policy nor the proportional policy. The optimal method for keeping average freshness high includes ignoring the pages that change too often, and the optimal for keeping average age low is to use access frequencies that monotonically (and sub-linearly) increase with the rate of change of each page. In both cases, the optimal is closer to the uniform policy than to the proportional policy: as Coffman et al. note, "in order to minimize the expected obsolescence time, the accesses to any particular page should be kept as evenly spaced as possible".[30] Explicit formulas for the re-visit policy are not attainable in general, but they are obtained numerically, as they depend on the distribution of page changes. Cho and Garcia-Molina show that the exponential distribution is a good fit for describing page changes,[32] while Ipeirotis et al. show how to use statistical tools to discover parameters that affect this distribution.[33] Note that the re-visiting policies considered here regard all pages as homogeneous in terms of quality ("all pages on the Web are worth the same"), something that is not a realistic scenario, so further information about the Web page quality should be included to achieve a better crawling policy.

 Politeness policy [ edit ]

 Crawlers can retrieve data much quicker and in greater depth than human searchers, so they can have a crippling impact on the performance of a site. If a single crawler is performing multiple requests per second and/or downloading large files, a server can have a hard time keeping up with requests from multiple crawlers.

 As noted by Koster, the use of Web crawlers is useful for a number of tasks, but comes with a price for the general community.[34] The costs of using Web crawlers include:

 * network resources, as crawlers require considerable bandwidth and operate with a high degree of parallelism during a long period of time;
 * server overload, especially if the frequency of accesses to a given server is too high;
 * poorly written crawlers, which can crash servers or routers, or which download pages they cannot handle; and
 * personal crawlers that, if deployed by too many users, can disrupt networks and Web servers.

 A partial solution to these problems is the robots exclusion protocol, also known as the robots.txt protocol that is a standard for administrators to indicate which parts of their Web servers should not be accessed by crawlers.[35] This standard does not include a suggestion for the interval of visits to the same server, even though this interval is the most effective way of avoiding server overload. Recently commercial search engines like Google, Ask Jeeves, MSN and Yahoo! Search are able to use an extra "Crawl-delay:" parameter in the robots.txt file to indicate the number of seconds to delay between requests.

 The first proposed interval between successive pageloads was 60 seconds.[36] However, if pages were downloaded at this rate from a website with more than 100,000 pages over a perfect connection with zero latency and infinite bandwidth, it would take more than 2 months to download only that entire Web site; also, only a fraction of the resources from that Web server would be used. This does not seem acceptable.

 Cho uses 10 seconds as an interval for accesses,[31] and the WIRE crawler uses 15 seconds as the default.[37] The MercatorWeb crawler follows an adaptive politeness policy: if it took t seconds to download a document from a given server, the crawler waits for 10t seconds before downloading the next page.[38] Dill et al. use 1 second.[39]

 For those using Web crawlers for research purposes, a more detailed cost-benefit analysis is needed and ethical considerations should be taken into account when deciding where to crawl and how fast to crawl.[40]

 Anecdotal evidence from access logs shows that access intervals from known crawlers vary between 20 seconds and 3–4 minutes. It is worth noticing that even when being very polite, and taking all the safeguards to avoid overloading Web servers, some complaints from Web server administrators are received. Brin and Page note that: "... running a crawler which connects to more than half a million servers ... generates a fair amount of e-mail and phone calls. Because of the vast number of people coming on line, there are always those who do not know what a crawler is, because this is the first one they have seen."[41]

 Parallelization policy [ edit ]

 Main article: Distributed web crawling

 A parallel crawler is a crawler that runs multiple processes in parallel. The goal is to maximize the download rate while minimizing the overhead from parallelization and to avoid repeated downloads of the same page. To avoid downloading the same page more than once, the crawling system requires a policy for assigning the new URLs discovered during the crawling process, as the same URL can be found by two different crawling processes.

 Architectures [ edit ]

 High-level architecture of a standard Web crawler

 A crawler must not only have a good crawling strategy, as noted in the previous sections, but it should also have a highly optimized architecture.

 Shkapenyuk and Suel noted that:[42]

 While it is fairly easy to build a slow crawler that downloads a few pages per second for a short period of time, building a high-performance system that can download hundreds of millions of pages over several weeks presents a number of challenges in system design, I/O and network efficiency, and robustness and manageability.

 Web crawlers are a central part of search engines, and details on their algorithms and architecture are kept as business secrets. When crawler designs are published, there is often an important lack of detail that prevents others from reproducing the work. There are also emerging concerns about "search engine spamming", which prevent major search engines from publishing their ranking algorithms.

 Security [ edit ]

 While most of the website owners are keen to have their pages indexed as broadly as possible to have strong presence in search engines, web crawling can also have unintended consequences and lead to a compromise or data breach if a search engine indexes resources that shouldn't be publicly available, or pages revealing potentially vulnerable versions of software.

 Main article: Google hacking

 Apart from standard web application security recommendations website owners can reduce their exposure to opportunistic hacking by only allowing search engines to index the public parts of their websites (with robots.txt) and explicitly blocking them from indexing transactional parts (login pages, private pages, etc.).

 Crawler identification [ edit ]

 Web crawlers typically identify themselves to a Web server by using the User-agent field of an HTTP request. Web site administrators typically examine their Web servers' log and use the user agent field to determine which crawlers have visited the web server and how often. The user agent field may include a URL where the Web site administrator may find out more information about the crawler. Examining Web server log is tedious task, and therefore some administrators use tools to identify, track and verify Web crawlers. Spambots and other malicious Web crawlers are unlikely to place identifying information in the user agent field, or they may mask their identity as a browser or other well-known crawler.

 Web site administrators prefer Web crawlers to identify themselves so that they can contact the owner if needed. In some cases, crawlers may be accidentally trapped in a crawler trap or they may be overloading a Web server with requests, and the owner needs to stop the crawler. Identification is also useful for administrators that are interested in knowing when they may expect their Web pages to be indexed by a particular search engine.

 Crawling the deep web [ edit ]

 A vast amount of web pages lie in the deep or invisible web.[43] These pages are typically only accessible by submitting queries to a database, and regular crawlers are unable to find these pages if there are no links that point to them. Google's Sitemaps protocol and mod oai[44] are intended to allow discovery of these deep-Web resources.

 Deep web crawling also multiplies the number of web links to be crawled. Some crawlers only take some of the URLs in <a href="URL"> form. In some cases, such as the Googlebot, Web crawling is done on all text contained inside the hypertext content, tags, or text.

 Strategic approaches may be taken to target deep Web content. With a technique called screen scraping, specialized software may be customized to automatically and repeatedly query a given Web form with the intention of aggregating the resulting data. Such software can be used to span multiple Web forms across multiple Websites. Data extracted from the results of one Web form submission can be taken and applied as input to another Web form thus establishing continuity across the Deep Web in a way not possible with traditional web crawlers.[45]

 Pages built on AJAX are among those causing problems to web crawlers. Google has proposed a format of AJAX calls that their bot can recognize and index.[46]

 Visual vs programmatic crawlers [ edit ]

 There are a number of "visual web scraper/crawler" products available on the web which will crawl pages and structure data into columns and rows based on the users requirements. One of the main difference between a classic and a visual crawler is the level of programming ability required to set up a crawler. The latest generation of "visual scrapers" remove the majority of the programming skill needed to be able to program and start a crawl to scrape web data.

 The visual scraping/crawling method relies on the user "teaching" a piece of crawler technology, which then follows patterns in semi-structured data sources. The dominant method for teaching a visual crawler is by highlighting data in a browser and training columns and rows. While the technology is not new, for example it was the basis of Needlebase which has been bought by Google (as part of a larger acquisition of ITA Labs[47]), there is continued growth and investment in this area by investors and end-users.[ citation needed ]

 List of web crawlers [ edit ]

 Further information: List of search engine software

 The following is a list of published crawler architectures for general-purpose crawlers (excluding focused web crawlers), with a brief description that includes the names given to the different components and outstanding features:

 Historical web crawlers [ edit ]

 * World Wide Web Worm was a crawler used to build a simple index of document titles and URLs. The index could be searched by using the grep Unix command.
 * Yahoo! Slurp was the name of the Yahoo! Search crawler until Yahoo! contracted with Microsoft to use Bingbot instead.

 In-house web crawlers [ edit ]

 * Applebot is Apple's web crawler. It supports Siri and other products.[48]
 * Bingbot is the name of Microsoft's Bing webcrawler. It replaced Msnbot.
 * Baiduspider is Baidu's web crawler.
 * Googlebot is described in some detail, but the reference is only about an early version of its architecture, which was written in C++ and Python. The crawler was integrated with the indexing process, because text parsing was done for full-text indexing and also for URL extraction. There is a URL server that sends lists of URLs to be fetched by several crawling processes. During parsing, the URLs found were passed to a URL server that checked if the URL have been previously seen. If not, the URL was added to the queue of the URL server.
 * WebCrawler was used to build the first publicly available full-text index of a subset of the Web. It was based on lib-WWW to download pages, and another program to parse and order URLs for breadth-first exploration of the Web graph. It also included a real-time crawler that followed links based on the similarity of the anchor text with the provided query.
 * WebFountain is a distributed, modular crawler similar to Mercator but written in C++.
 * Xenon is a web crawler used by government tax authorities to detect fraud.[49][50]

 Commercial web crawlers [ edit ]

 The following web crawlers are available, for a price::

 * Diffbot - programmatic general web crawler, available as an API
 * SortSite - crawler for analyzing websites, available for Windows and Mac OS
 * Swiftbot - Swiftype's web crawler, available as software as a service

 Open-source crawlers [ edit ]

 * GNU Wget is a command-line-operated crawler written in C and released under the GPL. It is typically used to mirror Web and FTP sites.
 * GRUB was an open source distributed search crawler that Wikia Search used to crawl the web.
 * Heritrix is the Internet Archive's archival-quality crawler, designed for archiving periodic snapshots of a large portion of the Web. It was written in Java.
 * ht://Dig includes a Web crawler in its indexing engine.
 * HTTrack uses a Web crawler to create a mirror of a web site for off-line viewing. It is written in C and released under the GPL.
 * mnoGoSearch is a crawler, indexer and a search engine written in C and licensed under the GPL (*NIX machines only)
 * Apache Nutch is a highly extensible and scalable web crawler written in Java and released under an Apache License. It is based on Apache Hadoop and can be used with Apache Solr or Elasticsearch.
 * Open Search Server is a search engine and web crawler software release under the GPL.
 * Scrapy, an open source webcrawler framework, written in python (licensed under BSD).
 * Seeks, a free distributed search engine (licensed under AGPL).
 * StormCrawler, a collection of resources for building low-latency, scalable web crawlers on Apache Storm (Apache License).
 * tkWWW Robot, a crawler based on the tkWWW web browser (licensed under GPL).
 * Xapian, a search crawler engine, written in c++.
 * YaCy, a free distributed search engine, built on principles of peer-to-peer networks (licensed under GPL).

 See also [ edit ]

 * Automatic indexing
 * Gnutella crawler
 * Web archiving
 * Webgraph
 * Website mirroring software
 * Search Engine Scraping
 * Web scraping

 References [ edit ]

 1. ^ "Web Crawlers: Browsing the Web". Archived from the original on 6 December 2021.
 2. ^ Spetka, Scott. "The TkWWW Robot: Beyond Browsing". NCSA. Archived from the original on 3 September 2004 . Retrieved 21 November 2010 .
 3. ^ Kobayashi, M. & Takeda, K. (2000). "Information retrieval on the web". ACM Computing Surveys. 32 (2): 144–173. CiteSeerX 10.1.1.126.6094. doi:10.1145/358923.358934. S2CID 3710903.
 4. ^ See definition of scutter on FOAF Project's wiki Archived 13 December 2009 at the Wayback Machine
 5. ^ Masanès, Julien (15 February 2007). Web Archiving. Springer. p. 1. ISBN 978-3-54046332-0 . Retrieved 24 April 2014 .
 6. ^ Edwards, J.; McCurley, K. S.; and Tomlin, J. A. (2001). "An adaptive model for optimizing performance of an incremental web crawler". Proceedings of the tenth international conference on World Wide Web - WWW '01. In Proceedings of the Tenth Conference on World Wide Web. pp. 106–113. CiteSeerX 10.1.1.1018.1506. doi:10.1145/371920.371960. ISBN 978-1581133486. S2CID 10316730.{{cite book}} : CS1 maint: multiple names: authors list ( link )
 7. ^ Castillo, Carlos (2004). Effective Web Crawling (PhD thesis). University of Chile . Retrieved 3 August 2010 .
 8. ^ Gulls, A.; A. Signori (2005). "The indexable web is more than 11.5 billion pages". Special interest tracks and posters of the 14th international conference on World Wide Web. ACM Press. pp. 902–903. doi:10.1145/1062745.1062789.
 9. ^ Lawrence, Steve; C. Lee Giles (8 July 1999). "Accessibility of information on the web". Nature. 400 (6740): 107–9. Bibcode:1999Natur.400..107L. doi:10.1038/21987. PMID 10428673. S2CID 4347646.
 10. ^ Cho, J.; Garcia-Molina, H.; Page, L. (April 1998). "Efficient Crawling Through URL Ordering". Seventh International World-Wide Web Conference. Brisbane, Australia. doi:10.1142/3725. ISBN 978-981-02-3400-3 . Retrieved 23 March 2009 .
 11. ^ Cho, Junghoo, "Crawling the Web: Discovery and Maintenance of a Large-Scale Web Data" , PhD dissertation, Department of Computer Science, Stanford University, November 2001.
 12. ^ Najork, Marc and Janet L. Wiener. "Breadth-first crawling yields high-quality pages". In: Proceedings of the Tenth Conference on World Wide Web , pages 114–118, Hong Kong, May 2001. Elsevier Science.
 13. ^ Abiteboul, Serge; Mihai Preda; Gregory Cobena (2003). "Adaptive on-line page importance computation". Proceedings of the 12th international conference on World Wide Web. Budapest, Hungary: ACM. pp. 280–290. doi:10.1145/775152.775192. ISBN 1-58113-680-3 . Retrieved 22 March 2009 .
 14. ^ Boldi, Paolo; Bruno Codenotti; Massimo Santini; Sebastiano Vigna (2004). "UbiCrawler: a scalable fully distributed Web crawler" (PDF) . Software: Practice and Experience. 34 (8): 711–726. CiteSeerX 10.1.1.2.5538. doi:10.1002/spe.587. S2CID 325714. Archived from the original (PDF) on 20 March 2009 . Retrieved 23 March 2009 .
 15. ^ Boldi, Paolo; Massimo Santini; Sebastiano Vigna (2004). "Do Your Worst to Make the Best: Paradoxical Effects in PageRank Incremental Computations" (PDF) . Algorithms and Models for the Web-Graph. Lecture Notes in Computer Science. Vol. 3243. pp. 168–180. doi:10.1007/978-3-540-30216-2_14. ISBN 978-3-540-23427-2 . Retrieved 23 March 2009 .
 16. ^ Baeza-Yates, R.; Castillo, C.; Marin, M. and Rodriguez, A. (2005). "Crawling a Country: Better Strategies than Breadth-First for Web Page Ordering." In: Proceedings of the Industrial and Practical Experience track of the 14th conference on World Wide Web , pages 864–872, Chiba, Japan. ACM Press.
 17. ^ Shervin Daneshpajouh, Mojtaba Mohammadi Nasiri, Mohammad Ghodsi, A Fast Community Based Algorithm for Generating Crawler Seeds Set . In: Proceedings of 4th International Conference on Web Information Systems and Technologies ( Webist -2008), Funchal, Portugal, May 2008.
 18. ^ Pant, Gautam; Srinivasan, Padmini; Menczer, Filippo (2004). "Crawling the Web" (PDF) . In Levene, Mark; Poulovassilis, Alexandra (eds.). Web Dynamics: Adapting to Change in Content, Size, Topology and Use. Springer. pp. 153–178. ISBN 978-3-540-40676-1. Archived from the original (PDF) on 20 March 2009 . Retrieved 9 May 2006 .
 19. ^ Cothey, Viv (2004). "Web-crawling reliability" (PDF) . Journal of the American Society for Information Science and Technology. 55 (14): 1228–1238. CiteSeerX 10.1.1.117.185. doi:10.1002/asi.20078.
 20. ^ Menczer, F. (1997). ARACHNID: Adaptive Retrieval Agents Choosing Heuristic Neighborhoods for Information Discovery Archived 21 December 2012 at the Wayback Machine . In D. Fisher, ed., Machine Learning: Proceedings of the 14th International Conference (ICML97). Morgan Kaufmann
 21. ^ Menczer, F. and Belew, R.K. (1998). Adaptive Information Agents in Distributed Textual Environments Archived 21 December 2012 at the Wayback Machine . In K. Sycara and M. Wooldridge (eds.) Proc. 2nd Intl. Conf. on Autonomous Agents (Agents '98). ACM Press
 22. ^ Chakrabarti, Soumen; Van Den Berg, Martin; Dom, Byron (1999). "Focused crawling: A new approach to topic-specific Web resource discovery" (PDF) . Computer Networks. 31 (11–16): 1623–1640. doi:10.1016/s1389-1286(99)00052-3. Archived from the original (PDF) on 17 March 2004.
 23. ^ Pinkerton, B. (1994). Finding what people want: Experiences with the WebCrawler . In Proceedings of the First World Wide Web Conference, Geneva, Switzerland.
 24. ^ Diligenti, M., Coetzee, F., Lawrence, S., Giles, C. L., and Gori, M. (2000). Focused crawling using context graphs . In Proceedings of 26th International Conference on Very Large Databases (VLDB), pages 527-534, Cairo, Egypt.
 25. ^ Wu, Jian; Teregowda, Pradeep; Khabsa, Madian; Carman, Stephen; Jordan, Douglas; San Pedro Wandelmer, Jose; Lu, Xin; Mitra, Prasenjit; Giles, C. Lee (2012). "Web crawler middleware for search engine digital libraries". Proceedings of the twelfth international workshop on Web information and data management - WIDM '12. p. 57. doi:10.1145/2389936.2389949. ISBN 9781450317207. S2CID 18513666.
 26. ^ Wu, Jian; Teregowda, Pradeep; Ramírez, Juan Pablo Fernández; Mitra, Prasenjit; Zheng, Shuyi; Giles, C. Lee (2012). "The evolution of a crawling strategy for an academic document search engine". Proceedings of the 3rd Annual ACM Web Science Conference on - Web Sci '12. pp. 340–343. doi:10.1145/2380718.2380762. ISBN 9781450312288. S2CID 16718130.
 27. ^ Dong, Hai; Hussain, Farookh Khadeer; Chang, Elizabeth (2009). "State of the Art in Semantic Focused Crawlers". Computational Science and Its Applications – ICCSA 2009. Lecture Notes in Computer Science. Vol. 5593. pp. 910–924. doi:10.1007/978-3-642-02457-3_74. hdl:20.500.11937/48288. ISBN 978-3-642-02456-6.
 28. ^ Dong, Hai; Hussain, Farookh Khadeer (2013). "SOF: A semi-supervised ontology-learning-based focused crawler". Concurrency and Computation: Practice and Experience. 25 (12): 1755–1770. doi:10.1002/cpe.2980. S2CID 205690364.
 29. ^ Junghoo Cho; Hector Garcia-Molina (2000). "Synchronizing a database to improve freshness" (PDF) . Proceedings of the 2000 ACM SIGMOD international conference on Management of data. Dallas, Texas, United States: ACM. pp. 117–128. doi:10.1145/342009.335391. ISBN 1-58113-217-4 . Retrieved 23 March 2009 .
 30. ^ a b E. G. Coffman Jr; Zhen Liu; Richard R. Weber (1998). "Optimal robot scheduling for Web search engines". Journal of Scheduling. 1 (1): 15–29. CiteSeerX 10.1.1.36.6087. doi:10.1002/(SICI)1099-1425(199806)1:1<15::AID-JOS3>3.0.CO;2-K.
 31. ^ a b Cho, Junghoo; Garcia-Molina, Hector (2003). "Effective page refresh policies for Web crawlers". ACM Transactions on Database Systems. 28 (4): 390–426. doi:10.1145/958942.958945. S2CID 147958.
 32. ^ a b Junghoo Cho; Hector Garcia-Molina (2003). "Estimating frequency of change". ACM Transactions on Internet Technology. 3 (3): 256–290. CiteSeerX 10.1.1.59.5877. doi:10.1145/857166.857170. S2CID 9362566.
 33. ^ Ipeirotis, P., Ntoulas, A., Cho, J., Gravano, L. (2005) Modeling and managing content changes in text databases . In Proceedings of the 21st IEEE International Conference on Data Engineering, pages 606-617, April 2005, Tokyo.
 34. ^ Koster, M. (1995). Robots in the web: threat or treat? ConneXions, 9(4).
 35. ^ Koster, M. (1996). A standard for robot exclusion Archived 7 November 2007 at the Wayback Machine .
 36. ^ Koster, M. (1993). Guidelines for robots writers Archived 22 April 2005 at the Wayback Machine .
 37. ^ Baeza-Yates, R. and Castillo, C. (2002). Balancing volume, quality and freshness in Web crawling . In Soft Computing Systems – Design, Management and Applications, pages 565–572, Santiago, Chile. IOS Press Amsterdam.
 38. ^ Heydon, Allan; Najork, Marc (26 June 1999). "Mercator: A Scalable, Extensible Web Crawler" (PDF) . Archived from the original (PDF) on 19 February 2006 . Retrieved 22 March 2009 . {{cite journal}} : Cite journal requires |journal= ( help )
 39. ^ Dill, S.; Kumar, R.; Mccurley, K. S.; Rajagopalan, S.; Sivakumar, D.; Tomkins, A. (2002). "Self-similarity in the web" (PDF) . ACM Transactions on Internet Technology. 2 (3): 205–223. doi:10.1145/572326.572328. S2CID 6416041.
 40. ^ M. Thelwall; D. Stuart (2006). "Web crawling ethics revisited: Cost, privacy and denial of service". Journal of the American Society for Information Science and Technology. 57 (13): 1771–1779. doi:10.1002/asi.20388.
 41. ^ Brin, Sergey; Page, Lawrence (1998). "The anatomy of a large-scale hypertextual Web search engine". Computer Networks and ISDN Systems. 30 (1–7): 107–117. doi:10.1016/s0169-7552(98)00110-x.
 42. ^ Shkapenyuk, V. and Suel, T. (2002). Design and implementation of a high performance distributed web crawler . In Proceedings of the 18th International Conference on Data Engineering (ICDE), pages 357-368, San Jose, California. IEEE CS Press.
 43. ^ Shestakov, Denis (2008). Search Interfaces on the Web: Querying and Characterizing Archived 6 July 2014 at the Wayback Machine . TUCS Doctoral Dissertations 104, University of Turku
 44. ^ Michael L Nelson; Herbert Van de Sompel; Xiaoming Liu; Terry L Harrison; Nathan McFarland (24 March 2005). "mod_oai: An Apache Module for Metadata Harvesting": cs/0503069. arXiv:cs/0503069. Bibcode:2005cs........3069N. {{cite journal}} : Cite journal requires |journal= ( help )
 45. ^ Shestakov, Denis; Bhowmick, Sourav S.; Lim, Ee-Peng (2005). "DEQUE: Querying the Deep Web" (PDF) . Data & Knowledge Engineering. 52 (3): 273–311. doi:10.1016/s0169-023x(04)00107-7.
 46. ^ "AJAX crawling: Guide for webmasters and developers" . Retrieved 17 March 2013 .
 47. ^ ITA Labs "ITA Labs Acquisition" 20 April 2011 1:28 AM
 48. ^ "About Applebot". Apple Inc . Retrieved 18 October 2021 .
 49. ^ Norton, Quinn (25 January 2007). "Tax takers send in the spiders". Business. Wired. Archived from the original on 22 December 2016 . Retrieved 13 October 2017 .
 50. ^ "Xenon web crawling initiative: privacy impact assessment (PIA) summary". Ottawa: Government of Canada. 11 April 2017. Archived from the original on 25 September 2017 . Retrieved 13 October 2017 .

 Further reading [ edit ]

 * Cho, Junghoo, "Web Crawling Project", UCLA Computer Science Department.
 * A History of Search Engines, from Wiley
 * WIVET is a benchmarking project by OWASP, which aims to measure if a web crawler can identify all the hyperlinks in a target website.
 * Shestakov, Denis, "Current Challenges in Web Crawling" and "Intelligent Web Crawling", slides for tutorials given at ICWE'13 and WI-IAT'13.
 * v 
 * t 
 * e 
 Internet search
 * Web search engine (List) 
 * Metasearch engine 
 * Multimedia search 
 * Collaborative search engine 
 * Cross-language search 
 * Local search 
 * Vertical search 
 Types * Social search 
 * Image search 
 * Audio search 
 * Video search engine 
 * Enterprise search 
 * Semantic search 
 * Natural language search engine 
 * Voice search 
 * Search by sound 
 * Search engine marketing 
 * Search engine optimization 
 * Evaluation measures 
 * Search oriented architecture 
 * Selection-based search 
 * Document retrieval 
 * Text mining 
 * Web crawler 
 * Multisearch 
 Tools * Federated search 
 * Search aggregator 
 * Index/Web indexing 
 * Focused crawler 
 * Spider trap 
 * Robots exclusion standard 
 * Distributed web crawling 
 * Web archiving 
 * Website mirroring software 
 * Web query 
 * Web query classification 
 * Z39.50 
 * Search/Retrieve Web Service 
 Protocols * Search/Retrieve via URL 
 and standards * OpenSearch 
 * Representational State Transfer
 * Website Parse Template 
 * Wide area information server 
 * Search engine 
 See also * Desktop search 
 * Online search 
 
 * v 
 * t 
 * e 
 Web crawlers 
 Internet bots designed for Web crawling and Web indexing
  * 80legs 
  * bingbot 
  * Fetcher 
  * Googlebot 
 Active * Heritrix 
  * HTTrack 
  * PHP-Crawler 
  * PowerMapper 
  * Wget 
  * FAST Crawler 
  * msnbot 
 Discontinued * RBSE 
  * TkWWW robot 
  * Twiceler 
 Types * Distributed web crawler
  * Focused crawler 
 
 Authority control: National libraries * Germany
 
 Retrieved from "https://en.wikipedia.org/w/index.php?title=Web_crawler&oldid=1137648114"
 Categories:
 * Search engine software
 * Web crawlers
 * Internet search algorithms
 Hidden categories:
 * Webarchive template wayback links
 * CS1 maint: multiple names: authors list
 * CS1 errors: missing periodical
 * Articles with short description
 * Short description is different from Wikidata
 * Use dmy dates from September 2020
 * All articles with unsourced statements
 * Articles with unsourced statements from February 2023
 * Articles with unsourced statements from June 2021
 * Articles with GND identifiers
 * This page was last edited on 5 February 2023, at 19:18 (UTC) .
 * Text is available under the Creative Commons Attribution-ShareAlike License 3.0; additional terms may apply. By using this site, you agree to the Terms of Use and Privacy Policy. Wikipedia® is a registered trademark of the Wikimedia Foundation, Inc., a non-profit organization.
 * Privacy policy
 * About Wikipedia
 * Disclaimers
 * Contact Wikipedia
 * Mobile view
 * Developers
 * Statistics
 * Cookie statement
 * 
 * 
This article is about the internet bot. For the search engine, see WebCrawler. "Web spider" redirects here. Not to be confused with Spider web. "Spiderbot" redirects here. For the video game, see Arac (video game).
      Architecture of a Web crawler  
A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering). [1] 
 Web search engines and some other websites use Web crawling or spidering software to update their web content or indices of other sites' web content. Web crawlers copy pages for processing by a search engine, which indexes the downloaded pages so that users can search more efficiently.
 Crawlers consume resources on visited systems and often visit sites unprompted. Issues of schedule, load, and "politeness" come into play when large collections of pages are accessed. Mechanisms exist for public sites not wishing to be crawled to make this known to the crawling agent. For example, including a  robots.txt  file can request bots to index only parts of a website, or nothing at all.
 The number of Internet pages is extremely large; even the largest crawlers fall short of making a complete index. For this reason, search engines struggled to give relevant search results in the early years of the World Wide Web, before 2000. Today, relevant results are given almost instantly.
 Crawlers can validate hyperlinks and HTML code. They can also be used for web scraping and data-driven programming.
Nomenclature[edit]
A web crawler is also known as a spider, [2]  an ant, an automatic indexer, [3]  or (in the FOAF software context) a Web scutter. [4] 
Overview[edit]
A Web crawler starts with a list of URLs to visit. Those first URLs are called the seeds. As the crawler visits these URLs, by communicating with web servers that respond to those URLs, it identifies all the hyperlinks in the retrieved web pages and adds them to the list of URLs to visit, called the  crawl frontier . URLs from the frontier are recursively visited according to a set of policies. If the crawler is performing archiving of websites (or web archiving), it copies and saves the information as it goes. The archives are usually stored in such a way they can be viewed, read and navigated as if they were on the live web, but are preserved as 'snapshots'. [5] 
 The archive is known as the repository and is designed to store and manage the collection of web pages. The repository only stores HTML pages and these pages are stored as distinct files. A repository is similar to any other system that stores data, like a modern-day database. The only difference is that a repository does not need all the functionality offered by a database system. The repository stores the most recent version of the web page retrieved by the crawler.[ citation needed ]
 The large volume implies the crawler can only download a limited number of the Web pages within a given time, so it needs to prioritize its downloads. The high rate of change can imply the pages might have already been updated or even deleted.
 The number of possible URLs crawled being generated by server-side software has also made it difficult for web crawlers to avoid retrieving duplicate content. Endless combinations of HTTP GET (URL-based) parameters exist, of which only a small selection will actually return unique content. For example, a simple online photo gallery may offer three options to users, as specified through HTTP GET parameters in the URL. If there exist four ways to sort images, three choices of thumbnail size, two file formats, and an option to disable user-provided content, then the same set of content can be accessed with 48 different URLs, all of which may be linked on the site. This mathematical combination creates a problem for crawlers, as they must sort through endless combinations of relatively minor scripted changes in order to retrieve unique content.
 As Edwards et al. noted, "Given that the bandwidth for conducting crawls is neither infinite nor free, it is becoming essential to crawl the Web in not only a scalable, but efficient way, if some reasonable measure of quality or freshness is to be maintained." [6]  A crawler must carefully choose at each step which pages to visit next.
Crawling policy[edit]
The behavior of a Web crawler is the outcome of a combination of policies: [7] 
 a selection policy which states the pages to download,
a re-visit policy which states when to check for changes to the pages,
a politeness policy that states how to avoid overloading Web sites.
a parallelization policy that states how to coordinate distributed web crawlers. 
Selection policy[edit]
Given the current size of the Web, even large search engines cover only a portion of the publicly available part. A 2009 study showed even large-scale search engines index no more than 40–70% of the indexable Web; [8]  a previous study by Steve Lawrence and Lee Giles showed that no search engine indexed more than 16% of the Web in 1999. [9]  As a crawler always downloads just a fraction of the Web pages, it is highly desirable for the downloaded fraction to contain the most relevant pages and not just a random sample of the Web.
 This requires a metric of importance for prioritizing Web pages. The importance of a page is a function of its intrinsic quality, its popularity in terms of links or visits, and even of its URL (the latter is the case of vertical search engines restricted to a single top-level domain, or search engines restricted to a fixed Web site). Designing a good selection policy has an added difficulty: it must work with partial information, as the complete set of Web pages is not known during crawling.
 Junghoo Cho et al. made the first study on policies for crawling scheduling. Their data set was a 180,000-pages crawl from the stanford.edu domain, in which a crawling simulation was done with different strategies. [10]  The ordering metrics tested were breadth-first, backlink count and partial PageRank calculations. One of the conclusions was that if the crawler wants to download pages with high Pagerank early during the crawling process, then the partial Pagerank strategy is the better, followed by breadth-first and backlink-count. However, these results are for just a single domain. Cho also wrote his PhD dissertation at Stanford on web crawling. [11] 
 Najork and Wiener performed an actual crawl on 328 million pages, using breadth-first ordering. [12]  They found that a breadth-first crawl captures pages with high Pagerank early in the crawl (but they did not compare this strategy against other strategies). The explanation given by the authors for this result is that "the most important pages have many links to them from numerous hosts, and those links will be found early, regardless of on which host or page the crawl originates."
 Abiteboul designed a crawling strategy based on an algorithm called OPIC (On-line Page Importance Computation). [13]  In OPIC, each page is given an initial sum of "cash" that is distributed equally among the pages it points to. It is similar to a PageRank computation, but it is faster and is only done in one step. An OPIC-driven crawler downloads first the pages in the crawling frontier with higher amounts of "cash". Experiments were carried in a 100,000-pages synthetic graph with a power-law distribution of in-links. However, there was no comparison with other strategies nor experiments in the real Web.
 Boldi et al. used simulation on subsets of the Web of 40 million pages from the .it domain and 100 million pages from the WebBase crawl, testing breadth-first against depth-first, random ordering and an omniscient strategy. The comparison was based on how well PageRank computed on a partial crawl approximates the true PageRank value. Surprisingly, some visits that accumulate PageRank very quickly (most notably, breadth-first and the omniscient visit) provide very poor progressive approximations. [14]   [15] 
 Baeza-Yates et al. used simulation on two subsets of the Web of 3 million pages from the .gr and .cl domain, testing several crawling strategies. [16]  They showed that both the OPIC strategy and a strategy that uses the length of the per-site queues are better than breadth-first crawling, and that it is also very effective to use a previous crawl, when it is available, to guide the current one.
 Daneshpajouh et al. designed a community based algorithm for discovering good seeds. [17]  Their method crawls web pages with high PageRank from different communities in less iteration in comparison with crawl starting from random seeds. One can extract good seed from a previously-crawled-Web graph using this new method. Using these seeds, a new crawl can be very effective.
Restricting followed links[edit]
A crawler may only want to seek out HTML pages and avoid all other MIME types. In order to request only HTML resources, a crawler may make an HTTP HEAD request to determine a Web resource's MIME type before requesting the entire resource with a GET request. To avoid making numerous HEAD requests, a crawler may examine the URL and only request a resource if the URL ends with certain characters such as .html, .htm, .asp, .aspx, .php, .jsp, .jspx or a slash. This strategy may cause numerous HTML Web resources to be unintentionally skipped.
 Some crawlers may also avoid requesting any resources that have a "?" in them (are dynamically produced) in order to avoid spider traps that may cause the crawler to download an infinite number of URLs from a Web site. This strategy is unreliable if the site uses URL rewriting to simplify its URLs.
URL normalization[edit]
Crawlers usually perform some type of URL normalization in order to avoid crawling the same resource more than once. The term URL normalization, also called URL canonicalization, refers to the process of modifying and standardizing a URL in a consistent manner. There are several types of normalization that may be performed including conversion of URLs to lowercase, removal of "." and ".." segments, and adding trailing slashes to the non-empty path component. [18] 
Path-ascending crawling[edit]
Some crawlers intend to download/upload as many resources as possible from a particular web site. So path-ascending crawler was introduced that would ascend to every path in each URL that it intends to crawl. [19]  For example, when given a seed URL of http://llama.org/hamster/monkey/page.html, it will attempt to crawl /hamster/monkey/, /hamster/, and /. Cothey found that a path-ascending crawler was very effective in finding isolated resources, or resources for which no inbound link would have been found in regular crawling.
Focused crawling[edit]
The importance of a page for a crawler can also be expressed as a function of the similarity of a page to a given query. Web crawlers that attempt to download pages that are similar to each other are called focused crawler or topical crawlers. The concepts of topical and focused crawling were first introduced by Filippo Menczer  [20]   [21]  and by Soumen Chakrabarti et al.  [22] 
 The main problem in focused crawling is that in the context of a Web crawler, we would like to be able to predict the similarity of the text of a given page to the query before actually downloading the page. A possible predictor is the anchor text of links; this was the approach taken by Pinkerton [23]  in the first web crawler of the early days of the Web. Diligenti et al.  [24]  propose using the complete content of the pages already visited to infer the similarity between the driving query and the pages that have not been visited yet. The performance of a focused crawling depends mostly on the richness of links in the specific topic being searched, and a focused crawling usually relies on a general Web search engine for providing starting points.
Academic focused crawler[edit]
An example of the focused crawlers are academic crawlers, which crawls free-access academic related documents, such as the citeseerxbot, which is the crawler of CiteSeer X search engine. Other academic search engines are Google Scholar and Microsoft Academic Search etc. Because most academic papers are published in PDF formats, such kind of crawler is particularly interested in crawling PDF, PostScript files, Microsoft Word including their zipped formats. Because of this, general open-source crawlers, such as Heritrix, must be customized to filter out other MIME types, or a middleware is used to extract these documents out and import them to the focused crawl database and repository. [25]  Identifying whether these documents are academic or not is challenging and can add a significant overhead to the crawling process, so this is performed as a post crawling process using machine learning or regular expression algorithms. These academic documents are usually obtained from home pages of faculties and students or from publication page of research institutes. Because academic documents make up only a small fraction of all web pages, a good seed selection is important in boosting the efficiencies of these web crawlers. [26]  Other academic crawlers may download plain text and HTML files, that contains metadata of academic papers, such as titles, papers, and abstracts. This increases the overall number of papers, but a significant fraction may not provide free PDF downloads.
Semantic focused crawler[edit]
Another type of focused crawlers is semantic focused crawler, which makes use of domain ontologies to represent topical maps and link Web pages with relevant ontological concepts for the selection and categorization purposes. [27]  In addition, ontologies can be automatically updated in the crawling process. Dong et al. [28]  introduced such an ontology-learning-based crawler using a support-vector machine to update the content of ontological concepts when crawling Web pages.
Re-visit policy[edit]
The Web has a very dynamic nature, and crawling a fraction of the Web can take weeks or months. By the time a Web crawler has finished its crawl, many events could have happened, including creations, updates, and deletions.
 From the search engine's point of view, there is a cost associated with not detecting an event, and thus having an outdated copy of a resource. The most-used cost functions are freshness and age. [29] 
  Freshness: This is a binary measure that indicates whether the local copy is accurate or not. The freshness of a page p in the repository at time t is defined as:
   
 Age: This is a measure that indicates how outdated the local copy is. The age of a page p in the repository, at time t is defined as:
   
 Coffman et al. worked with a definition of the objective of a Web crawler that is equivalent to freshness, but use a different wording: they propose that a crawler must minimize the fraction of time pages remain outdated. They also noted that the problem of Web crawling can be modeled as a multiple-queue, single-server polling system, on which the Web crawler is the server and the Web sites are the queues. Page modifications are the arrival of the customers, and switch-over times are the interval between page accesses to a single Web site. Under this model, mean waiting time for a customer in the polling system is equivalent to the average age for the Web crawler. [30] 
 The objective of the crawler is to keep the average freshness of pages in its collection as high as possible, or to keep the average age of pages as low as possible. These objectives are not equivalent: in the first case, the crawler is just concerned with how many pages are out-dated, while in the second case, the crawler is concerned with how old the local copies of pages are.
      Evolution of Freshness and Age in a web crawler  
Two simple re-visiting policies were studied by Cho and Garcia-Molina: [31] 
 Uniform policy: This involves re-visiting all pages in the collection with the same frequency, regardless of their rates of change.
Proportional policy: This involves re-visiting more often the pages that change more frequently. The visiting frequency is directly proportional to the (estimated) change frequency. 
In both cases, the repeated crawling order of pages can be done either in a random or a fixed order.
 Cho and Garcia-Molina proved the surprising result that, in terms of average freshness, the uniform policy outperforms the proportional policy in both a simulated Web and a real Web crawl. Intuitively, the reasoning is that, as web crawlers have a limit to how many pages they can crawl in a given time frame, (1) they will allocate too many new crawls to rapidly changing pages at the expense of less frequently updating pages, and (2) the freshness of rapidly changing pages lasts for shorter period than that of less frequently changing pages. In other words, a proportional policy allocates more resources to crawling frequently updating pages, but experiences less overall freshness time from them.
 To improve freshness, the crawler should penalize the elements that change too often. [32]  The optimal re-visiting policy is neither the uniform policy nor the proportional policy. The optimal method for keeping average freshness high includes ignoring the pages that change too often, and the optimal for keeping average age low is to use access frequencies that monotonically (and sub-linearly) increase with the rate of change of each page. In both cases, the optimal is closer to the uniform policy than to the proportional policy: as Coffman et al. note, "in order to minimize the expected obsolescence time, the accesses to any particular page should be kept as evenly spaced as possible". [30]  Explicit formulas for the re-visit policy are not attainable in general, but they are obtained numerically, as they depend on the distribution of page changes. Cho and Garcia-Molina show that the exponential distribution is a good fit for describing page changes, [32]  while Ipeirotis et al. show how to use statistical tools to discover parameters that affect this distribution. [33]  Note that the re-visiting policies considered here regard all pages as homogeneous in terms of quality ("all pages on the Web are worth the same"), something that is not a realistic scenario, so further information about the Web page quality should be included to achieve a better crawling policy.
Politeness policy[edit]
Crawlers can retrieve data much quicker and in greater depth than human searchers, so they can have a crippling impact on the performance of a site. If a single crawler is performing multiple requests per second and/or downloading large files, a server can have a hard time keeping up with requests from multiple crawlers.
 As noted by Koster, the use of Web crawlers is useful for a number of tasks, but comes with a price for the general community. [34]  The costs of using Web crawlers include:
 network resources, as crawlers require considerable bandwidth and operate with a high degree of parallelism during a long period of time;
server overload, especially if the frequency of accesses to a given server is too high;
poorly written crawlers, which can crash servers or routers, or which download pages they cannot handle; and
personal crawlers that, if deployed by too many users, can disrupt networks and Web servers. 
A partial solution to these problems is the robots exclusion protocol, also known as the robots.txt protocol that is a standard for administrators to indicate which parts of their Web servers should not be accessed by crawlers. [35]  This standard does not include a suggestion for the interval of visits to the same server, even though this interval is the most effective way of avoiding server overload. Recently commercial search engines like Google, Ask Jeeves, MSN and Yahoo! Search are able to use an extra "Crawl-delay:" parameter in the robots.txt file to indicate the number of seconds to delay between requests.
 The first proposed interval between successive pageloads was 60 seconds. [36]  However, if pages were downloaded at this rate from a website with more than 100,000 pages over a perfect connection with zero latency and infinite bandwidth, it would take more than 2 months to download only that entire Web site; also, only a fraction of the resources from that Web server would be used. This does not seem acceptable.
 Cho uses 10 seconds as an interval for accesses, [31]  and the WIRE crawler uses 15 seconds as the default. [37]  The MercatorWeb crawler follows an adaptive politeness policy: if it took t seconds to download a document from a given server, the crawler waits for 10t seconds before downloading the next page. [38]  Dill et al. use 1 second. [39] 
 For those using Web crawlers for research purposes, a more detailed cost-benefit analysis is needed and ethical considerations should be taken into account when deciding where to crawl and how fast to crawl. [40] 
 Anecdotal evidence from access logs shows that access intervals from known crawlers vary between 20 seconds and 3–4 minutes. It is worth noticing that even when being very polite, and taking all the safeguards to avoid overloading Web servers, some complaints from Web server administrators are received. Brin and Page note that: "... running a crawler which connects to more than half a million servers ... generates a fair amount of e-mail and phone calls. Because of the vast number of people coming on line, there are always those who do not know what a crawler is, because this is the first one they have seen." [41] 
Parallelization policy[edit]
A parallel crawler is a crawler that runs multiple processes in parallel. The goal is to maximize the download rate while minimizing the overhead from parallelization and to avoid repeated downloads of the same page. To avoid downloading the same page more than once, the crawling system requires a policy for assigning the new URLs discovered during the crawling process, as the same URL can be found by two different crawling processes.
Architectures[edit]
      High-level architecture of a standard Web crawler  
A crawler must not only have a good crawling strategy, as noted in the previous sections, but it should also have a highly optimized architecture.
 Shkapenyuk and Suel noted that: [42] 
 While it is fairly easy to build a slow crawler that downloads a few pages per second for a short period of time, building a high-performance system that can download hundreds of millions of pages over several weeks presents a number of challenges in system design, I/O and network efficiency, and robustness and manageability. 
Web crawlers are a central part of search engines, and details on their algorithms and architecture are kept as business secrets. When crawler designs are published, there is often an important lack of detail that prevents others from reproducing the work. There are also emerging concerns about "search engine spamming", which prevent major search engines from publishing their ranking algorithms.
Security[edit]
While most of the website owners are keen to have their pages indexed as broadly as possible to have strong presence in search engines, web crawling can also have unintended consequences and lead to a compromise or data breach if a search engine indexes resources that shouldn't be publicly available, or pages revealing potentially vulnerable versions of software.
Apart from standard web application security recommendations website owners can reduce their exposure to opportunistic hacking by only allowing search engines to index the public parts of their websites (with robots.txt) and explicitly blocking them from indexing transactional parts (login pages, private pages, etc.).
Crawler identification[edit]
Web crawlers typically identify themselves to a Web server by using the User-agent field of an HTTP request. Web site administrators typically examine their Web servers' log and use the user agent field to determine which crawlers have visited the web server and how often. The user agent field may include a URL where the Web site administrator may find out more information about the crawler. Examining Web server log is tedious task, and therefore some administrators use tools to identify, track and verify Web crawlers. Spambots and other malicious Web crawlers are unlikely to place identifying information in the user agent field, or they may mask their identity as a browser or other well-known crawler.
 Web site administrators prefer Web crawlers to identify themselves so that they can contact the owner if needed. In some cases, crawlers may be accidentally trapped in a crawler trap or they may be overloading a Web server with requests, and the owner needs to stop the crawler. Identification is also useful for administrators that are interested in knowing when they may expect their Web pages to be indexed by a particular search engine.
Crawling the deep web[edit]
A vast amount of web pages lie in the deep or invisible web. [43]  These pages are typically only accessible by submitting queries to a database, and regular crawlers are unable to find these pages if there are no links that point to them. Google's Sitemaps protocol and mod oai  [44]  are intended to allow discovery of these deep-Web resources.
 Deep web crawling also multiplies the number of web links to be crawled. Some crawlers only take some of the URLs in &lt;a href="URL"&gt; form. In some cases, such as the Googlebot, Web crawling is done on all text contained inside the hypertext content, tags, or text.
 Strategic approaches may be taken to target deep Web content. With a technique called screen scraping, specialized software may be customized to automatically and repeatedly query a given Web form with the intention of aggregating the resulting data. Such software can be used to span multiple Web forms across multiple Websites. Data extracted from the results of one Web form submission can be taken and applied as input to another Web form thus establishing continuity across the Deep Web in a way not possible with traditional web crawlers. [45] 
 Pages built on AJAX are among those causing problems to web crawlers. Google has proposed a format of AJAX calls that their bot can recognize and index. [46] 
Visual vs programmatic crawlers[edit]
There are a number of "visual web scraper/crawler" products available on the web which will crawl pages and structure data into columns and rows based on the users requirements. One of the main difference between a classic and a visual crawler is the level of programming ability required to set up a crawler. The latest generation of "visual scrapers" remove the majority of the programming skill needed to be able to program and start a crawl to scrape web data.
 The visual scraping/crawling method relies on the user "teaching" a piece of crawler technology, which then follows patterns in semi-structured data sources. The dominant method for teaching a visual crawler is by highlighting data in a browser and training columns and rows. While the technology is not new, for example it was the basis of Needlebase which has been bought by Google (as part of a larger acquisition of ITA Labs [47] ), there is continued growth and investment in this area by investors and end-users.[ citation needed ]
List of web crawlers[edit]
The following is a list of published crawler architectures for general-purpose crawlers (excluding focused web crawlers), with a brief description that includes the names given to the different components and outstanding features:
Historical web crawlers[edit]
  World Wide Web Worm was a crawler used to build a simple index of document titles and URLs. The index could be searched by using the  grep  Unix command.
Yahoo! Slurp was the name of the Yahoo! Search crawler until Yahoo! contracted with Microsoft to use Bingbot instead. 
In-house web crawlers[edit]
 Applebot is Apple's web crawler. It supports Siri and other products. [48]  
 Bingbot is the name of Microsoft's Bing webcrawler. It replaced  Msnbot .
Baiduspider is Baidu's web crawler.
 Googlebot is described in some detail, but the reference is only about an early version of its architecture, which was written in C++ and Python. The crawler was integrated with the indexing process, because text parsing was done for full-text indexing and also for URL extraction. There is a URL server that sends lists of URLs to be fetched by several crawling processes. During parsing, the URLs found were passed to a URL server that checked if the URL have been previously seen. If not, the URL was added to the queue of the URL server.
 WebCrawler was used to build the first publicly available full-text index of a subset of the Web. It was based on lib-WWW to download pages, and another program to parse and order URLs for breadth-first exploration of the Web graph. It also included a real-time crawler that followed links based on the similarity of the anchor text with the provided query.
 WebFountain is a distributed, modular crawler similar to Mercator but written in C++.
 Xenon is a web crawler used by government tax authorities to detect fraud. [49]   [50]   
Commercial web crawlers[edit]
The following web crawlers are available, for a price::
  Diffbot - programmatic general web crawler, available as an API 
 SortSite - crawler for analyzing websites, available for Windows and Mac OS 
Swiftbot - Swiftype's web crawler, available as software as a service  
Open-source crawlers[edit]
  GNU Wget is a command-line-operated crawler written in C and released under the GPL. It is typically used to mirror Web and FTP sites.
 GRUB was an open source distributed search crawler that Wikia Search used to crawl the web.
 Heritrix is the Internet Archive's archival-quality crawler, designed for archiving periodic snapshots of a large portion of the Web. It was written in Java.
 ht://Dig includes a Web crawler in its indexing engine.
 HTTrack uses a Web crawler to create a mirror of a web site for off-line viewing. It is written in C and released under the GPL.
 mnoGoSearch is a crawler, indexer and a search engine written in C and licensed under the GPL (*NIX machines only)
 Apache Nutch is a highly extensible and scalable web crawler written in Java and released under an Apache License. It is based on Apache Hadoop and can be used with Apache Solr or Elasticsearch.
 Open Search Server is a search engine and web crawler software release under the GPL.
 Scrapy, an open source webcrawler framework, written in python (licensed under BSD).
 Seeks, a free distributed search engine (licensed under AGPL).
 StormCrawler, a collection of resources for building low-latency, scalable web crawlers on Apache Storm (Apache License).
 tkWWW Robot, a crawler based on the tkWWW web browser (licensed under GPL).
 Xapian, a search crawler engine, written in c++.
 YaCy, a free distributed search engine, built on principles of peer-to-peer networks (licensed under GPL). 
See also[edit]
  Automatic indexing 
 Gnutella crawler 
 Web archiving 
 Webgraph 
 Website mirroring software 
 Search Engine Scraping 
 Web scraping  
References[edit]
  ^   "Web Crawlers: Browsing the Web". Archived from the original on 6 December 2021.
  ^  Spetka, Scott. "The TkWWW Robot: Beyond Browsing". NCSA. Archived from the original on 3 September 2004. Retrieved 21 November 2010.
  ^  Kobayashi, M. &amp; Takeda, K. (2000). "Information retrieval on the web". ACM Computing Surveys. 32 (2): 144–173. CiteSeerX 10.1.1.126.6094. doi:10.1145/358923.358934. S2CID 3710903.
  ^  See definition of scutter on FOAF Project's wiki Archived 13 December 2009 at the Wayback Machine
  ^  Masanès, Julien (15 February 2007).  Web Archiving . Springer. p. 1. ISBN  978-3-54046332-0 . Retrieved 24 April 2014.
  ^  Edwards, J.; McCurley, K. S.; and Tomlin, J. A. (2001). "An adaptive model for optimizing performance of an incremental web crawler".  Proceedings of the tenth international conference on World Wide Web - WWW '01 . In Proceedings of the Tenth Conference on World Wide Web. pp. 106–113. CiteSeerX 10.1.1.1018.1506. doi:10.1145/371920.371960. ISBN  978-1581133486 . S2CID 10316730. {{cite book}}: CS1 maint: multiple names: authors list (link)
  ^  Castillo, Carlos (2004).  Effective Web Crawling  (PhD thesis). University of Chile. Retrieved 3 August 2010.
  ^  Gulls, A.; A. Signori (2005). "The indexable web is more than 11.5 billion pages". Special interest tracks and posters of the 14th international conference on World Wide Web. ACM Press. pp. 902–903. doi:10.1145/1062745.1062789.
  ^  Lawrence, Steve; C. Lee Giles (8 July 1999). "Accessibility of information on the web". Nature. 400 (6740): 107–9. Bibcode:1999Natur.400..107L. doi:10.1038/21987. PMID 10428673. S2CID 4347646.
  ^  Cho, J.; Garcia-Molina, H.; Page, L. (April 1998). "Efficient Crawling Through URL Ordering". Seventh International World-Wide Web Conference. Brisbane, Australia. doi:10.1142/3725. ISBN  978-981-02-3400-3 . Retrieved 23 March 2009.
  ^  Cho, Junghoo, "Crawling the Web: Discovery and Maintenance of a Large-Scale Web Data", PhD dissertation, Department of Computer Science, Stanford University, November 2001.
  ^  Najork, Marc and Janet L. Wiener. "Breadth-first crawling yields high-quality pages". In: Proceedings of the Tenth Conference on World Wide Web, pages 114–118, Hong Kong, May 2001. Elsevier Science.
  ^  Abiteboul, Serge; Mihai Preda; Gregory Cobena (2003). "Adaptive on-line page importance computation". Proceedings of the 12th international conference on World Wide Web. Budapest, Hungary: ACM. pp. 280–290. doi:10.1145/775152.775192. ISBN  1-58113-680-3 . Retrieved 22 March 2009.
  ^  Boldi, Paolo; Bruno Codenotti; Massimo Santini; Sebastiano Vigna (2004). "UbiCrawler: a scalable fully distributed Web crawler" (PDF). Software: Practice and Experience. 34 (8): 711–726. CiteSeerX 10.1.1.2.5538. doi:10.1002/spe.587. S2CID 325714. Archived from the original (PDF) on 20 March 2009. Retrieved 23 March 2009.
  ^  Boldi, Paolo; Massimo Santini; Sebastiano Vigna (2004). "Do Your Worst to Make the Best: Paradoxical Effects in PageRank Incremental Computations" (PDF). Algorithms and Models for the Web-Graph. Lecture Notes in Computer Science. Vol. 3243. pp. 168–180. doi:10.1007/978-3-540-30216-2_14. ISBN  978-3-540-23427-2 . Retrieved 23 March 2009.
  ^  Baeza-Yates, R.; Castillo, C.; Marin, M. and Rodriguez, A. (2005). "Crawling a Country: Better Strategies than Breadth-First for Web Page Ordering." In: Proceedings of the Industrial and Practical Experience track of the 14th conference on World Wide Web, pages 864–872, Chiba, Japan. ACM Press.
  ^  Shervin Daneshpajouh, Mojtaba Mohammadi Nasiri, Mohammad Ghodsi, A Fast Community Based Algorithm for Generating Crawler Seeds Set. In: Proceedings of 4th International Conference on Web Information Systems and Technologies ( Webist -2008), Funchal, Portugal, May 2008.
  ^  Pant, Gautam; Srinivasan, Padmini; Menczer, Filippo (2004). "Crawling the Web" (PDF). In Levene, Mark; Poulovassilis, Alexandra (eds.). Web Dynamics: Adapting to Change in Content, Size, Topology and Use. Springer. pp. 153–178. ISBN  978-3-540-40676-1 . Archived from the original (PDF) on 20 March 2009. Retrieved 9 May 2006.
  ^  Cothey, Viv (2004). "Web-crawling reliability" (PDF). Journal of the American Society for Information Science and Technology. 55 (14): 1228–1238. CiteSeerX 10.1.1.117.185. doi:10.1002/asi.20078.
  ^  Menczer, F. (1997). ARACHNID: Adaptive Retrieval Agents Choosing Heuristic Neighborhoods for Information Discovery Archived 21 December 2012 at the Wayback Machine. In D. Fisher, ed., Machine Learning: Proceedings of the 14th International Conference (ICML97). Morgan Kaufmann
  ^  Menczer, F. and Belew, R.K. (1998). Adaptive Information Agents in Distributed Textual Environments Archived 21 December 2012 at the Wayback Machine. In K. Sycara and M. Wooldridge (eds.) Proc. 2nd Intl. Conf. on Autonomous Agents (Agents '98). ACM Press
  ^  Chakrabarti, Soumen; Van Den Berg, Martin; Dom, Byron (1999). "Focused crawling: A new approach to topic-specific Web resource discovery" (PDF). Computer Networks. 31 (11–16): 1623–1640. doi:10.1016/s1389-1286(99)00052-3. Archived from the original (PDF) on 17 March 2004.
  ^  Pinkerton, B. (1994). Finding what people want: Experiences with the WebCrawler. In Proceedings of the First World Wide Web Conference, Geneva, Switzerland.
  ^  Diligenti, M., Coetzee, F., Lawrence, S., Giles, C. L., and Gori, M. (2000). Focused crawling using context graphs. In Proceedings of 26th International Conference on Very Large Databases (VLDB), pages 527-534, Cairo, Egypt.
  ^  Wu, Jian; Teregowda, Pradeep; Khabsa, Madian; Carman, Stephen; Jordan, Douglas; San Pedro Wandelmer, Jose; Lu, Xin; Mitra, Prasenjit; Giles, C. Lee (2012). "Web crawler middleware for search engine digital libraries". Proceedings of the twelfth international workshop on Web information and data management - WIDM '12. p. 57. doi:10.1145/2389936.2389949. ISBN  9781450317207 . S2CID 18513666.
  ^  Wu, Jian; Teregowda, Pradeep; Ramírez, Juan Pablo Fernández; Mitra, Prasenjit; Zheng, Shuyi; Giles, C. Lee (2012). "The evolution of a crawling strategy for an academic document search engine". Proceedings of the 3rd Annual ACM Web Science Conference on - Web Sci '12. pp. 340–343. doi:10.1145/2380718.2380762. ISBN  9781450312288 . S2CID 16718130.
  ^  Dong, Hai; Hussain, Farookh Khadeer; Chang, Elizabeth (2009). "State of the Art in Semantic Focused Crawlers". Computational Science and Its Applications – ICCSA 2009. Lecture Notes in Computer Science. Vol. 5593. pp. 910–924. doi:10.1007/978-3-642-02457-3_74. hdl:20.500.11937/48288. ISBN  978-3-642-02456-6 .
  ^  Dong, Hai; Hussain, Farookh Khadeer (2013). "SOF: A semi-supervised ontology-learning-based focused crawler". Concurrency and Computation: Practice and Experience. 25 (12): 1755–1770. doi:10.1002/cpe.2980. S2CID 205690364.
  ^  Junghoo Cho; Hector Garcia-Molina (2000). "Synchronizing a database to improve freshness" (PDF). Proceedings of the 2000 ACM SIGMOD international conference on Management of data. Dallas, Texas, United States: ACM. pp. 117–128. doi:10.1145/342009.335391. ISBN  1-58113-217-4 . Retrieved 23 March 2009.
^    a       b    E. G. Coffman Jr; Zhen Liu; Richard R. Weber (1998). "Optimal robot scheduling for Web search engines". Journal of Scheduling. 1 (1): 15–29. CiteSeerX 10.1.1.36.6087. doi:10.1002/(SICI)1099-1425(199806)1:1&lt;15::AID-JOS3&gt;3.0.CO;2-K.
^    a       b    Cho, Junghoo; Garcia-Molina, Hector (2003). "Effective page refresh policies for Web crawlers". ACM Transactions on Database Systems. 28 (4): 390–426. doi:10.1145/958942.958945. S2CID 147958.
^    a       b    Junghoo Cho; Hector Garcia-Molina (2003). "Estimating frequency of change". ACM Transactions on Internet Technology. 3 (3): 256–290. CiteSeerX 10.1.1.59.5877. doi:10.1145/857166.857170. S2CID 9362566.
  ^  Ipeirotis, P., Ntoulas, A., Cho, J., Gravano, L. (2005) Modeling and managing content changes in text databases. In Proceedings of the 21st IEEE International Conference on Data Engineering, pages 606-617, April 2005, Tokyo.
  ^  Koster, M. (1995). Robots in the web: threat or treat? ConneXions, 9(4).
  ^  Koster, M. (1996). A standard for robot exclusion Archived 7 November 2007 at the Wayback Machine.
  ^  Koster, M. (1993). Guidelines for robots writers Archived 22 April 2005 at the Wayback Machine.
  ^  Baeza-Yates, R. and Castillo, C. (2002). Balancing volume, quality and freshness in Web crawling. In Soft Computing Systems – Design, Management and Applications, pages 565–572, Santiago, Chile. IOS Press Amsterdam.
  ^  Heydon, Allan; Najork, Marc (26 June 1999). "Mercator: A Scalable, Extensible Web Crawler" (PDF). Archived from the original (PDF) on 19 February 2006. Retrieved 22 March 2009. 
  ^  Dill, S.; Kumar, R.; Mccurley, K. S.; Rajagopalan, S.; Sivakumar, D.; Tomkins, A. (2002). "Self-similarity in the web" (PDF). ACM Transactions on Internet Technology. 2 (3): 205–223. doi:10.1145/572326.572328. S2CID 6416041.
  ^  M. Thelwall; D. Stuart (2006). "Web crawling ethics revisited: Cost, privacy and denial of service". Journal of the American Society for Information Science and Technology. 57 (13): 1771–1779. doi:10.1002/asi.20388.
  ^  Brin, Sergey; Page, Lawrence (1998). "The anatomy of a large-scale hypertextual Web search engine". Computer Networks and ISDN Systems. 30 (1–7): 107–117. doi:10.1016/s0169-7552(98)00110-x.
  ^  Shkapenyuk, V. and Suel, T. (2002). Design and implementation of a high performance distributed web crawler. In Proceedings of the 18th International Conference on Data Engineering (ICDE), pages 357-368, San Jose, California. IEEE CS Press.
  ^  Shestakov, Denis (2008).  Search Interfaces on the Web: Querying and Characterizing  Archived 6 July 2014 at the Wayback Machine. TUCS Doctoral Dissertations 104, University of Turku
  ^  Michael L Nelson; Herbert Van de Sompel; Xiaoming Liu; Terry L Harrison; Nathan McFarland (24 March 2005). "mod_oai: An Apache Module for Metadata Harvesting": cs/0503069. arXiv:cs/0503069. Bibcode:2005cs........3069N. 
  ^  Shestakov, Denis; Bhowmick, Sourav S.; Lim, Ee-Peng (2005). "DEQUE: Querying the Deep Web" (PDF). Data &amp; Knowledge Engineering. 52 (3): 273–311. doi:10.1016/s0169-023x(04)00107-7.
  ^   "AJAX crawling: Guide for webmasters and developers". Retrieved 17 March 2013.
  ^  ITA Labs "ITA Labs Acquisition" 20 April 2011 1:28 AM
  ^   "About Applebot". Apple Inc. Retrieved 18 October 2021.
  ^  
Norton, Quinn (25 January 2007). "Tax takers send in the spiders". Business.  Wired . Archived from the original on 22 December 2016. Retrieved 13 October 2017.
  ^  
 "Xenon web crawling initiative: privacy impact assessment (PIA) summary". Ottawa: Government of Canada. 11 April 2017. Archived from the original on 25 September 2017. Retrieved 13 October 2017.
 
Further reading[edit]
 Cho, Junghoo, "Web Crawling Project", UCLA Computer Science Department.
 A History of Search Engines, from Wiley 
 WIVET is a benchmarking project by OWASP, which aims to measure if a web crawler can identify all the hyperlinks in a target website.
Shestakov, Denis, "Current Challenges in Web Crawling" and "Intelligent Web Crawling", slides for tutorials given at ICWE'13 and WI-IAT'13.
Web crawler - Wikipedia
https://en.wikipedia.org/wiki/Web_crawler#Crawling_policy
https://webscraper.io/
Web Scraper
2023-01-01
None
<div><div>
	<h2>Export data in CSV, XLSX and JSON
	formats</h2>
	<p>Build scrapers, scrape sites and export data in CSV format directly from your browser.
	Use Web Scraper Cloud to export data in CSV, XLSX and JSON formats, access it via API, webhooks or
	get it exported via Dropbox, Google Sheets or Amazon S3.
	</p>
	</div></div>||||I|||| Toggle navigation
 * 

 * Web Scraper

 * Cloud Scraper

 * Pricing

 * Learn

 + Documentation
 + Video Tutorials
 + How to
 + Test Sites
 + Forum
 * Install
 * Login

 Making web data extraction easy and accessible for everyone

 More than 500,000 users are proud of using our solutions!

 Point and click interface

 Our goal is to make web data extraction as simple as possible. Configure scraper by simply pointing and clicking on elements. No coding required.

 Extract data from dynamic web sites

 Web Scraper can extract data from sites with multiple levels of navigation. It can navigate a website on all levels.

 * Categories and subcategories
 * Pagination
 * Product pages

 Built for the modern web

 Websites today are built on top of JavaScript frameworks that make user interface easier to use but are less accessible to scrapers. Web Scraper solves this by:

 * Full JavaScript execution
 * Waiting for Ajax requests
 * Pagination handlers
 * Page scroll down

 Modular selector system

 Web Scraper allows you to build Site Maps from different types of selectors. This system makes it possible to tailor data extraction to different site structures.

 Export data in CSV, XLSX and JSON formats

 Build scrapers, scrape sites and export data in CSV format directly from your browser. Use Web Scraper Cloud to export data in CSV, XLSX and JSON formats, access it via API, webhooks or get it exported via Dropbox, Google Sheets or Amazon S3.

 Start using Web Scraper now!

 Install Web Scraper

 Chrome extension Firefox add-on

 Scrape your first site

 Need to automate data extraction?

 Run Web Scraper jobs in our Cloud. Configure scheduled scraping and access data via API or get it in your Dropbox

 Scheduler

 Scrape a site on hourly, daily or weekly basis

 API

 Manage scrapers through an API

 Proxy

 IP rotation through thousands of IP addresses

 Parser

 Streamline data post processing

 Start free trial

 Check out what people say about our solution!

 Diego Kremer

 Simply AMAZING. Was thinking about coding myself a simple scraper for a project and then found this super easy to use and very powerful scraper. Worked perfectly with all the websites I tried on. Saves a lot of time. Thanks for that!

 Carlos Figueroa

 Powerful tool that beats the others out there. Has a learning curve to it but once you conquer that the sky's the limit. Definitely a tool worth making a donation on and supporting for continued development. Way to go for the authoring crew behind this tool.

 Jonathan H

 This is fantastic! I'm saving hours, possibly days. I was trying to scrap and old site, badly made, no proper divs or markup. Using the WebScraper magic, it somehow "knew" the pattern after I selected 2 elements. Amazing. Yes, it's a learning curve and you HAVE to watch the video and read the docs. Don't rate it down just because you can't be bothered to learn it. If you put the effort in, this will save your butt one day!

 * Products

 * Web Scraper browser extension
 * Web Scraper Cloud

 * Company

 * About us
 * Contact
 * Website Privacy Policy
 * Browser Extension Privacy Policy
 * Media kit
 * Jobs

 * Resources

 * Blog
 * Documentation
 * Video Tutorials
 * Screenshots
 * Test Sites
 * Forum

 * CONTACT US

 * info@webscraper.io
 * Ubelu 5-71,
 Adazi, Latvia, LV-2164
 * 
 * 

 Copyright © 2023 Web Scraper | All rights reserved | Made by zoom59

Export data in CSV, XLSX and JSON
	formats
	Build scrapers, scrape sites and export data in CSV format directly from your browser.
	Use Web Scraper Cloud to export data in CSV, XLSX and JSON formats, access it via API, webhooks or
	get it exported via Dropbox, Google Sheets or Amazon S3.
Web Scraper
https://webscraper.io/
https://www.seroundtable.com/google-increase-in-crawl-errors-22669.html
Google: The Increase In Crawl Errors Are Nothing To Worry About; Just Hungry Crawlers
2016-09-09
Barry Schwartz
<div><div>
	 <p></p><p>Over the past week or so, some webmasters have been reporting an increase in crawl errors. It is mostly documented in this <a href="https://www.webmasterworld.com/google/4817870.htm">WebmasterWorld</a> thread. </p><p>Some suspect it has to do with an algorithm update but we've covered not <a href="https://www.seroundtable.com/google-crawl-rate-spike-update-myth-21616.html">just once</a> but <a href="https://www.seroundtable.com/google-crawl-spikes-not-a-sign-of-an-upcoming-algorithm-updates-22238.html">twice</a> that these crawl changes in Google Search Console have no relation to upcoming algorithm updates - at least that is what Google told us.</p><p>John Mueller was asked about this by Simon in the Google Hangout, the last one using <a href="https://plus.google.com/u/0/events/cg84co37du0l3lhfmkdd2jt1iec">Google+</a>, at the <a href="https://www.youtube.com/watch?v=Wcnz1kCoiks#t=1447">24:07 mark</a> into the video. John essentially said that he looked into some of these reports and yes, it is unrelated to any algorithm thing. He said in the cases he looked up, it was related to the algorithm saying they wanted to double check a lot of old URLs. So maybe GoogleBot felt a burst of energy and was eager to try out a bunch of old URLs.</p><p>Here was the question:</p><blockquote>On the 3rd August we saw our crawl errors jump from 1,000 to 10,000 and then on the 1st September, we saw a jump from 10,000 to 50,000. Now, nothing is wrong with the site - these are all crawl errors from two, three, four years ago. And you mentioned to look where they're linked from, is it a sitemap file and so forth. But even the sitemap files referenced in the 'link from’ are two years old. And I'm just wondering - and I appreciate that you've already said that an increase in googlebot activity doesn't mean anything's happening or there's anything to worry about and I appreciate that this is information only and these crawl errors don't mean there's anything wrong although you may want to check if there is.<p>But is there any reason why people are seeing this at the moment? And this sudden increase - is it penguin related at all?</p></blockquote><p>John said:</p><blockquote>No. <p>So I I've seen a few reports like that and I wanted to double check with the team on that as well. </p><p>We looked into a couple of these reports and they were essentially just our algorithm saying, oh I I have all these old URLs I just want to double check.</p><p>I’s weird that a lot of these are kind of jumbled together now and like like you mentioned that they're kind of all all on the same time. So I wonder if we will change anything specific there. But it in practice for from a webmasters point of view you don't need to take action on this. For us it's it's also a good good push to try to figure out how we can make these reports a little bit more actionable so that we don't scare people out.</p></blockquote><p>Here is the video embed:</p><p></p><p>Forum discussion at <a href="https://plus.google.com/u/0/events/cg84co37du0l3lhfmkdd2jt1iec">Google+</a>.</p>	 </div>||||I|||| Home > Google News > Google SEO > Google: The Increase In Crawl Errors Are Nothing To Worry About; Just Hungry Crawlers

 Google: The Increase In Crawl Errors Are Nothing To Worry About; Just Hungry Crawlers

 Sep 9, 2016 • 7:32 am | (4) by Barry Schwartz | Filed Under Google Search Engine Optimization
 Prev Story Next story
 Tweet
 Share
 * 
 Prev Story Next Story

 Over the past week or so, some webmasters have been reporting an increase in crawl errors. It is mostly documented in this WebmasterWorld thread.

 Some suspect it has to do with an algorithm update but we've covered not just once but twice that these crawl changes in Google Search Console have no relation to upcoming algorithm updates - at least that is what Google told us.

 John Mueller was asked about this by Simon in the Google Hangout, the last one using Google+, at the 24:07 mark into the video. John essentially said that he looked into some of these reports and yes, it is unrelated to any algorithm thing. He said in the cases he looked up, it was related to the algorithm saying they wanted to double check a lot of old URLs. So maybe GoogleBot felt a burst of energy and was eager to try out a bunch of old URLs.

 Here was the question:

 On the 3rd August we saw our crawl errors jump from 1,000 to 10,000 and then on the 1st September, we saw a jump from 10,000 to 50,000. Now, nothing is wrong with the site - these are all crawl errors from two, three, four years ago. And you mentioned to look where they're linked from, is it a sitemap file and so forth. But even the sitemap files referenced in the 'link from’ are two years old. And I'm just wondering - and I appreciate that you've already said that an increase in googlebot activity doesn't mean anything's happening or there's anything to worry about and I appreciate that this is information only and these crawl errors don't mean there's anything wrong although you may want to check if there is.

 But is there any reason why people are seeing this at the moment? And this sudden increase - is it penguin related at all?

 John said:

 No.

 So I I've seen a few reports like that and I wanted to double check with the team on that as well.

 We looked into a couple of these reports and they were essentially just our algorithm saying, oh I I have all these old URLs I just want to double check.

 I’s weird that a lot of these are kind of jumbled together now and like like you mentioned that they're kind of all all on the same time. So I wonder if we will change anything specific there. But it in practice for from a webmasters point of view you don't need to take action on this. For us it's it's also a good good push to try to figure out how we can make these reports a little bit more actionable so that we don't scare people out.

 Here is the video embed:

 Forum discussion at Google+.

 Previous story: Daily Search Forum Recap: September 8, 2016
 blog comments powered by Disqus

 The pulse of the search community

 Subscribe

 Subscribe options
 * Let’s Talk AI and Google.
 * How to Detect AI-Generated Content (and Can Google Do That)
 * How to Do Link Building in 2022-2023
 * Many sites aren’t using their blog correctly – are you?
 * Using and Optimizing Images: Search and Social Optimization Cheat Sheets

 Search Video Recaps

 02/24/2023 08:01 am
 Subscribe More videos Video Details

 Search Vlog

 02/20/2023 08:01 am
 Subscribe More vlogs Vlog Details

 Browse by:

 * Browse by Date
 * Find by Category
 * Discover by Author
 * Scan Most Recent
 * See Comments
 * View Tag Cloud
 * Submit a Thread
 * Forum Search
 * Awards
 * About Us
 * Seals
 * Advertise
 * Contact
 * Google
 + Google Updates
 + Google News
 + Google SEO
 + AdSense
 + Google Ads
 + Misc Google
 + Google Maps
 * Bing
 + Bing SEO
 + Bing Ads
 + ContentAds
 * Yahoo
 + Yahoo SEO
 + Yahoo Search Ads
 + Yahoo Publisher Network
 + Yahoo News
 + Other Yahoo
 + Overture
 * SEO
 + Sitemaps
 + Spiders
 + Informational Sites SEO
 + SEO Copywriting
 + Keyword Research
 + Link Building
 + Dynamic Site SEO
 + Cloaking
 + Local SEO
 * Other Search Topics
 + Honor
 + Search News
 + SEO Tools
 + Spam
 + Forum News
 + SEM Companies
 + Web Promotion
 + SEO Interviews
 + Search Technology
 + Search Theory
 + SEM Organizations
 + SEO Books
 + Legal Search Issues
 + Affiliate Marketing
 + Social Search
 * More...
 + Logos
 + Other Search Engines
 + SEO Conferences
 + PPC Engines
 + Web Directories
 + Ask.com
 + Photos
 + Shopping Search
 + Multilingual Search
 + Search Pulse
 + Search Video Recaps
 + Digg Digest
 + Search Forum Recap
 + Search Engine Roundtable Vlog
 + Misc
 + Ad Email
 Search ›
 * Submit a Thread
 * Forum Search
 * Awards
 * About Us
 * Seals
 * Advertise
 * Contact
 * Mobile Site

 The content at the Search Engine Roundtable are the sole opinion of the authors and in no way reflect views of RustyBrick ®, Inc
 Copyright © 1994-2023 RustyBrick ®, Inc. Web Development All Rights Reserved.
 This work by Search Engine Roundtable is licensed under a Creative Commons Attribution 3.0 United States License. Creative Commons License and YouTube videos under YouTube's ToS.

Over the past week or so, some webmasters have been reporting an increase in crawl errors. It is mostly documented in this WebmasterWorld thread.  Some suspect it has to do with an algorithm update but we've covered not just once but twice that these crawl changes in Google Search Console have no relation to upcoming algorithm updates - at least that is what Google told us. John Mueller was asked about this by Simon in the Google Hangout, the last one using Google+, at the 24:07 mark into the video. John essentially said that he looked into some of these reports and yes, it is unrelated to any algorithm thing. He said in the cases he looked up, it was related to the algorithm saying they wanted to double check a lot of old URLs. So maybe GoogleBot felt a burst of energy and was eager to try out a bunch of old URLs. Here was the question: On the 3rd August we saw our crawl errors jump from 1,000 to 10,000 and then on the 1st September, we saw a jump from 10,000 to 50,000. Now, nothing is wrong with the site - these are all crawl errors from two, three, four years ago. And you mentioned to look where they're linked from, is it a sitemap file and so forth. But even the sitemap files referenced in the 'link from’ are two years old. And I'm just wondering - and I appreciate that you've already said that an increase in googlebot activity doesn't mean anything's happening or there's anything to worry about and I appreciate that this is information only and these crawl errors don't mean there's anything wrong although you may want to check if there is.But is there any reason why people are seeing this at the moment? And this sudden increase - is it penguin related at all?  John said: No. So I I've seen a few reports like that and I wanted to double check with the team on that as well.  We looked into a couple of these reports and they were essentially just our algorithm saying, oh I I have all these old URLs I just want to double check. I’s weird that a lot of these are kind of jumbled together now and like like you mentioned that they're kind of all all on the same time. So I wonder if we will change anything specific there. But it in practice for from a webmasters point of view you don't need to take action on this. For us it's it's also a good good push to try to figure out how we can make these reports a little bit more actionable so that we don't scare people out.  Here is the video embed:   Forum discussion at Google+.
Google: The Increase In Crawl Errors Are Nothing To Worry About; Just Hungry Crawlers
https://www.seroundtable.com/google-increase-in-crawl-errors-22669.html?utm_content=buffer129bf&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer
https://machinelearningmastery.com/web-crawling-in-python
Web Crawling in Python - MachineLearningMastery.com
2022-04-15
Adrian Tam
<div><section>
<article>
<header>
 </header>
<section>
<p>Last Updated on June 21, 2022</p>
<p>In the old days, it was a tedious job to collect data, and it was sometimes very expensive. Machine learning projects cannot live without data. Luckily, we have a lot of data on the web at our disposal nowadays. We can copy data from the web to create our dataset. We can manually download files and save them to the disk. But we can do it more efficiently by automating the data harvesting. There are several tools in Python that can help the automation.</p>
<p>After finishing this tutorial, you will learn:</p>
<ul>
<li>How to use the requests library to read online data using HTTP</li>
<li>How to read tables on web pages using pandas</li>
<li>How to use Selenium to emulate browser operations</li>
</ul>
<p><strong>Kick-start your project</strong> with my new book <a href="https://machinelearningmastery.com/python-for-machine-learning/">Python for Machine Learning</a>, including <em>step-by-step tutorials</em> and the <em>Python source code</em> files for all examples.</p>Let’s get started!
<div><p></p><p>Web Crawling in Python<br />Photo by <a href="https://www.pexels.com/photo/black-and-red-spider-on-web-in-close-up-photography-4805619/">Ray Bilcliff</a>. Some rights reserved.</p></div>
<h2>Overview</h2>
<p>This tutorial is divided into three parts; they are:</p>
<ul>
<li>Using the requests library</li>
<li>Reading tables on the web using pandas</li>
<li>Reading dynamic content with Selenium</li>
</ul>
<h2>Using the Requests Library</h2>
<p>When we talk about writing a Python program to read from the web, it is inevitable that we can’t avoid the <code>requests</code> library. You need to install it (as well as BeautifulSoup and lxml that we will cover later):</p>
<div>
<table>
<tbody><tr>
<td>
</td>
<td><div><p>pip install requests beautifulsoup4 lxml</p></div></td>
</tr>
</tbody></table>
</div><p>It provides you with an interface that allows you to interact with the web easily.</p>
<p>The very simple use case would be to read a web page from a URL:</p>
<div>
<table>
<tbody><tr>
<td>
</td>
<td><div><p>import requests</p><p># Lat-Lon of New York</p><p>URL = "https://weather.com/weather/today/l/40.75,-73.98"</p><p>resp = requests.get(URL)</p><p>print(resp.status_code)</p><p>print(resp.text)</p></div></td>
</tr>
</tbody></table>
</div>
<div>
<table>
<tbody><tr>
<td>
</td>
<td><div><p>200</p><p>&lt;!doctype html&gt;&lt;html dir="ltr" lang="en-US"&gt;&lt;head&gt;</p><p>      &lt;meta data-react-helmet="true" charset="utf-8"/&gt;&lt;meta data-react-helmet="true"</p><p>name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/&gt;</p><p>...</p></div></td>
</tr>
</tbody></table>
</div><p>If you’re familiar with HTTP, you can probably recall that a status code of 200 means the request is successfully fulfilled. Then we can read the response. In the above, we read the textual response and get the HTML of the web page. Should it be a CSV or some other textual data, we can get them in the <code>text</code> attribute of the response object. For example, this is how we can read a CSV from the Federal Reserve Economics Data:</p>
<div>
<table>
<tbody><tr>
<td>
</td>
<td><div><p>import io</p><p>import pandas as pd</p><p>import requests</p><p>URL = "https://fred.stlouisfed.org/graph/fredgraph.csv?id=T10YIE&amp;cosd=2017-04-14&amp;coed=2022-04-14"</p><p>resp = requests.get(URL)</p><p>if resp.status_code == 200:</p><p>   csvtext = resp.text</p><p>   csvbuffer = io.StringIO(csvtext)</p><p>   df = pd.read_csv(csvbuffer)</p><p>   print(df)</p></div></td>
</tr>
</tbody></table>
</div>
<div>
<table>
<tbody><tr>
<td>
</td>
<td><div><p>            DATE T10YIE</p><p>0     2017-04-17   1.88</p><p>1     2017-04-18   1.85</p><p>2     2017-04-19   1.85</p><p>3     2017-04-20   1.85</p><p>4     2017-04-21   1.84</p><p>...          ...    ...</p><p>1299  2022-04-08   2.87</p><p>1300  2022-04-11   2.91</p><p>1301  2022-04-12   2.86</p><p>1302  2022-04-13    2.8</p><p>1303  2022-04-14   2.89</p><p>[1304 rows x 2 columns]</p></div></td>
</tr>
</tbody></table>
</div><p>If the data is in the form of JSON, we can read it as text or even let <code>requests</code> decode it for you. For example, the following is to pull some data from GitHub in JSON format and convert it into a Python dictionary:</p>
<div>
<table>
<tbody><tr>
<td>
</td>
<td><div><p>import requests</p><p>URL = "https://api.github.com/users/jbrownlee"</p><p>resp = requests.get(URL)</p><p>if resp.status_code == 200:</p><p>    data = resp.json()</p><p>    print(data)</p></div></td>
</tr>
</tbody></table>
</div>
<div>
<table>
<tbody><tr>
<td>
</td>
<td><div><p>{'login': 'jbrownlee', 'id': 12891, 'node_id': 'MDQ6VXNlcjEyODkx',</p><p>'avatar_url': 'https://avatars.githubusercontent.com/u/12891?v=4',</p><p>'gravatar_id': '', 'url': 'https://api.github.com/users/jbrownlee',</p><p>'html_url': 'https://github.com/jbrownlee',</p><p>...</p><p>'company': 'Machine Learning Mastery', 'blog': 'http://MachineLearningMastery.com',</p><p>'location': None, 'email': None, 'hireable': None,</p><p>'bio': 'Making developers awesome at machine learning.', 'twitter_username': None,</p><p>'public_repos': 5, 'public_gists': 0, 'followers': 1752, 'following': 0,</p><p>'created_at': '2008-06-07T02:20:58Z', 'updated_at': '2022-02-22T19:56:27Z'</p><p>}</p></div></td>
</tr>
</tbody></table>
</div><p>But if the URL gives you some binary data, such as a ZIP file or a JPEG image, you need to get them in the <code>content</code> attribute instead, as this would be the binary data. For example, this is how we can download an image (the logo of Wikipedia):</p>
<div>
<table>
<tbody><tr>
<td>
</td>
<td><div><p>import requests</p><p>URL = "https://en.wikipedia.org/static/images/project-logos/enwiki.png"</p><p>wikilogo = requests.get(URL)</p><p>if wikilogo.status_code == 200:</p><p>    with open("enwiki.png", "wb") as fp:</p><p>        fp.write(wikilogo.content)</p></div></td>
</tr>
</tbody></table>
</div><p>Given we already obtained the web page, how should we extract the data? This is beyond what the <code>requests</code> library can provide to us, but we can use a different library to help. There are two ways we can do it, depending on how we want to specify the data.</p>
<p>The first way is to consider the HTML as a kind of XML document and use the XPath language to extract the element. In this case, we can make use of the <code>lxml</code> library to first create a document object model (DOM) and then search by XPath:</p>
<div>
<table>
<tbody><tr>
<td>
</td>
<td><div><p>...</p><p>from lxml import etree</p><p># Create DOM from HTML text</p><p>dom = etree.HTML(resp.text)</p><p># Search for the temperature element and get the content</p><p>elements = dom.xpath("//span[@data-testid='TemperatureValue' and contains(@class,'CurrentConditions')]")</p><p>print(elements[0].text)</p></div></td>
</tr>
</tbody></table>
</div>
<p>XPath is a string that specifies how to find an element. The lxml object provides a function <code>xpath()</code> to search the DOM for elements that match the XPath string, which can be multiple matches. The XPath above means to find an HTML element anywhere with the <code>&lt;span&gt;</code> tag and with the attribute <code>data-testid</code> matching “<code>TemperatureValue</code>” and <code>class</code> beginning with “<code>CurrentConditions</code>.” We can learn this from the developer tools of the browser (e.g., the Chrome screenshot below) by inspecting the HTML source.</p>
<p></p>
<p>This example is to find the temperature of New York City, provided by this particular element we get from this web page. We know the first element matched by the XPath is what we need, and we can read the text inside the <code>&lt;span&gt;</code> tag.</p>
<p>The other way is to use CSS selectors on the HTML document, which we can make use of the BeautifulSoup library:</p>
<div>
<table>
<tbody><tr>
<td>
</td>
<td><div><p>...</p><p>from bs4 import BeautifulSoup</p><p>soup = BeautifulSoup(resp.text, "lxml")</p><p>elements = soup.select('span[data-testid="TemperatureValue"][class^="CurrentConditions"]')</p><p>print(elements[0].text)</p></div></td>
</tr>
</tbody></table>
</div>
<p>In the above, we first pass our HTML text to BeautifulSoup. BeautifulSoup supports various HTML parsers, each with different capabilities. In the above, we use the <code>lxml</code> library as the parser as recommended by BeautifulSoup (and it is also often the fastest). CSS selector is a different mini-language, with pros and cons compared to XPath. The selector above is identical to the XPath we used in the previous example. Therefore, we can get the same temperature from the first matched element.</p>
<p>The following is a complete code to print the current temperature of New York according to the real-time information on the web:</p>
<div>
<table>
<tbody><tr>
<td>
<div><p>1</p><p>2</p><p>3</p><p>4</p><p>5</p><p>6</p><p>7</p><p>8</p><p>9</p><p>10</p><p>11</p><p>12</p><p>13</p><p>14</p><p>15</p><p>16</p><p>17</p></div>
</td>
<td><div><p>import requests</p><p>from lxml import etree</p><p># Reading temperature of New York</p><p>URL = "https://weather.com/weather/today/l/40.75,-73.98"</p><p>resp = requests.get(URL)</p><p>if resp.status_code == 200:</p><p>    # Using lxml</p><p>    dom = etree.HTML(resp.text)</p><p>    elements = dom.xpath("//span[@data-testid='TemperatureValue' and contains(@class,'CurrentConditions')]")</p><p>    print(elements[0].text)</p><p>    # Using BeautifulSoup</p><p>    soup = BeautifulSoup(resp.text, "lxml")</p><p>    elements = soup.select('span[data-testid="TemperatureValue"][class^="CurrentConditions"]')</p><p>    print(elements[0].text)</p></div></td>
</tr>
</tbody></table>
</div><p>As you can imagine, you can collect a time series of the temperature by running this script on a regular schedule. Similarly, we can collect data automatically from various websites. This is how we can obtain data for our machine learning projects.</p>
<h2>Reading Tables on the Web Using Pandas</h2>
<p>Very often, web pages will use tables to carry data. If the page is simple enough, we may even skip inspecting it to find out the XPath or CSS selector and use pandas to get all tables on the page in one shot. It is simple enough to be done in one line:</p>
<div>
<table>
<tbody><tr>
<td>
</td>
<td><div><p>import pandas as pd</p><p>tables = pd.read_html("https://www.federalreserve.gov/releases/h15/")</p><p>print(tables)</p></div></td>
</tr>
</tbody></table>
</div>
<div>
<table>
<tbody><tr>
<td>
<div><p>1</p><p>2</p><p>3</p><p>4</p><p>5</p><p>6</p><p>7</p><p>8</p><p>9</p><p>10</p><p>11</p><p>12</p><p>13</p><p>14</p><p>15</p><p>16</p><p>17</p><p>18</p><p>19</p><p>20</p><p>21</p><p>22</p><p>23</p><p>24</p><p>25</p><p>26</p><p>27</p><p>28</p><p>29</p><p>30</p><p>31</p><p>32</p><p>33</p><p>34</p><p>35</p><p>36</p><p>37</p><p>38</p><p>39</p><p>40</p></div>
</td>
<td><div><p>[                               Instruments 2022Apr7 2022Apr8 2022Apr11 2022Apr12 2022Apr13</p><p>0          Federal funds (effective) 1 2 3     0.33     0.33      0.33      0.33      0.33</p><p>1                 Commercial Paper 3 4 5 6      NaN      NaN       NaN       NaN       NaN</p><p>2                             Nonfinancial      NaN      NaN       NaN       NaN       NaN</p><p>3                                  1-month     0.30     0.34      0.36      0.39      0.39</p><p>4                                  2-month     n.a.     0.48      n.a.      n.a.      n.a.</p><p>5                                  3-month     n.a.     n.a.      n.a.      0.78      0.78</p><p>6                                Financial      NaN      NaN       NaN       NaN       NaN</p><p>7                                  1-month     0.49     0.45      0.46      0.39      0.46</p><p>8                                  2-month     n.a.     n.a.      0.60      0.71      n.a.</p><p>9                                  3-month     0.85     0.81      0.75      n.a.      0.86</p><p>10                   Bank prime loan 2 3 7     3.50     3.50      3.50      3.50      3.50</p><p>11      Discount window primary credit 2 8     0.50     0.50      0.50      0.50      0.50</p><p>12              U.S. government securities      NaN      NaN       NaN       NaN       NaN</p><p>13   Treasury bills (secondary market) 3 4      NaN      NaN       NaN       NaN       NaN</p><p>14                                  4-week     0.21     0.20      0.21      0.19      0.23</p><p>15                                 3-month     0.68     0.69      0.78      0.74      0.75</p><p>16                                 6-month     1.12     1.16      1.22      1.18      1.17</p><p>17                                  1-year     1.69     1.72      1.75      1.67      1.67</p><p>18            Treasury constant maturities      NaN      NaN       NaN       NaN       NaN</p><p>19                               Nominal 9      NaN      NaN       NaN       NaN       NaN</p><p>20                                 1-month     0.21     0.20      0.22      0.21      0.26</p><p>21                                 3-month     0.68     0.70      0.77      0.74      0.75</p><p>22                                 6-month     1.15     1.19      1.23      1.20      1.20</p><p>23                                  1-year     1.78     1.81      1.85      1.77      1.78</p><p>24                                  2-year     2.47     2.53      2.50      2.39      2.37</p><p>25                                  3-year     2.66     2.73      2.73      2.58      2.57</p><p>26                                  5-year     2.70     2.76      2.79      2.66      2.66</p><p>27                                  7-year     2.73     2.79      2.84      2.73      2.71</p><p>28                                 10-year     2.66     2.72      2.79      2.72      2.70</p><p>29                                 20-year     2.87     2.94      3.02      2.99      2.97</p><p>30                                 30-year     2.69     2.76      2.84      2.82      2.81</p><p>31                    Inflation indexed 10      NaN      NaN       NaN       NaN       NaN</p><p>32                                  5-year    -0.56    -0.57     -0.58     -0.65     -0.59</p><p>33                                  7-year    -0.34    -0.33     -0.32     -0.36     -0.31</p><p>34                                 10-year    -0.16    -0.15     -0.12     -0.14     -0.10</p><p>35                                 20-year     0.09     0.11      0.15      0.15      0.18</p><p>36                                 30-year     0.21     0.23      0.27      0.28      0.30</p><p>37  Inflation-indexed long-term average 11     0.23     0.26      0.30      0.30      0.33,       0               1</p><p>0  n.a.  Not available.]</p></div></td>
</tr>
</tbody></table>
</div><p>The <code>read_html()</code> function in pandas reads a URL and finds all the tables on the page. Each table is converted into a pandas DataFrame and then returns all of them in a list. In this example, we are reading the various interest rates from the Federal Reserve, which happens to have only one table on this page. The table columns are identified by pandas automatically.</p>
<p>Chances are that not all tables are what we are interested in. Sometimes, the web page will use a table merely as a way to format the page, but pandas may not be smart enough to tell. Hence we need to test and cherry-pick the result returned by the <code>read_html()</code> function.<br />
</p>
<h3>Want to Get Started With Python for Machine Learning?</h3>
<p>Take my free 7-day email crash course now (with sample code).</p>
<p>Click to sign-up and also get a free PDF Ebook version of the course.</p>
<h2>Reading Dynamic Content With Selenium</h2>
<p>A significant portion of modern-day web pages is full of JavaScripts. This gives us a fancier experience but becomes a hurdle to use as a program to extract data. One example is Yahoo’s home page, which, if we just load the page and find all news headlines, there are far fewer than what we can see on the browser:</p>
<div>
<table>
<tbody><tr>
<td>
</td>
<td><div><p>import requests</p><p># Read Yahoo home page</p><p>URL = "https://www.yahoo.com/"</p><p>resp = requests.get(URL)</p><p>dom = etree.HTML(resp.text)</p><p># Print news headlines</p><p>elements = dom.xpath("//h3/a[u[@class='StretchedBox']]")</p><p>for elem in elements:</p><p>    print(etree.tostring(elem, method="text", encoding="unicode"))</p></div></td>
</tr>
</tbody></table>
</div><p>This is because web pages like this rely on JavaScript to populate the content. Famous web frameworks such as AngularJS or React are behind powering this category. The Python library, such as <code>requests</code>, does not understand JavaScript. Therefore, you will see the result differently. If the data you want to fetch from the web is one of them, you can study how the JavaScript is invoked and mimic the browser’s behavior in your program. But this is probably too tedious to make it work.</p>
<p>The other way is to ask a real browser to read the web page rather than using <code>requests</code>. This is what Selenium can do. Before we can use it, we need to install the library:</p>
<p>But Selenium is only a framework to control browsers. You need to have the browser installed on your computer as well as the driver to connect Selenium to the browser. If you intend to use Chrome, you need to download and install <a href="https://chromedriver.chromium.org/downloads">ChromeDriver</a> too. You need to put the driver in the executable path so that Selenium can invoke it like a normal command. For example, in Linux, you just need to get the <code>chromedriver</code> executable from the ZIP file downloaded and put it in <code>/usr/local/bin</code>.</p>
<p>Similarly, if you’re using Firefox, you need the <a href="https://github.com/mozilla/geckodriver/releases/">GeckoDriver</a>. For more details on setting up Selenium, you should refer to <a href="https://www.selenium.dev/downloads/">its documentation</a>.</p>
<p>Afterward, you can use a Python script to control the browser behavior. For example:</p>
<div>
<table>
<tbody><tr>
<td>
<div><p>1</p><p>2</p><p>3</p><p>4</p><p>5</p><p>6</p><p>7</p><p>8</p><p>9</p><p>10</p><p>11</p><p>12</p><p>13</p><p>14</p><p>15</p><p>16</p><p>17</p><p>18</p><p>19</p><p>20</p><p>21</p><p>22</p><p>23</p><p>24</p><p>25</p><p>26</p><p>27</p><p>28</p><p>29</p><p>30</p><p>31</p></div>
</td>
<td><div><p>import time</p><p>from selenium import webdriver</p><p>from selenium.webdriver.support.ui import WebDriverWait</p><p>from selenium.webdriver.common.by import By</p><p># Launch Chrome browser in headless mode</p><p>options = webdriver.ChromeOptions()</p><p>options.add_argument("headless")</p><p>browser = webdriver.Chrome(options=options)</p><p># Load web page</p><p>browser.get("https://www.yahoo.com")</p><p># Network transport takes time. Wait until the page is fully loaded</p><p>def is_ready(browser):</p><p>    return browser.execute_script(r"""</p><p>        return document.readyState === 'complete'</p><p>    """)</p><p>WebDriverWait(browser, 30).until(is_ready)</p><p># Scroll to bottom of the page to trigger JavaScript action</p><p>browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")</p><p>time.sleep(1)</p><p>WebDriverWait(browser, 30).until(is_ready)</p><p># Search for news headlines and print</p><p>elements = browser.find_elements(By.XPATH, "//h3/a[u[@class='StretchedBox']]")</p><p>for elem in elements:</p><p>    print(elem.text)</p><p># Close the browser once finish</p><p>browser.close()</p></div></td>
</tr>
</tbody></table>
</div><p>The above code works as follows. We first launch the browser in headless mode, meaning we ask Chrome to start but not display on the screen. This is important if we want to run our script remotely as there may not be any GUI support. Note that every browser is developed differently, and thus the options syntax we used is specific to Chrome. If we use Firefox, the code would be this instead:</p>
<div>
<table>
<tbody><tr>
<td>
</td>
<td><div><p>options = webdriver.FirefoxOptions()</p><p>options.set_headless()</p><p>browser = webdriver.Firefox(firefox_options=options)</p></div></td>
</tr>
</tbody></table>
</div><p>After we launch the browser, we give it a URL to load. But since it takes time for the network to deliver the page, and the browser will take time to render it, we should wait until the browser is ready before we proceed to the next operation. We detect if the browser has finished rendering by using JavaScript. We make Selenium run a JavaScript code for us and tell us the result using the <code>execute_script()</code> function. We leverage Selenium’s <code>WebDriverWait</code> tool to run it until it succeeds or until a 30-second timeout. As the page is loaded, we scroll to the bottom of the page so the JavaScript can be triggered to load more content. Then we wait for one second unconditionally to make sure the browser triggered the JavaScript, then wait until the page is ready again. Afterward, we can extract the news headline element using XPath (or alternatively using a CSS selector). Because the browser is an external program, we are responsible for closing it in our script.</p>
<p>Using Selenium is different from using the <code>requests</code> library in several aspects. First, you never have the web content in your Python code directly. Instead, you refer to the browser’s content whenever you need it. Hence the web elements returned by the <code>find_elements()</code> function refer to objects inside the external browser, so we must not close the browser before we finish consuming them. Secondly, all operations should be based on browser interaction rather than network requests. Thus you need to control the browser by emulating keyboard and mouse movements. But in return, you have the full-featured browser with JavaScript support. For example, you can use JavaScript to check the size and position of an element on the page, which you will know only after the HTML elements are rendered.</p>
<p>There are a lot more functions provided by the Selenium framework that we can cover here. It is powerful, but since it is connected to the browser, using it is more demanding than the <code>requests</code> library and much slower. Usually, this is the last resort for harvesting information from the web.</p>
<h2>Further Reading</h2>
<p>Another famous web crawling library in Python that we didn’t cover above is Scrapy. It is like combining the <code>requests</code> library with BeautifulSoup into one. The web protocol is complex. Sometimes we need to manage web cookies or provide extra data to the requests using the POST method. All these can be done with the requests library with a different function or extra arguments. The following are some resources for you to go deeper:</p>
<h4>Articles</h4>
<ul>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview">An overview of HTTP from MDN</a></li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/XPath">XPath from MDN</a></li>
<li><a href="https://www.w3schools.com/xml/xpath_intro.asp">XPath tutorial from W3Schools</a></li>
<li><a href="https://www.w3schools.com/cssref/css_selectors.asp">CSS Selector Reference from W3Schools</a></li>
<li><a href="https://www.selenium.dev/selenium/docs/api/py/index.html">Selenium Python binding</a></li>
</ul>
<h4>API documentation</h4>
<ul>
<li><a href="https://docs.python-requests.org/en/latest/">Requests library</a></li>
<li><a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">BeautifulSoup</a></li>
<li><a href="https://lxml.de/api.html">lxml.etree</a></li>
<li><a href="https://selenium-python.readthedocs.io/">Selenium Python API</a></li>
<li><a href="https://docs.scrapy.org/en/latest/">Scrapy</a></li>
</ul>
<h4>Books</h4>
<ul>
<li><a href="https://www.amazon.com/dp/1786462583">Python Web Scraping</a>, 2nd Edition, by Katharine Jarmul and Richard Lawson</li>
<li><a href="https://www.amazon.com/dp/1491985577/">Web Scraping with Python</a>, 2nd Edition, by Ryan Mitchell</li>
<li><a href="https://www.amazon.com/dp/1784399787/">Learning Scrapy</a>, by Dimitrios Kouzis-Loukas</li>
<li><a href="https://www.amazon.com/dp/1484262484/">Python Testing with Selenium</a>, by Sujay Raghavendra</li>
<li><a href="https://www.amazon.com/dp/1789533392">Hands-On Web Scraping with Python</a>, by Anish Chapagain</li>
</ul>
<h2><strong>Summary</strong></h2>
<p>In this tutorial, you saw the tools we can use to fetch content from the web.</p>
<p>Specifically, you learned:</p>
<ul>
<li>How to use the requests library to send the HTTP request and extract data from its response</li>
<li>How to build a document object model from HTML so we can find some specific information on a web page</li>
<li>How to read tables on a web page quickly and easily using pandas</li>
<li>How to use Selenium to control a browser to tackle dynamic content on a web page</li>
</ul>
<div>
<h2>Get a Handle on Python for Machine Learning!</h2>
<p><a href="/python-for-machine-learning/"></a></p><h4>Be More Confident to Code in Python</h4>
<p>...from learning the practical Python tricks</p>
<p>Discover how in my new Ebook:<br />
<a href="/python-for-machine-learning/">Python for Machine Learning</a></p>
<p>It provides <strong>self-study tutorials</strong> with <strong>hundreds of working code</strong> to equip you with skills including:<br />
<em>debugging</em>, <em>profiling</em>, <em>duck typing</em>, <em>decorators</em>, <em>deployment</em>,
and much more...</p>
<h4>Showing You the Python Toolbox at a High Level for <br />Your Projects</h4>
<p><a href="/python-for-machine-learning/">See What's Inside</a></p>
</div>
 </section>
</article>
 
</section>||||I|||| Navigation

 MachineLearningMastery.com Making developers awesome at machine learning
 Click to Take the FREE Python for Machine Learning Crash-Course
 Home

 Main Menu

 * Get Started
 * Blog
 * Topics
 + Attention
 + Deep Learning (keras)
 + Computer Vision
 + Neural Net Time Series
 + NLP (Text)
 + GANs
 + LSTMs
 + Better Deep Learning
 + Calculus
 + Intro to Algorithms
 + Code Algorithms
 + Intro to Time Series
 + Python (scikit-learn)
 + Ensemble Learning
 + Imbalanced Learning
 + Data Preparation
 + R (caret)
 + Weka (no code)
 + Linear Algebra
 + Statistics
 + Optimization
 + Probability
 + XGBoost
 + Python for Machine Learning
 * EBooks
 * FAQ
 * About
 * Contact
 Return to Content

 Web Crawling in Python

 By Adrian Tam on April 16, 2022 in Python for Machine Learning
 Tweet Tweet Share Share

 Last Updated on June 21, 2022

 In the old days, it was a tedious job to collect data, and it was sometimes very expensive. Machine learning projects cannot live without data. Luckily, we have a lot of data on the web at our disposal nowadays. We can copy data from the web to create our dataset. We can manually download files and save them to the disk. But we can do it more efficiently by automating the data harvesting. There are several tools in Python that can help the automation.

 After finishing this tutorial, you will learn:

 * How to use the requests library to read online data using HTTP
 * How to read tables on web pages using pandas
 * How to use Selenium to emulate browser operations

 Kick-start your project with my new book Python for Machine Learning, including step-by-step tutorials and the Python source code files for all examples.

 Let’s get started!

 Web Crawling in Python
 Photo by Ray Bilcliff. Some rights reserved.

 Overview

 This tutorial is divided into three parts; they are:

 * Using the requests library
 * Reading tables on the web using pandas
 * Reading dynamic content with Selenium

 Using the Requests Library

 When we talk about writing a Python program to read from the web, it is inevitable that we can’t avoid the requests library. You need to install it (as well as BeautifulSoup and lxml that we will cover later):


 pip install requests beautifulsoup4 lxml


 1 pip install requests beautifulsoup4 lxml
 


 It provides you with an interface that allows you to interact with the web easily.

 The very simple use case would be to read a web page from a URL:


 import requests # Lat-Lon of New York URL = "https://weather.com/weather/today/l/40.75,-73.98" resp = requests.get(URL) print(resp.status_code) print(resp.text)


 1 import requests 
 2 # Lat-Lon of New York 
 3 URL = "https://weather.com/weather/today/l/40.75,-73.98"
 4 resp = requests . get ( URL ) 
 5 print ( resp . status_code ) 
 6 print ( resp . text ) 
 7 
 


 200 <!doctype html><html dir="ltr" lang="en-US"><head> <meta data-react-helmet="true" charset="utf-8"/><meta data-react-helmet="true" name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/> ...


 1 200  
 2 <!doctype html><html dir="ltr" lang="en-US"><head> 
 3 <meta data-react-helmet="true" charset="utf-8"/><meta data-react-helmet="true" 
 4 name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/>
 5 ...  
 


 If you’re familiar with HTTP, you can probably recall that a status code of 200 means the request is successfully fulfilled. Then we can read the response. In the above, we read the textual response and get the HTML of the web page. Should it be a CSV or some other textual data, we can get them in the text attribute of the response object. For example, this is how we can read a CSV from the Federal Reserve Economics Data:


 import io import pandas as pd import requests URL = "https://fred.stlouisfed.org/graph/fredgraph.csv?id=T10YIE&cosd=2017-04-14&coed=2022-04-14" resp = requests.get(URL) if resp.status_code == 200: csvtext = resp.text csvbuffer = io.StringIO(csvtext) df = pd.read_csv(csvbuffer) print(df)


 1 import io  
 2 import pandas as pd  
 3 import requests  
 4 URL = "https://fred.stlouisfed.org/graph/fredgraph.csv?id=T10YIE&cosd=2017-04-14&coed=2022-04-14"
 5 resp = requests . get ( URL )  
 6 if resp . status_code == 200 :  
 7 csvtext = resp . text  
 8 csvbuffer = io . StringIO ( csvtext ) 
 9 df = pd . read_csv ( csvbuffer )  
 10 print ( df )  
 11  
 


 DATE T10YIE 0 2017-04-17 1.88 1 2017-04-18 1.85 2 2017-04-19 1.85 3 2017-04-20 1.85 4 2017-04-21 1.84 ... ... ... 1299 2022-04-08 2.87 1300 2022-04-11 2.91 1301 2022-04-12 2.86 1302 2022-04-13 2.8 1303 2022-04-14 2.89 [1304 rows x 2 columns]


 1 DATE T10YIE 
 2 0 2017-04-17 1.88 
 3 1 2017-04-18 1.85 
 4 2 2017-04-19 1.85 
 5 3 2017-04-20 1.85 
 6 4 2017-04-21 1.84 
 7 ... ... ... 
 8 1299 2022-04-08 2.87 
 9 1300 2022-04-11 2.91 
 10 1301 2022-04-12 2.86 
 11 1302 2022-04-13 2.8 
 12 1303 2022-04-14 2.89 
 13 [1304 rows x 2 columns]
 14 
 


 If the data is in the form of JSON, we can read it as text or even let requests decode it for you. For example, the following is to pull some data from GitHub in JSON format and convert it into a Python dictionary:


 import requests URL = "https://api.github.com/users/jbrownlee" resp = requests.get(URL) if resp.status_code == 200: data = resp.json() print(data)


 1 import requests 
 2 URL = "https://api.github.com/users/jbrownlee"
 3 resp = requests . get ( URL ) 
 4 if resp . status_code == 200 : 
 5 data = resp . json ( ) 
 6 print ( data ) 
 7 
 


 {'login': 'jbrownlee', 'id': 12891, 'node_id': 'MDQ6VXNlcjEyODkx', 'avatar_url': 'https://avatars.githubusercontent.com/u/12891?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/jbrownlee', 'html_url': 'https://github.com/jbrownlee', ... 'company': 'Machine Learning Mastery', 'blog': 'http://MachineLearningMastery.com', 'location': None, 'email': None, 'hireable': None, 'bio': 'Making developers awesome at machine learning.', 'twitter_username': None, 'public_repos': 5, 'public_gists': 0, 'followers': 1752, 'following': 0, 'created_at': '2008-06-07T02:20:58Z', 'updated_at': '2022-02-22T19:56:27Z' }


 1 {'login': 'jbrownlee', 'id': 12891, 'node_id': 'MDQ6VXNlcjEyODkx', 
 2 'avatar_url': 'https://avatars.githubusercontent.com/u/12891?v=4', 
 3 'gravatar_id': '', 'url': 'https://api.github.com/users/jbrownlee', 
 4 'html_url': 'https://github.com/jbrownlee', 
 5 ...  
 6 'company': 'Machine Learning Mastery', 'blog': 'http://MachineLearningMastery.com',
 7 'location': None, 'email': None, 'hireable': None, 
 8 'bio': 'Making developers awesome at machine learning.', 'twitter_username': None, 
 9 'public_repos': 5, 'public_gists': 0, 'followers': 1752, 'following': 0, 
 10 'created_at': '2008-06-07T02:20:58Z', 'updated_at': '2022-02-22T19:56:27Z' 
 11 }  
 


 But if the URL gives you some binary data, such as a ZIP file or a JPEG image, you need to get them in the content attribute instead, as this would be the binary data. For example, this is how we can download an image (the logo of Wikipedia):


 import requests URL = "https://en.wikipedia.org/static/images/project-logos/enwiki.png" wikilogo = requests.get(URL) if wikilogo.status_code == 200: with open("enwiki.png", "wb") as fp: fp.write(wikilogo.content)


 1 import requests 
 2 URL = "https://en.wikipedia.org/static/images/project-logos/enwiki.png"
 3 wikilogo = requests . get ( URL ) 
 4 if wikilogo . status_code == 200 : 
 5 with open ( "enwiki.png" , "wb" ) as fp : 
 6 fp . write ( wikilogo . content ) 
 7  
 


 Given we already obtained the web page, how should we extract the data? This is beyond what the requests library can provide to us, but we can use a different library to help. There are two ways we can do it, depending on how we want to specify the data.

 The first way is to consider the HTML as a kind of XML document and use the XPath language to extract the element. In this case, we can make use of the lxml library to first create a document object model (DOM) and then search by XPath:


 ... from lxml import etree # Create DOM from HTML text dom = etree.HTML(resp.text) # Search for the temperature element and get the content elements = dom.xpath("//span[@data-testid='TemperatureValue' and contains(@class,'CurrentConditions')]") print(elements[0].text)


 1 . . .  
 2 from lxml import etree  
 3 # Create DOM from HTML text  
 4 dom = etree . HTML ( resp . text )  
 5 # Search for the temperature element and get the content 
 6 elements = dom . xpath ( "//span[@data-testid='TemperatureValue' and contains(@class,'CurrentConditions')]" )
 7 print ( elements [ 0 ] . text )  
 8  
 


 61°


 1 61°
 


 XPath is a string that specifies how to find an element. The lxml object provides a function xpath() to search the DOM for elements that match the XPath string, which can be multiple matches. The XPath above means to find an HTML element anywhere with the <span> tag and with the attribute data-testid matching “TemperatureValue” and class beginning with “CurrentConditions.” We can learn this from the developer tools of the browser (e.g., the Chrome screenshot below) by inspecting the HTML source.

 This example is to find the temperature of New York City, provided by this particular element we get from this web page. We know the first element matched by the XPath is what we need, and we can read the text inside the <span> tag.

 The other way is to use CSS selectors on the HTML document, which we can make use of the BeautifulSoup library:


 ... from bs4 import BeautifulSoup soup = BeautifulSoup(resp.text, "lxml") elements = soup.select('span[data-testid="TemperatureValue"][class^="CurrentConditions"]') print(elements[0].text)


 1 . . .  
 2 from bs4 import BeautifulSoup  
 3 soup = BeautifulSoup ( resp . text , "lxml" ) 
 4 elements = soup . select ( 'span[data-testid="TemperatureValue"][class^="CurrentConditions"]' )
 5 print ( elements [ 0 ] . text ) 
 6  
 


 61°


 1 61 °
 


 In the above, we first pass our HTML text to BeautifulSoup. BeautifulSoup supports various HTML parsers, each with different capabilities. In the above, we use the lxml library as the parser as recommended by BeautifulSoup (and it is also often the fastest). CSS selector is a different mini-language, with pros and cons compared to XPath. The selector above is identical to the XPath we used in the previous example. Therefore, we can get the same temperature from the first matched element.

 The following is a complete code to print the current temperature of New York according to the real-time information on the web:


 import requests from lxml import etree # Reading temperature of New York URL = "https://weather.com/weather/today/l/40.75,-73.98" resp = requests.get(URL) if resp.status_code == 200: # Using lxml dom = etree.HTML(resp.text) elements = dom.xpath("//span[@data-testid='TemperatureValue' and contains(@class,'CurrentConditions')]") print(elements[0].text) # Using BeautifulSoup soup = BeautifulSoup(resp.text, "lxml") elements = soup.select('span[data-testid="TemperatureValue"][class^="CurrentConditions"]') print(elements[0].text)


 1  
 2 import requests  
 3 from lxml import etree  
 4 # Reading temperature of New York  
 5 URL = "https://weather.com/weather/today/l/40.75,-73.98" 
 6 resp = requests . get ( URL )  
 7 if resp . status_code == 200 :  
 8 # Using lxml  
 9 dom = etree . HTML ( resp . text )  
 10 elements = dom . xpath ( "//span[@data-testid='TemperatureValue' and contains(@class,'CurrentConditions')]" )
 11 print ( elements [ 0 ] . text )  
 12 # Using BeautifulSoup  
 13 soup = BeautifulSoup ( resp . text , "lxml" ) 
 14 elements = soup . select ( 'span[data-testid="TemperatureValue"][class^="CurrentConditions"]' ) 
 15 print ( elements [ 0 ] . text )  
 16  
 17  
 


 As you can imagine, you can collect a time series of the temperature by running this script on a regular schedule. Similarly, we can collect data automatically from various websites. This is how we can obtain data for our machine learning projects.

 Reading Tables on the Web Using Pandas

 Very often, web pages will use tables to carry data. If the page is simple enough, we may even skip inspecting it to find out the XPath or CSS selector and use pandas to get all tables on the page in one shot. It is simple enough to be done in one line:


 import pandas as pd tables = pd.read_html("https://www.federalreserve.gov/releases/h15/") print(tables)


 1 import pandas as pd 
 2 tables = pd . read_html ( "https://www.federalreserve.gov/releases/h15/" )
 3 print ( tables ) 
 4  
 


 [ Instruments 2022Apr7 2022Apr8 2022Apr11 2022Apr12 2022Apr13 0 Federal funds (effective) 1 2 3 0.33 0.33 0.33 0.33 0.33 1 Commercial Paper 3 4 5 6 NaN NaN NaN NaN NaN 2 Nonfinancial NaN NaN NaN NaN NaN 3 1-month 0.30 0.34 0.36 0.39 0.39 4 2-month n.a. 0.48 n.a. n.a. n.a. 5 3-month n.a. n.a. n.a. 0.78 0.78 6 Financial NaN NaN NaN NaN NaN 7 1-month 0.49 0.45 0.46 0.39 0.46 8 2-month n.a. n.a. 0.60 0.71 n.a. 9 3-month 0.85 0.81 0.75 n.a. 0.86 10 Bank prime loan 2 3 7 3.50 3.50 3.50 3.50 3.50 11 Discount window primary credit 2 8 0.50 0.50 0.50 0.50 0.50 12 U.S. government securities NaN NaN NaN NaN NaN 13 Treasury bills (secondary market) 3 4 NaN NaN NaN NaN NaN 14 4-week 0.21 0.20 0.21 0.19 0.23 15 3-month 0.68 0.69 0.78 0.74 0.75 16 6-month 1.12 1.16 1.22 1.18 1.17 17 1-year 1.69 1.72 1.75 1.67 1.67 18 Treasury constant maturities NaN NaN NaN NaN NaN 19 Nominal 9 NaN NaN NaN NaN NaN 20 1-month 0.21 0.20 0.22 0.21 0.26 21 3-month 0.68 0.70 0.77 0.74 0.75 22 6-month 1.15 1.19 1.23 1.20 1.20 23 1-year 1.78 1.81 1.85 1.77 1.78 24 2-year 2.47 2.53 2.50 2.39 2.37 25 3-year 2.66 2.73 2.73 2.58 2.57 26 5-year 2.70 2.76 2.79 2.66 2.66 27 7-year 2.73 2.79 2.84 2.73 2.71 28 10-year 2.66 2.72 2.79 2.72 2.70 29 20-year 2.87 2.94 3.02 2.99 2.97 30 30-year 2.69 2.76 2.84 2.82 2.81 31 Inflation indexed 10 NaN NaN NaN NaN NaN 32 5-year -0.56 -0.57 -0.58 -0.65 -0.59 33 7-year -0.34 -0.33 -0.32 -0.36 -0.31 34 10-year -0.16 -0.15 -0.12 -0.14 -0.10 35 20-year 0.09 0.11 0.15 0.15 0.18 36 30-year 0.21 0.23 0.27 0.28 0.30 37 Inflation-indexed long-term average 11 0.23 0.26 0.30 0.30 0.33, 0 1 0 n.a. Not available.]


 1 [ Instruments 2022Apr7 2022Apr8 2022Apr11 2022Apr12 2022Apr13 
 2 0 Federal funds (effective) 1 2 3 0.33 0.33 0.33 0.33 0.33 
 3 1 Commercial Paper 3 4 5 6 NaN NaN NaN NaN NaN 
 4 2 Nonfinancial NaN NaN NaN NaN NaN 
 5 3 1-month 0.30 0.34 0.36 0.39 0.39 
 6 4 2-month n.a. 0.48 n.a. n.a. n.a. 
 7 5 3-month n.a. n.a. n.a. 0.78 0.78 
 8 6 Financial NaN NaN NaN NaN NaN 
 9 7 1-month 0.49 0.45 0.46 0.39 0.46 
 10 8 2-month n.a. n.a. 0.60 0.71 n.a. 
 11 9 3-month 0.85 0.81 0.75 n.a. 0.86 
 12 10 Bank prime loan 2 3 7 3.50 3.50 3.50 3.50 3.50 
 13 11 Discount window primary credit 2 8 0.50 0.50 0.50 0.50 0.50 
 14 12 U.S. government securities NaN NaN NaN NaN NaN 
 15 13 Treasury bills (secondary market) 3 4 NaN NaN NaN NaN NaN 
 16 14 4-week 0.21 0.20 0.21 0.19 0.23 
 17 15 3-month 0.68 0.69 0.78 0.74 0.75 
 18 16 6-month 1.12 1.16 1.22 1.18 1.17 
 19 17 1-year 1.69 1.72 1.75 1.67 1.67 
 20 18 Treasury constant maturities NaN NaN NaN NaN NaN 
 21 19 Nominal 9 NaN NaN NaN NaN NaN 
 22 20 1-month 0.21 0.20 0.22 0.21 0.26 
 23 21 3-month 0.68 0.70 0.77 0.74 0.75 
 24 22 6-month 1.15 1.19 1.23 1.20 1.20 
 25 23 1-year 1.78 1.81 1.85 1.77 1.78 
 26 24 2-year 2.47 2.53 2.50 2.39 2.37 
 27 25 3-year 2.66 2.73 2.73 2.58 2.57 
 28 26 5-year 2.70 2.76 2.79 2.66 2.66 
 29 27 7-year 2.73 2.79 2.84 2.73 2.71 
 30 28 10-year 2.66 2.72 2.79 2.72 2.70 
 31 29 20-year 2.87 2.94 3.02 2.99 2.97 
 32 30 30-year 2.69 2.76 2.84 2.82 2.81 
 33 31 Inflation indexed 10 NaN NaN NaN NaN NaN 
 34 32 5-year -0.56 -0.57 -0.58 -0.65 -0.59 
 35 33 7-year -0.34 -0.33 -0.32 -0.36 -0.31 
 36 34 10-year -0.16 -0.15 -0.12 -0.14 -0.10 
 37 35 20-year 0.09 0.11 0.15 0.15 0.18 
 38 36 30-year 0.21 0.23 0.27 0.28 0.30 
 39 37 Inflation-indexed long-term average 11 0.23 0.26 0.30 0.30 0.33, 0 1
 40 0 n.a. Not available.] 
 


 The read_html() function in pandas reads a URL and finds all the tables on the page. Each table is converted into a pandas DataFrame and then returns all of them in a list. In this example, we are reading the various interest rates from the Federal Reserve, which happens to have only one table on this page. The table columns are identified by pandas automatically.

 Chances are that not all tables are what we are interested in. Sometimes, the web page will use a table merely as a way to format the page, but pandas may not be smart enough to tell. Hence we need to test and cherry-pick the result returned by the read_html() function.

 Want to Get Started With Python for Machine Learning?

 Take my free 7-day email crash course now (with sample code).

 Click to sign-up and also get a free PDF Ebook version of the course.

 Download Your FREE Mini-Course

 Reading Dynamic Content With Selenium

 A significant portion of modern-day web pages is full of JavaScripts. This gives us a fancier experience but becomes a hurdle to use as a program to extract data. One example is Yahoo’s home page, which, if we just load the page and find all news headlines, there are far fewer than what we can see on the browser:


 import requests # Read Yahoo home page URL = "https://www.yahoo.com/" resp = requests.get(URL) dom = etree.HTML(resp.text) # Print news headlines elements = dom.xpath("//h3/a[u[@class='StretchedBox']]") for elem in elements: print(etree.tostring(elem, method="text", encoding="unicode"))


 1  
 2 import requests 
 3 # Read Yahoo home page 
 4 URL = "https://www.yahoo.com/" 
 5 resp = requests . get ( URL ) 
 6 dom = etree . HTML ( resp . text ) 
 7 # Print news headlines 
 8 elements = dom . xpath ( "//h3/a[u[@class='StretchedBox']]" ) 
 9 for elem in elements : 
 10 print ( etree . tostring ( elem , method = "text" , encoding = "unicode" ) )
 11  
 


 This is because web pages like this rely on JavaScript to populate the content. Famous web frameworks such as AngularJS or React are behind powering this category. The Python library, such as requests, does not understand JavaScript. Therefore, you will see the result differently. If the data you want to fetch from the web is one of them, you can study how the JavaScript is invoked and mimic the browser’s behavior in your program. But this is probably too tedious to make it work.

 The other way is to ask a real browser to read the web page rather than using requests. This is what Selenium can do. Before we can use it, we need to install the library:


 pip install selenium


 1 pip install selenium
 


 But Selenium is only a framework to control browsers. You need to have the browser installed on your computer as well as the driver to connect Selenium to the browser. If you intend to use Chrome, you need to download and install ChromeDriver too. You need to put the driver in the executable path so that Selenium can invoke it like a normal command. For example, in Linux, you just need to get the chromedriver executable from the ZIP file downloaded and put it in /usr/local/bin.

 Similarly, if you’re using Firefox, you need the GeckoDriver. For more details on setting up Selenium, you should refer to its documentation.

 Afterward, you can use a Python script to control the browser behavior. For example:


 import time from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By # Launch Chrome browser in headless mode options = webdriver.ChromeOptions() options.add_argument("headless") browser = webdriver.Chrome(options=options) # Load web page browser.get("https://www.yahoo.com") # Network transport takes time. Wait until the page is fully loaded def is_ready(browser): return browser.execute_script(r""" return document.readyState === 'complete' """) WebDriverWait(browser, 30).until(is_ready) # Scroll to bottom of the page to trigger JavaScript action browser.execute_script("window.scrollTo(0, document.body.scrollHeight);") time.sleep(1) WebDriverWait(browser, 30).until(is_ready) # Search for news headlines and print elements = browser.find_elements(By.XPATH, "//h3/a[u[@class='StretchedBox']]") for elem in elements: print(elem.text) # Close the browser once finish browser.close()


 1  
 2  
 3 import time  
 4 from selenium import webdriver 
 5 from selenium . webdriver . support . ui import WebDriverWait 
 6 from selenium . webdriver . common . by import By 
 7 # Launch Chrome browser in headless mode 
 8 options = webdriver . ChromeOptions ( ) 
 9 options . add_argument ( "headless" ) 
 10 browser = webdriver . Chrome ( options = options ) 
 11 # Load web page  
 12 browser . get ( "https://www.yahoo.com" ) 
 13 # Network transport takes time. Wait until the page is fully loaded 
 14 def is_ready ( browser ) : 
 15 return browser . execute_script ( r "" " 
 16 return document.readyState === 'complete' 
 17 " "" )  
 18 WebDriverWait ( browser , 30 ) . until ( is_ready ) 
 19 # Scroll to bottom of the page to trigger JavaScript action 
 20 browser . execute_script ( "window.scrollTo(0, document.body.scrollHeight);" ) 
 21 time . sleep ( 1 )  
 22 WebDriverWait ( browser , 30 ) . until ( is_ready ) 
 23 # Search for news headlines and print 
 24 elements = browser . find_elements ( By . XPATH , "//h3/a[u[@class='StretchedBox']]" )
 25 for elem in elements : 
 26 print ( elem . text )  
 27 # Close the browser once finish 
 28 browser . close ( )  
 29  
 30  
 31  
 


 The above code works as follows. We first launch the browser in headless mode, meaning we ask Chrome to start but not display on the screen. This is important if we want to run our script remotely as there may not be any GUI support. Note that every browser is developed differently, and thus the options syntax we used is specific to Chrome. If we use Firefox, the code would be this instead:


 options = webdriver.FirefoxOptions() options.set_headless() browser = webdriver.Firefox(firefox_options=options)


 1 options = webdriver . FirefoxOptions ( ) 
 2 options . set_headless ( ) 
 3 browser = webdriver . Firefox ( firefox_options = options )
 


 After we launch the browser, we give it a URL to load. But since it takes time for the network to deliver the page, and the browser will take time to render it, we should wait until the browser is ready before we proceed to the next operation. We detect if the browser has finished rendering by using JavaScript. We make Selenium run a JavaScript code for us and tell us the result using the execute_script() function. We leverage Selenium’s WebDriverWait tool to run it until it succeeds or until a 30-second timeout. As the page is loaded, we scroll to the bottom of the page so the JavaScript can be triggered to load more content. Then we wait for one second unconditionally to make sure the browser triggered the JavaScript, then wait until the page is ready again. Afterward, we can extract the news headline element using XPath (or alternatively using a CSS selector). Because the browser is an external program, we are responsible for closing it in our script.

 Using Selenium is different from using the requests library in several aspects. First, you never have the web content in your Python code directly. Instead, you refer to the browser’s content whenever you need it. Hence the web elements returned by the find_elements() function refer to objects inside the external browser, so we must not close the browser before we finish consuming them. Secondly, all operations should be based on browser interaction rather than network requests. Thus you need to control the browser by emulating keyboard and mouse movements. But in return, you have the full-featured browser with JavaScript support. For example, you can use JavaScript to check the size and position of an element on the page, which you will know only after the HTML elements are rendered.

 There are a lot more functions provided by the Selenium framework that we can cover here. It is powerful, but since it is connected to the browser, using it is more demanding than the requests library and much slower. Usually, this is the last resort for harvesting information from the web.

 Further Reading

 Another famous web crawling library in Python that we didn’t cover above is Scrapy. It is like combining the requests library with BeautifulSoup into one. The web protocol is complex. Sometimes we need to manage web cookies or provide extra data to the requests using the POST method. All these can be done with the requests library with a different function or extra arguments. The following are some resources for you to go deeper:

 Articles

 * An overview of HTTP from MDN
 * XPath from MDN
 * XPath tutorial from W3Schools
 * CSS Selector Reference from W3Schools
 * Selenium Python binding

 API documentation

 * Requests library
 * BeautifulSoup
 * lxml.etree
 * Selenium Python API
 * Scrapy

 Books

 * Python Web Scraping, 2nd Edition, by Katharine Jarmul and Richard Lawson
 * Web Scraping with Python, 2nd Edition, by Ryan Mitchell
 * Learning Scrapy, by Dimitrios Kouzis-Loukas
 * Python Testing with Selenium, by Sujay Raghavendra
 * Hands-On Web Scraping with Python, by Anish Chapagain

 Summary

 In this tutorial, you saw the tools we can use to fetch content from the web.

 Specifically, you learned:

 * How to use the requests library to send the HTTP request and extract data from its response
 * How to build a document object model from HTML so we can find some specific information on a web page
 * How to read tables on a web page quickly and easily using pandas
 * How to use Selenium to control a browser to tackle dynamic content on a web page

 Get a Handle on Python for Machine Learning!

 Be More Confident to Code in Python

 ...from learning the practical Python tricks

 Discover how in my new Ebook:
 Python for Machine Learning

 It provides self-study tutorials with hundreds of working code to equip you with skills including:
 debugging, profiling, duck typing, decorators, deployment, and much more...

 Showing You the Python Toolbox at a High Level for
 Your Projects


 See What's Inside

 Tweet Tweet Share Share

 More On This Topic

 * Web Frameworks for Your Python Projects

 * How To Use R For Machine Learning

 * How to Train Keras Deep Learning Models on AWS EC2…

 * A Gentle Introduction to Unit Testing in Python

 * Setting Breakpoints and Exception Hooks in Python

 * BigML Review: Discover the Clever Features in This…

 About Adrian Tam

 Adrian Tam, PhD is a data scientist and software engineer.
 View all posts by Adrian Tam →
 Massaging Data Using Pandas
 Managing Data for Machine Learning Projects

 14 Responses to Web Crawling in Python

 1. YF April 22, 2022 at 12:52 pm #

 there is a typo here.

 df = ppd.read_csv(csvbuffer)

 the “ppd” should be “pd”

 Reply
 + James Carmichael April 24, 2022 at 3:36 am #

 Thank you for the feedback YF!

 Reply
 2. Selda April 22, 2022 at 5:13 pm #

 Thank you for this detailed and useful page. One thing though, isn’t this web scraping? As far as I know web crawling is finding or discovering URLs or links on the web.

 Reply
 + James Carmichael April 24, 2022 at 3:35 am #

 Hi Selda…Yes, this technique is often referred to as web scraping.

 Reply
 3. Luis Gonzalez April 23, 2022 at 12:04 am #

 error:
 df = ppd.read_csv(csvbuffer)

 correct:

 df = pd.read_csv(csvbuffer)

 Reply
 + James Carmichael April 24, 2022 at 3:32 am #

 Hi Luis…Thank you for the feedback!

 Reply
 4. suchi banerjee April 26, 2022 at 2:21 pm #

 Great and comprehensive article. It helps in understanding different approaches. However, without any standard template for website development it is really difficult to have a generic tool to collect data from the web pages. Do you have any thought on this?

 Reply
 + James Carmichael April 30, 2022 at 10:23 am #

 Hi Suchi…the following may be of interest:

 https://machinelearningmastery.com/web-crawling-in-python/

 Reply
 5. Rina May 29, 2022 at 7:52 am #

 Thank your for the comprehensive steps. Can I know how to crawl Tweets data? Can anyone here provide the link?

 Reply
 + James Carmichael May 29, 2022 at 9:58 am #

 Hi Rina…The following resource may be of interest to you:

 https://machinelearningmastery.com/web-crawling-in-python/

 Reply
 6. Billie June 10, 2022 at 2:53 pm #

 Thank you for the tutorial. Can I know how to crawl data from private github repos?

 Reply
 + James Carmichael June 11, 2022 at 9:04 am #

 Thank you for the feedback Billie! I am not aware of a method to perform that task.

 Reply
 7. Abdul Hannan December 26, 2022 at 12:46 am #

 Hy , hope you are fine .
 Can I Know how to Crawl any Data from Any Website?

 Reply
 + James Carmichael December 26, 2022 at 8:36 am #

 Hi Abdul…The following resource may be of interest to you:

 https://medium.com/dataseries/build-a-crawler-to-extract-web-data-in-10-mins-691b2cc4f1c3

 Reply

 Leave a Reply

 Comment *

 Name (required)

 Email (will not be published) (required)

 Δ

 Welcome!
 I'm Jason Brownlee PhD
 and I help developers get results with machine learning.
 Read more

 Never miss a tutorial:


 Picked for you:


 Python for Machine Learning (7-day mini-course)
 Some Language Features in Python
 Web Crawling in Python
 Multiprocessing in Python
 How to Learn Python for Machine Learning

 Loving the Tutorials?

 The Python for Machine Learning is where
 you'll find the Really Good stuff.

 >> See What's Inside

 © 2023 Guiding Tech Media. All Rights Reserved.
 LinkedIn | Twitter | Facebook | Newsletter | RSS

 Privacy | Disclaimer | Terms | Contact | Sitemap | Search

Last Updated on June 21, 2022
In the old days, it was a tedious job to collect data, and it was sometimes very expensive. Machine learning projects cannot live without data. Luckily, we have a lot of data on the web at our disposal nowadays. We can copy data from the web to create our dataset. We can manually download files and save them to the disk. But we can do it more efficiently by automating the data harvesting. There are several tools in Python that can help the automation.
After finishing this tutorial, you will learn:
How to use the requests library to read online data using HTTP
How to read tables on web pages using pandas
How to use Selenium to emulate browser operations
 Kick-start your project with my new book Python for Machine Learning, including step-by-step tutorials and the Python source code files for all examples.Let’s get started!
   Web Crawling in PythonPhoto by Ray Bilcliff. Some rights reserved. 
Overview
This tutorial is divided into three parts; they are:
Using the requests library
Reading tables on the web using pandas
Reading dynamic content with Selenium
Using the Requests Library
When we talk about writing a Python program to read from the web, it is inevitable that we can’t avoid the requests library. You need to install it (as well as BeautifulSoup and lxml that we will cover later):
 
  pip install requests beautifulsoup4 lxml  
 
 It provides you with an interface that allows you to interact with the web easily.
The very simple use case would be to read a web page from a URL:
 
  import requests # Lat-Lon of New York URL = "https://weather.com/weather/today/l/40.75,-73.98" resp = requests.get(URL) print(resp.status_code) print(resp.text)  
 
 
  200 &lt;!doctype html&gt;&lt;html dir="ltr" lang="en-US"&gt;&lt;head&gt;       &lt;meta data-react-helmet="true" charset="utf-8"/&gt;&lt;meta data-react-helmet="true" name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/&gt; ...  
 
 If you’re familiar with HTTP, you can probably recall that a status code of 200 means the request is successfully fulfilled. Then we can read the response. In the above, we read the textual response and get the HTML of the web page. Should it be a CSV or some other textual data, we can get them in the text attribute of the response object. For example, this is how we can read a CSV from the Federal Reserve Economics Data:
 
  import io import pandas as pd import requests URL = "https://fred.stlouisfed.org/graph/fredgraph.csv?id=T10YIE&amp;cosd=2017-04-14&amp;coed=2022-04-14" resp = requests.get(URL) if resp.status_code == 200:    csvtext = resp.text    csvbuffer = io.StringIO(csvtext)    df = pd.read_csv(csvbuffer)    print(df)  
 
 
              DATE T10YIE 0     2017-04-17   1.88 1     2017-04-18   1.85 2     2017-04-19   1.85 3     2017-04-20   1.85 4     2017-04-21   1.84 ...          ...    ... 1299  2022-04-08   2.87 1300  2022-04-11   2.91 1301  2022-04-12   2.86 1302  2022-04-13    2.8 1303  2022-04-14   2.89 [1304 rows x 2 columns]  
 
 If the data is in the form of JSON, we can read it as text or even let requests decode it for you. For example, the following is to pull some data from GitHub in JSON format and convert it into a Python dictionary:
 
  import requests URL = "https://api.github.com/users/jbrownlee" resp = requests.get(URL) if resp.status_code == 200:     data = resp.json()     print(data)  
 
 
  {'login': 'jbrownlee', 'id': 12891, 'node_id': 'MDQ6VXNlcjEyODkx', 'avatar_url': 'https://avatars.githubusercontent.com/u/12891?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/jbrownlee', 'html_url': 'https://github.com/jbrownlee', ... 'company': 'Machine Learning Mastery', 'blog': 'http://MachineLearningMastery.com', 'location': None, 'email': None, 'hireable': None, 'bio': 'Making developers awesome at machine learning.', 'twitter_username': None, 'public_repos': 5, 'public_gists': 0, 'followers': 1752, 'following': 0, 'created_at': '2008-06-07T02:20:58Z', 'updated_at': '2022-02-22T19:56:27Z' }  
 
 But if the URL gives you some binary data, such as a ZIP file or a JPEG image, you need to get them in the content attribute instead, as this would be the binary data. For example, this is how we can download an image (the logo of Wikipedia):
 
  import requests URL = "https://en.wikipedia.org/static/images/project-logos/enwiki.png" wikilogo = requests.get(URL) if wikilogo.status_code == 200:     with open("enwiki.png", "wb") as fp:         fp.write(wikilogo.content)  
 
 Given we already obtained the web page, how should we extract the data? This is beyond what the requests library can provide to us, but we can use a different library to help. There are two ways we can do it, depending on how we want to specify the data.
The first way is to consider the HTML as a kind of XML document and use the XPath language to extract the element. In this case, we can make use of the lxml library to first create a document object model (DOM) and then search by XPath:
 
  ... from lxml import etree # Create DOM from HTML text dom = etree.HTML(resp.text) # Search for the temperature element and get the content elements = dom.xpath("//span[@data-testid='TemperatureValue' and contains(@class,'CurrentConditions')]") print(elements[0].text)  
 
XPath is a string that specifies how to find an element. The lxml object provides a function xpath() to search the DOM for elements that match the XPath string, which can be multiple matches. The XPath above means to find an HTML element anywhere with the &lt;span&gt; tag and with the attribute data-testid matching “TemperatureValue” and class beginning with “CurrentConditions.” We can learn this from the developer tools of the browser (e.g., the Chrome screenshot below) by inspecting the HTML source.
 
This example is to find the temperature of New York City, provided by this particular element we get from this web page. We know the first element matched by the XPath is what we need, and we can read the text inside the &lt;span&gt; tag.
The other way is to use CSS selectors on the HTML document, which we can make use of the BeautifulSoup library:
 
  ... from bs4 import BeautifulSoup soup = BeautifulSoup(resp.text, "lxml") elements = soup.select('span[data-testid="TemperatureValue"][class^="CurrentConditions"]') print(elements[0].text)  
 
In the above, we first pass our HTML text to BeautifulSoup. BeautifulSoup supports various HTML parsers, each with different capabilities. In the above, we use the lxml library as the parser as recommended by BeautifulSoup (and it is also often the fastest). CSS selector is a different mini-language, with pros and cons compared to XPath. The selector above is identical to the XPath we used in the previous example. Therefore, we can get the same temperature from the first matched element.
The following is a complete code to print the current temperature of New York according to the real-time information on the web:
 
 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 
  import requests from lxml import etree # Reading temperature of New York URL = "https://weather.com/weather/today/l/40.75,-73.98" resp = requests.get(URL) if resp.status_code == 200:     # Using lxml     dom = etree.HTML(resp.text)     elements = dom.xpath("//span[@data-testid='TemperatureValue' and contains(@class,'CurrentConditions')]")     print(elements[0].text)     # Using BeautifulSoup     soup = BeautifulSoup(resp.text, "lxml")     elements = soup.select('span[data-testid="TemperatureValue"][class^="CurrentConditions"]')     print(elements[0].text)  
 
 As you can imagine, you can collect a time series of the temperature by running this script on a regular schedule. Similarly, we can collect data automatically from various websites. This is how we can obtain data for our machine learning projects.
Reading Tables on the Web Using Pandas
Very often, web pages will use tables to carry data. If the page is simple enough, we may even skip inspecting it to find out the XPath or CSS selector and use pandas to get all tables on the page in one shot. It is simple enough to be done in one line:
 
  import pandas as pd tables = pd.read_html("https://www.federalreserve.gov/releases/h15/") print(tables)  
 
 
 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 
  [                               Instruments 2022Apr7 2022Apr8 2022Apr11 2022Apr12 2022Apr13 0          Federal funds (effective) 1 2 3     0.33     0.33      0.33      0.33      0.33 1                 Commercial Paper 3 4 5 6      NaN      NaN       NaN       NaN       NaN 2                             Nonfinancial      NaN      NaN       NaN       NaN       NaN 3                                  1-month     0.30     0.34      0.36      0.39      0.39 4                                  2-month     n.a.     0.48      n.a.      n.a.      n.a. 5                                  3-month     n.a.     n.a.      n.a.      0.78      0.78 6                                Financial      NaN      NaN       NaN       NaN       NaN 7                                  1-month     0.49     0.45      0.46      0.39      0.46 8                                  2-month     n.a.     n.a.      0.60      0.71      n.a. 9                                  3-month     0.85     0.81      0.75      n.a.      0.86 10                   Bank prime loan 2 3 7     3.50     3.50      3.50      3.50      3.50 11      Discount window primary credit 2 8     0.50     0.50      0.50      0.50      0.50 12              U.S. government securities      NaN      NaN       NaN       NaN       NaN 13   Treasury bills (secondary market) 3 4      NaN      NaN       NaN       NaN       NaN 14                                  4-week     0.21     0.20      0.21      0.19      0.23 15                                 3-month     0.68     0.69      0.78      0.74      0.75 16                                 6-month     1.12     1.16      1.22      1.18      1.17 17                                  1-year     1.69     1.72      1.75      1.67      1.67 18            Treasury constant maturities      NaN      NaN       NaN       NaN       NaN 19                               Nominal 9      NaN      NaN       NaN       NaN       NaN 20                                 1-month     0.21     0.20      0.22      0.21      0.26 21                                 3-month     0.68     0.70      0.77      0.74      0.75 22                                 6-month     1.15     1.19      1.23      1.20      1.20 23                                  1-year     1.78     1.81      1.85      1.77      1.78 24                                  2-year     2.47     2.53      2.50      2.39      2.37 25                                  3-year     2.66     2.73      2.73      2.58      2.57 26                                  5-year     2.70     2.76      2.79      2.66      2.66 27                                  7-year     2.73     2.79      2.84      2.73      2.71 28                                 10-year     2.66     2.72      2.79      2.72      2.70 29                                 20-year     2.87     2.94      3.02      2.99      2.97 30                                 30-year     2.69     2.76      2.84      2.82      2.81 31                    Inflation indexed 10      NaN      NaN       NaN       NaN       NaN 32                                  5-year    -0.56    -0.57     -0.58     -0.65     -0.59 33                                  7-year    -0.34    -0.33     -0.32     -0.36     -0.31 34                                 10-year    -0.16    -0.15     -0.12     -0.14     -0.10 35                                 20-year     0.09     0.11      0.15      0.15      0.18 36                                 30-year     0.21     0.23      0.27      0.28      0.30 37  Inflation-indexed long-term average 11     0.23     0.26      0.30      0.30      0.33,       0               1 0  n.a.  Not available.]  
 
 The read_html() function in pandas reads a URL and finds all the tables on the page. Each table is converted into a pandas DataFrame and then returns all of them in a list. In this example, we are reading the various interest rates from the Federal Reserve, which happens to have only one table on this page. The table columns are identified by pandas automatically.
Chances are that not all tables are what we are interested in. Sometimes, the web page will use a table merely as a way to format the page, but pandas may not be smart enough to tell. Hence we need to test and cherry-pick the result returned by the read_html() function.
Want to Get Started With Python for Machine Learning?
Take my free 7-day email crash course now (with sample code).
Click to sign-up and also get a free PDF Ebook version of the course.
Reading Dynamic Content With Selenium
A significant portion of modern-day web pages is full of JavaScripts. This gives us a fancier experience but becomes a hurdle to use as a program to extract data. One example is Yahoo’s home page, which, if we just load the page and find all news headlines, there are far fewer than what we can see on the browser:
 
  import requests # Read Yahoo home page URL = "https://www.yahoo.com/" resp = requests.get(URL) dom = etree.HTML(resp.text) # Print news headlines elements = dom.xpath("//h3/a[u[@class='StretchedBox']]") for elem in elements:     print(etree.tostring(elem, method="text", encoding="unicode"))  
 
 This is because web pages like this rely on JavaScript to populate the content. Famous web frameworks such as AngularJS or React are behind powering this category. The Python library, such as requests, does not understand JavaScript. Therefore, you will see the result differently. If the data you want to fetch from the web is one of them, you can study how the JavaScript is invoked and mimic the browser’s behavior in your program. But this is probably too tedious to make it work.
The other way is to ask a real browser to read the web page rather than using requests. This is what Selenium can do. Before we can use it, we need to install the library:
But Selenium is only a framework to control browsers. You need to have the browser installed on your computer as well as the driver to connect Selenium to the browser. If you intend to use Chrome, you need to download and install ChromeDriver too. You need to put the driver in the executable path so that Selenium can invoke it like a normal command. For example, in Linux, you just need to get the chromedriver executable from the ZIP file downloaded and put it in /usr/local/bin.
Similarly, if you’re using Firefox, you need the GeckoDriver. For more details on setting up Selenium, you should refer to its documentation.
Afterward, you can use a Python script to control the browser behavior. For example:
 
 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 
  import time from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By # Launch Chrome browser in headless mode options = webdriver.ChromeOptions() options.add_argument("headless") browser = webdriver.Chrome(options=options) # Load web page browser.get("https://www.yahoo.com") # Network transport takes time. Wait until the page is fully loaded def is_ready(browser):     return browser.execute_script(r"""         return document.readyState === 'complete'     """) WebDriverWait(browser, 30).until(is_ready) # Scroll to bottom of the page to trigger JavaScript action browser.execute_script("window.scrollTo(0, document.body.scrollHeight);") time.sleep(1) WebDriverWait(browser, 30).until(is_ready) # Search for news headlines and print elements = browser.find_elements(By.XPATH, "//h3/a[u[@class='StretchedBox']]") for elem in elements:     print(elem.text) # Close the browser once finish browser.close()  
 
 The above code works as follows. We first launch the browser in headless mode, meaning we ask Chrome to start but not display on the screen. This is important if we want to run our script remotely as there may not be any GUI support. Note that every browser is developed differently, and thus the options syntax we used is specific to Chrome. If we use Firefox, the code would be this instead:
 
  options = webdriver.FirefoxOptions() options.set_headless() browser = webdriver.Firefox(firefox_options=options)  
 
 After we launch the browser, we give it a URL to load. But since it takes time for the network to deliver the page, and the browser will take time to render it, we should wait until the browser is ready before we proceed to the next operation. We detect if the browser has finished rendering by using JavaScript. We make Selenium run a JavaScript code for us and tell us the result using the execute_script() function. We leverage Selenium’s WebDriverWait tool to run it until it succeeds or until a 30-second timeout. As the page is loaded, we scroll to the bottom of the page so the JavaScript can be triggered to load more content. Then we wait for one second unconditionally to make sure the browser triggered the JavaScript, then wait until the page is ready again. Afterward, we can extract the news headline element using XPath (or alternatively using a CSS selector). Because the browser is an external program, we are responsible for closing it in our script.
Using Selenium is different from using the requests library in several aspects. First, you never have the web content in your Python code directly. Instead, you refer to the browser’s content whenever you need it. Hence the web elements returned by the find_elements() function refer to objects inside the external browser, so we must not close the browser before we finish consuming them. Secondly, all operations should be based on browser interaction rather than network requests. Thus you need to control the browser by emulating keyboard and mouse movements. But in return, you have the full-featured browser with JavaScript support. For example, you can use JavaScript to check the size and position of an element on the page, which you will know only after the HTML elements are rendered.
There are a lot more functions provided by the Selenium framework that we can cover here. It is powerful, but since it is connected to the browser, using it is more demanding than the requests library and much slower. Usually, this is the last resort for harvesting information from the web.
Further Reading
Another famous web crawling library in Python that we didn’t cover above is Scrapy. It is like combining the requests library with BeautifulSoup into one. The web protocol is complex. Sometimes we need to manage web cookies or provide extra data to the requests using the POST method. All these can be done with the requests library with a different function or extra arguments. The following are some resources for you to go deeper:
Articles
 An overview of HTTP from MDN 
 XPath from MDN 
 XPath tutorial from W3Schools 
 CSS Selector Reference from W3Schools 
 Selenium Python binding 
API documentation
 Requests library 
 BeautifulSoup 
 lxml.etree 
 Selenium Python API 
 Scrapy 
Books
 Python Web Scraping, 2nd Edition, by Katharine Jarmul and Richard Lawson
 Web Scraping with Python, 2nd Edition, by Ryan Mitchell
 Learning Scrapy, by Dimitrios Kouzis-Loukas
 Python Testing with Selenium, by Sujay Raghavendra
 Hands-On Web Scraping with Python, by Anish Chapagain
 Summary 
In this tutorial, you saw the tools we can use to fetch content from the web.
Specifically, you learned:
How to use the requests library to send the HTTP request and extract data from its response
How to build a document object model from HTML so we can find some specific information on a web page
How to read tables on a web page quickly and easily using pandas
How to use Selenium to control a browser to tackle dynamic content on a web page
Get a Handle on Python for Machine Learning!
    Be More Confident to Code in Python
...from learning the practical Python tricks
Discover how in my new Ebook:
Python for Machine Learning 
It provides self-study tutorials with hundreds of working code to equip you with skills including:
debugging, profiling, duck typing, decorators, deployment,
and much more...
Showing You the Python Toolbox at a High Level for Your Projects
 See What's Inside
Web Crawling in Python - MachineLearningMastery.com
https://machinelearningmastery.com/web-crawling-in-python/