title: How do I build a Web Crawler using Python 3?
body: <p>I have been watching 3 videos of thenewboston on how to make a webcrawler. It seems they are outdated and the links are not there. I would be thankfull if someone could fullfill the first part of the tutorial. This is as far as I have come. I have tried different websites but to no avail.</p>
  
  <pre><code>import requests
  from bs4 import BeautifulSoup
  
  def my_spider(max_pages):
      page = 1
      while page &lt;= max_pages:
          url = """Here I tried different websites""" + str(page)
          source_code = requests.get(url)
          plain_text = source_code.text
          soup = BeautifulSoup(plain_text)
          for link in soup.findALL("a", {"class" : "item-name"}):
              href = """example site like following: https://example.com""" + link.get("href")
              print(href)
          page += 1
  
  print(my_spider(2))
  </code></pre>
  
  <p>Using this code I would for example want to crawl to the site for title links or something similiar. </p>
  
  <p>FYI here is the video. <a href="https://www.youtube.com/watch?v=sVNJOiTBi_8&amp;list=PL6gx4Cwl9DGAcbMi1sH6oAMk4JHw91mC_&amp;index=26" rel="nofollow">https://www.youtube.com/watch?v=sVNJOiTBi_8&amp;list=PL6gx4Cwl9DGAcbMi1sH6oAMk4JHw91mC_&amp;index=26</a></p>
  
  <p>It is tutorial 25-27</p>
  
  <p>Thanks in advance!</p>