title: How to scrape many HTML documents quickly using Python?
body: <p>I have a list of about 100 HTML webpages (all have different structures, such as divs, anchors, classes, etc.) and I am trying to scrape the title of each page (where the title is under a certain div and class). To do this, I was using get requests and Beautifulsoup, however, this takes way to long (10 minutes every time I want to do it)!</p>
  
  <p>I used a timer to see what is taking the most time: it's the get requests. Apparently Python (3.7) executes the code one after another, and since each get request takes about 5-6 seconds, it's taking approximately 500-600 seconds to complete the ~100 requests. </p>
  
  <p>I've searched for ways to make these requests work faster and came across many different solutions. However, a common theme seemed to be that making my requests asynchronous (so all requests start at the same time) will solve the problem (by making it faster).</p>
  
  <p>There were many possible solutions for doing this that I read online including: multithreading, using grequest, using Scrapy, parsing lxml, etc. However, I'm new to programming and am not skilled enough to learn and experiment with each way (in fact, I tried following the answers to similar questions on SO, but wasn't successful), so I am unsure what is the best route for me to take. </p>
  
  <p>I don't need anything fancy; all I want to do is extract the titles from the HTML documents as text and then print them out. I don't need to download any CSS files, images, media, etc. Also, I'm hoping to keep the code as simple/bare as possible. How can I do this as fast as possible in Python? I'd appreciate it if someone could suggest the best path to take (i.e. using Scrapy), and give a short explanation of what I must do using that tool to get the results I'm hoping for. You don't have to write out the whole code for me. Thanks!</p>