title: BeautifulSoup - Pagination error on the last page
body: <p>The battle to finish my first <strong>scraping</strong> script continues. I think that I'm almost finishing but I hit a new roadblock.</p>
  <p>So, the problem is that when I reach the last pagination page I'm getting this error:</p>
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code>Traceback (most recent call last):
    File "C:/Users/Andre/Desktop/scripts python/scrape_learn/ttc_quase.py", line 50, in &lt;module&gt;
      url_tag = soup.find('li', {"id": "next-page-link"}).find('a')
  AttributeError: 'NoneType' object has no attribute 'find'</code></pre>
  </div>
  </div>
  </p>
  <p>I think that the error is related with the way I'm finding url_tag, but I'm not seeign any other way to grab the &quot;next page&quot;. I tried to use the Try/Except method but when apply it I just get the listings on the first page.</p>
  <p>So I'm not sure what should be my next step. If someone could help I will appreciate.</p>
  <p>My full code:</p>
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code>from bs4 import BeautifulSoup
  import requests
  import pandas as pd
  
  url = "https://timetochoose.co.ao/?ct_keyword&amp;ct_ct_status&amp;ct_property_type&amp;ct_beds&amp;search-listings=true&amp;ct_country=portugal&amp;ct_state&amp;ct_city&amp;ct_price_to&amp;ct_mls&amp;lat&amp;lng"
  
  headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0'}
  
  anuncios_ttc = {}
  anuncios_nr = 0
  
  while True:
      response = requests.get(url, headers=headers)
  
      print(response)
  
      data = response.text
  
      print(data)
  
  
      soup = BeautifulSoup(requests.get(url, headers=headers).content, 'html.parser')
  
      anuncios = soup.find_all("div", {"class": "grid-listing-info"})
  
      for anuncios in anuncios:
          titles = anuncios.find("a",{"class": "listing-link"}).text
          location = anuncios.find("p",{"class": "location muted marB0"}).text
          link = anuncios.find("a",{"class": "listing-link"}).get("href")
          anuncios_response = requests.get(link, headers=headers)
          anuncios_data = anuncios_response.text
          anuncios_soup = BeautifulSoup(anuncios_data, 'html.parser')
          conteudo = anuncios_soup.find("div", {"id":"listing-content"}).text
          preco = anuncios_soup.find("span",{"class": "listing-price"})
          preco_imo = preco.text if preco else "N/A"
          quartos = anuncios_soup.find("li", {"class": "row beds"})
          nr_quartos = quartos.text if quartos else "N/A"
          wcs = anuncios_soup.find("li", {"class": "row baths"})
          nr_wcs = wcs.text if wcs else "N/A"
          tipo = anuncios_soup.find("li", {"class": "row property-type"})
          tipo_imo = tipo.text if tipo else "N/A"
          bairro = anuncios_soup.find("li", {"class": "row community"})
          bairro1 = bairro.text if bairro else "N/A"
          ref = anuncios_soup.find("li", {"class": "row propid"}).text
  
          anuncios_nr+=1
          anuncios_ttc[anuncios_nr] = [titles, location, bairro1, preco_imo, tipo_imo, nr_quartos, nr_wcs, conteudo, ref, link]
          print("Título", titles, "\nLocalização", location, "\nPreço", preco_imo, "\nLink", link, "\nReferencia", ref, "\nTipo", tipo_imo, "\nQuartos", nr_quartos, "\nWC", nr_wcs, "\nBairro", bairro1, "\nConteudo", conteudo)
  
      url_tag = soup.find('li', {"id": "next-page-link"}).find('a')
      if url_tag.get('href'):
          url = url_tag.get('href')
          print(url)
      else:
          break
  
  print("Nr Total de Anuncios: ", anuncios_nr)
  
  anuncios_ttc_df = pd.DataFrame.from_dict(anuncios_ttc, orient = 'index', columns =['Titulo', 'Localização', 'Bairro', 'Preço', 'Tipo', 'Quartos', 'WCs', 'Descrição', 'Referência', 'Ligação'])
  
  anuncios_ttc_df.head()
  
  anuncios_ttc_df.to_csv('ttc_python.csv')</code></pre>
  </div>
  </div>
  </p>