title: Web scraping VIVINO using Python
body: <p>I have made a script for scraping all red wines on Vivinos page for price, ratings, country, grapes etc. However when I use the below code I get everything I need until I reach around 2000 wines. After this, the loop just repeats it self. As I can see there are many more red wines avaliable on Vivino, I would kindly ask for any help regarding getting as many wines as possible with the stated variables.</p>
  <pre><code>enter code herimport requests
  import pandas as pd
  import urllib, json
  import urllib.request, json
  import time
  
  def scrape():
      dataframe = pd.DataFrame()
      results = []
  print(&quot;Retrieved Wines:&quot;)
  print(&quot;---&quot;, 0)
  
  for i in range(500):
      r = requests.get(&quot;https://www.vivino.com/api/explore/explore&quot;,
      params = {
      &quot;min_rating&quot;:&quot;1&quot;,
      &quot;order&quot;:&quot;asc&quot;,
      &quot;page&quot;: i+1,
      &quot;price_range_max&quot;:&quot;500&quot;,
      &quot;price_range_min&quot;:&quot;0&quot;,
      &quot;wine_type_ids[]&quot;:&quot;1&quot;
  },
      headers = {&quot;User-Agent&quot;: &quot;Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0&quot;})
  
  
      for j in range(24):
          try:
              result_list = []
              result_list.append(r.json()[&quot;explore_vintage&quot;][&quot;matches&quot;][j][&quot;vintage&quot;][&quot;name&quot;]) #Name
              result_list.append(r.json()[&quot;explore_vintage&quot;][&quot;matches&quot;][j][&quot;vintage&quot;][&quot;year&quot;]) #Year
              result_list.append(r.json()[&quot;explore_vintage&quot;][&quot;matches&quot;][j][&quot;vintage&quot;][&quot;wine&quot;]['region'][&quot;country&quot;][&quot;name&quot;]) #country
              result_list.append(r.json()[&quot;explore_vintage&quot;][&quot;matches&quot;][j][&quot;vintage&quot;][&quot;wine&quot;][&quot;winery&quot;][&quot;name&quot;]) #wine_yard
              result_list.append(r.json()[&quot;explore_vintage&quot;][&quot;matches&quot;][j][&quot;vintage&quot;][&quot;wine&quot;][&quot;region&quot;][&quot;name&quot;]) #region
              result_list.append(r.json()[&quot;explore_vintage&quot;][&quot;matches&quot;][j][&quot;vintage&quot;][&quot;statistics&quot;][&quot;ratings_average&quot;]) #Rating
              result_list.append(r.json()[&quot;explore_vintage&quot;][&quot;matches&quot;][j][&quot;vintage&quot;][&quot;statistics&quot;][&quot;ratings_count&quot;]) #reviews
              result_list.append(r.json()[&quot;explore_vintage&quot;][&quot;matches&quot;][j][&quot;price&quot;][&quot;amount&quot;]) #price
              result_list.append(r.json()[&quot;explore_vintage&quot;][&quot;matches&quot;][j][&quot;vintage&quot;][&quot;wine&quot;][&quot;vintage_type&quot;]) #vintage_type
  
              number_of_grapes_in_wine = (int(len(r.json()[&quot;explore_vintage&quot;][&quot;matches&quot;][j][&quot;vintage&quot;][&quot;wine&quot;][&quot;style&quot;][&quot;grapes&quot;])))
              for grape in range(number_of_grapes_in_wine):
                  result_list.append(r.json()[&quot;explore_vintage&quot;][&quot;matches&quot;][j][&quot;vintage&quot;][&quot;wine&quot;][&quot;style&quot;][&quot;grapes&quot;][grape][&quot;name&quot;]) #Grape(s)
          except:
              pass
  
          results.append(result_list)
  
      print(&quot;-- &quot;,len(results))
  
  df_results = pd.DataFrame(results)
  df_results.to_excel(&quot;Red_wine_Data3.xlsx&quot;)
  </code></pre>
  <p>scrape()
  e</p>