title: Is there a way to obtain all rows/data from paginated json file into pandas dataframe
body: <p>Edited:</p>
  <p>Here are my steps:</p>
  <pre><code>url = &quot;https://ted.europa.eu/api/v2.0/notices/search?&amp;q=TD%3D%5B3%5D&amp;reverseOrder=true&amp;scope=3&amp;sortField=PD&quot;
  
  # get data from url 
  response = requests.get(url)
  
  # return the json data, and read the output dict keys
  data = response.json()
  data
  </code></pre>
  <p>I have obtained a json file from an api as such:</p>
  <blockquote>
  <pre><code>{'took': 205, 
   total': 1703997,
  'results': [{'AA': '1',
  'AC': '2',
  'BI': [],
  'CY': 'MK',
  'DI': '1046/2018',
  'TY': '1'},
  {'AA': '6',
  'AC': '1',
  'BI': [],
  'CY': 'RS',
  'DI': 'CODE_OTHERS',
  'TY': '1'},
  {'AA': '5',
  'AC': '1',
  'BI': [],
  'CY': 'BE',
  'DI': '1046/2018',
  'TY': '1'},
  ...
  </code></pre>
  </blockquote>
  <pre><code>#read the output dict keys
  data.keys()
  </code></pre>
  <p>When I convert it to a pd df</p>
  <pre><code>df = pd.DataFrame(data[&quot;results&quot;])
  </code></pre>
  <blockquote>
  <p>dict_keys(['took', 'total', 'results'])</p>
  </blockquote>
  <pre><code># create dataframe from key of interest
  df = pd.DataFrame(data[&quot;results&quot;])
  df.head()
  </code></pre>
  <p>This returned the dataframe as expected...</p>
  <pre><code># count number of rows
  len(df.index)
  </code></pre>
  <blockquote>
  <p>1000</p>
  </blockquote>
  <p>However, I expected a total of 1703997.</p>
  <p>I'm still pondering how to fix this</p>
  <p>Any idea on how I can do that?</p>