title: requests.get() not crawling entire common crawl records for a given warc path
body: <p>i have implemented <a href="https://dmorgan.info/posts/common-crawl-python/" rel="nofollow noreferrer">https://dmorgan.info/posts/common-crawl-python/</a> as described in this link. However, I want to crawl entire data rather than partial data unlike as described in this post. So, in this code chunk,</p>
  
  <pre><code>def get_partial_warc_file(url, num_bytes=1024 * 10):
  with closing(requests.get(url, stream=True)) as r:
      buf = StringIO(r.raw.read(num_bytes))
  return warc.WARCFile(fileobj=buf, compress=True)
  </code></pre>
  
  <p>I have made the following change:</p>
  
  <pre><code>def get_partial_warc_file(url):
  with closing(requests.get(url, stream=True)) as r:
      buf = StringIO(r.raw.data)
  return warc.WARCFile(fileobj=buf, compress=True)
  </code></pre>
  
  <p>This code chunk increases the number of records for a given warc path but it does not crawl entire number of records. I can't find a possible reason for the same. Any help would be appreciated.</p>