title: Apparent limit to re.findall in Python
body: <p>I have a file containing data (defining some mathematical objects) in a format which is inconvenient and I would like to convert it to JSON using Python 3. Since the file is huge (it contains 921,497 objects, each needing several lines to be defined), I used the module <code>mmap</code> to load content that I analyse using <code>re.findall</code>. My problem is that I get only the first 99999 entries, and I am wondering where does the limit come from? I don't get any error and I could not find an explanation for this.</p>
  
  <p>I have made some small tests to find limits, but the number I obtained are much bigger than 99999:</p>
  
  <pre><code>&gt;&gt;&gt; import re
  &gt;&gt;&gt; len(re.search(r"a{1,9999999}", "a"*(2*10**6)).group())
  2000000
  &gt;&gt;&gt; len(re.findall(r"a", "a"*(2*10**6)))
  2000000
  &gt;&gt;&gt; import sre_constants
  &gt;&gt;&gt; sre_constants.MAXGROUPS
  2147483647
  </code></pre>
  
  <p>Does anyone have an idea of how to surpass this limit, or of a better approach to collect the data?</p>
  
  <hr>
  
  <p>My code (up to saving the files) is the following:</p>
  
  <pre><code>"""
  Transform dataset for CICY 4-fold to JSON
  
  arxiv: 1303.1832
  http://www-thphys.physics.ox.ac.uk/projects/CalabiYau/Cicy4folds/index.html
  """
  
  import os, mmap, json, tarfile, tempfile
  import regex as re
  
  CICY4_REGEX = b"""\
  MATRIX NUMBER :\s+(\d+)
  SIZE :\s+(\d+)\s+(\d+)
  EULERNUMBER :\s+(\d+)
  ([\d\s\|]+)
  """
  cicy4_pattern = re.compile(CICY4_REGEX, re.MULTILINE)
  
  json_data = {}
  last_ind = 0
  
  with open('cicy4folds.txt', 'r') as f,\
          mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as m:
  
      for data in cicy4_pattern.findall(m):
  
          dic = {"vec": [], "matrix": []}
  
          ind = int(data[0])
          if ind != last_ind + 1:
              print(ind)
          last_ind = ind
  
          #dic["id"] = int(data[0])
          dic["size"] = (int(data[1]), int(data[2]))
          dic["euler"] = int(data[3])
  
          for line in data[4].decode("utf-8").split("\n")[:-1]:
              ind, row = line.split(" | ")
              dic["vec"].append(int(ind))
              dic["matrix"].append(list(map(int, row.split())))
  
          json_data[int(data[0])] = dic
  
  print("Size: ", len(json_data))
  </code></pre>
  
  <p>The file <code>cicy4folds.txt</code> can be found <a href="http://www-thphys.physics.ox.ac.uk/projects/CalabiYau/Cicy4folds/index.html" rel="nofollow noreferrer">here</a>. Here are two examples of data:</p>
  
  <pre><code>MATRIX NUMBER : 26538
  SIZE :   6 10
  EULERNUMBER :  480
  1 | 0 0 0 0 0 0 0 0 1 1
  1 | 0 0 0 0 0 0 1 1 0 0
  1 | 0 0 0 0 1 1 0 0 0 0
  1 | 0 0 1 1 0 0 0 0 0 0
  1 | 1 1 0 0 0 0 0 0 0 0
  9 | 1 1 1 1 1 1 1 1 1 1
  
  MATRIX NUMBER : 26539
  SIZE :   6  4
  EULERNUMBER :  720
  1 | 0 0 0 2
  1 | 0 0 0 2
  1 | 0 0 1 1
  1 | 0 0 1 1
  2 | 1 1 0 1
  2 | 1 1 1 0
  </code></pre>