title: mongo: update $push failed with "Resulting document after update is larger than 16777216"
body: <p>I want to extend an large array using the update(.. $push ..) operation.</p>
  
  <p>Here are the details:</p>
  
  <p>I have a large collection 'A' with many fields. Amongst the fields, I want to extract the values of the 'F' field, and transfer them into one large array stored inside one single field of a document in collection 'B'.</p>
  
  <p>I split the process into steps (<em>to limit the memory used</em>)</p>
  
  <p>Here is the python program:</p>
  
  <pre><code>...
  steps = 1000  # number of steps
  step = 10000  # each step will handle this number of documents
  start = 0
  
  for j in range(steps):
      print('step:', j, 'start:', start)
  
      project = {'$project': {'_id':0, 'F':1} }
      skip = {'$skip': start}
      limit = {'$limit': step}
      cursor = A.aggregate( [ skip, limit, project ], allowDiskUse=True )
  
      a = []
      for i, o in enumerate(cursor):
          value = o['F']
          a.append(value)
  
      print('len:', len(a))
      B.update( {'_id': 1}, { '$push': {'v' : { '$each': a } } } )
  
      start += step
  </code></pre>
  
  <p>Here is the oupput of this program:</p>
  
  <pre><code>step: 0 start: 0
  step: 1 start: 100000
  step: 2 start: 200000
  step: 3 start: 300000
  step: 4 start: 400000
  step: 5 start: 500000
  step: 6 start: 600000
  step: 7 start: 700000
  step: 8 start: 800000
  step: 9 start: 900000
  step: 10 start: 1000000
  Traceback (most recent call last):
    File "u_psfFlux.py", line 109, in &lt;module&gt;
      lsst[k].update( {'_id': 1}, { '$push': {'v' : { '$each': a } } } )
    File "/home/ubuntu/.local/lib/python3.5/site-packages/pymongo/collection.py", line 2503, in update
      collation=collation)
    File "/home/ubuntu/.local/lib/python3.5/site-packages/pymongo/collection.py", line 754, in _update
      _check_write_command_response([(0, result)])
    File "/home/ubuntu/.local/lib/python3.5/site-packages/pymongo/helpers.py", line 315, in _check_write_command_response
      raise WriteError(error.get("errmsg"), error.get("code"), error)
  pymongo.errors.WriteError: Resulting document after update is larger than 16777216
  </code></pre>
  
  <p>Apparently the $push operation has to fetch the complete array !!! (<em>my expectation was that this operation would always need the same amount of memory since we always append the same amount of values to the array</em>)</p>
  
  <p>In short, I don't understand why the update/$push operation fails with error...</p>
  
  <p>Or... is there a way to avoid this unneeded buffering ?</p>
  
  <p>Thanks for your suggestion</p>
  
  <p>Christian</p>