title: GAE: Aggregate work results from tasks? (for a GAE query performance issue)
body: <p>Which of those options is best to span out work on GAE (to be completed within a reuest timeframe)?</p>
  
  <ul>
  <li>Use of tasks, store the results in memcache, periodically query memcache in the request and hope the tasks complete in time</li>
  <li>Use of urlfetch to get results of tasks, error handling and security will be a pain though.</li>
  <li>Use of backend instances? (seems insane)</li>
  <li>Or a JAVA instance (seems totally insane)</li>
  </ul>
  
  <p>Background:
  It´s ridiculous to even have to do this. I need to deliver 10k datastore items as a JSON. Apparently the issue is that Python takes a lot of time to process the datastore results (Java seems much faster). This is well covered:
  <a href="https://stackoverflow.com/questions/25796142/appengine-query-fetch-async-not-very-asynchronous">25796142</a>, <a href="https://stackoverflow.com/questions/11509368/best-practice-to-query-large-number-of-ndb-entities-from-datastore">11509368</a> and
   <a href="https://stackoverflow.com/questions/21941954/improve-throughput-of-ndb-query-over-large-data">21941954</a></p>
  
  <p>Approach:
  As there is nothing to optimize on the Software side (can´t re-write GAE), the approach would be to span work out over multiple instances and to aggregate the results.</p>
  
  <p>Querying keys only and getting query cursors for chunks of 2k items performs reasonably well and there tasks could be spun off to get the results in 2k chunks. The question is about how to best aggregate the results.</p>