Take the 2-minute tour ×

I am trying to improve efficiency of my current query from appengine datastore. Currently, I am using a synchronous method:

class Hospital(ndb.Model):
      name = ndb.StringProperty()
      buildings= ndb.KeyProperty(kind=Building,repeated=True)
class Building(ndb.Model):
      name = ndb.StringProperty()
      rooms= ndb.KeyProperty(kind=Room,repeated=True)
class Room(ndb.Model):
      name = ndb.StringProperty()
      beds = ndb.KeyProperty(kind=Bed,repeated=True)
class Bed(ndb.Model):
      name = ndb.StringProperty()

Currently I go through stupidly:

currhosp = ndb.Key(urlsafe=valid_hosp_key).get()
nbuilds = ndb.get_multi(currhosp.buildings)
for b in nbuilds:
   rms = ndb.get_multi(b.rooms)
   for r in rms:
      bds = ndb.get_multi(r.beds)
      for b in bds:
          do something with b object

I would like to transform this into a much faster query using get_multi_async

My difficulty is in how I can do this? Any ideas?

Best Jon

share|improve this question
add comment

3 Answers

up vote 4 down vote accepted

using the given structures above, it is possible, and was confirmed that you can solve this with a set of tasklets. It is a SIGNIFICANT speed up over the iterative method.

def get_bed_info(bed_key):
    bed_info = {}
    bed = yield bed_key.get_async()
    format and store bed information into bed_info
    raise ndb.Return(bed_info)

def get_room_info(room_key):
    room_info = {}
    room = yield room_key.get_async()
    beds = yield map(get_bed_info,room.beds)
    store room info in room_info
    room_info["beds"] = beds
    raise ndb.Return(room_info)

def get_building_info(build_key):
    build_info = {}
    building = yield build_key.get_async()
    rooms = yield map(get_room_info,building.rooms)
    store building info in build_info
    build_info["rooms"] = rooms
    raise ndb.Return(build_info)

def get_hospital_buildings(hospital_object):
    buildings = yield map(get_building_info,hospital_object.buildings)
    raise ndb.Return(buildings)

Now comes the main call from the hospital function where you have the hospital object (hosp).

hosp_info = {}
buildings = get_hospital_buildings(hospital_obj)
store hospital info in hosp_info
hosp_info["buildings"] = buildings
return hosp_info

There you go! It is incredibly efficient and lets the schedule complete all the information in the fastest possible manner within the GAE backbone.

share|improve this answer
If someone sees how to make it even more efficient, let me know!!! –  Jon Jan 14 '13 at 6:19
You can (and should!) accept your own answer. It's good SO karma. –  Guido van Rossum Jan 14 '13 at 17:46
I fixed some of the code to make sure it works completely. –  Jon Mar 9 '13 at 21:26
add comment

You can do something with query.map(). See https://developers.google.com/appengine/docs/python/ndb/async#tasklets and https://developers.google.com/appengine/docs/python/ndb/queryclass#Query_map

share|improve this answer
This allowed me to explore the possibilities including tasklets and thus I was able to construct the answer that I posted. –  Jon Jan 14 '13 at 6:18
add comment

Its impossible. Your 2nd query (ndb.get_multi(b.rooms)) depends on the result of your first query. So pulling it async dosnt work, as at this point the (first) result of the first query has to be avaiable anyway. NDB does something like that in the background (it allready buffers the next items of ndb.get_multi(currhosp.buildings) while you process the first result). However, you could use denormalization, i.e. keeping a big table with one entry per Building-Room-Bed pair, and pull your results from that table. If you have more reads than writes to this table, this will get you a massive speed improvement (1 DB read, instead of 3).

share|improve this answer
It is possible - see my comment above –  Jon Jan 14 '13 at 6:18
Seems i was wrong there, sorry for that. After figuring out whats happening in the tasklets, you can actually speed this up. But this affects only the overall-time. The time until the first result is available and the amounts of DB-Reads (=costs) still stay high. So depending on your scenario, denormalization still might be the better solution. –  T. Steinrücken Jan 14 '13 at 9:33
Yeah - I agree denormalization would work - that's my next step! Any recommendations on what I should read to help create a denormalization scheme? –  Jon Jan 14 '13 at 18:04
add comment

Your Answer


