Take the 2-minute tour ×

I am building a simple application to download a set of XML files and parse them into a database using the async module (https://npmjs.org/package/node-async) for flow control. The overall flow is as follows:

  1. Download list of datasets from API (single Request call)
  2. Download metadata for each dataset to get link to XML file (async.each)
  3. Download XML for each dataset (async.parallel)
  4. Parse XML for each dataset into JSON objects (async.parallel)
  5. Save each JSON object to a database (async.each)

In effect, for each dataset there is a parent process (2) which sets of a series of asynchronous child processes (3, 4, 5). The challenge that I am facing is that, because so many parent processes fire before all of the children of a particular process are complete, child processes seem to be getting queued up in the event loop, and it takes a long time for all of the child processes for a particular parent process to resolve and allow garbage collection to clean everything up. The result of this is that even though the program doesn't appear to have any memory leaks, memory usage is still too high, ultimately crashing the program.

One solution which worked was to make some of the child processes synchronous so that they can be grouped together in the event loop. However, I have also seen an alternative solution discussed here: https://groups.google.com/forum/#!topic/nodejs/Xp4htMTfvYY, which pushes parent processes into a queue and only allows a certain number to be running at once. My question then is does anyone know of a more robust module for handling this type of queueing, or any other viable alternative for handling this kind of flow control. I have been searching but so far no luck.


share|improve this question
add comment

1 Answer

up vote 1 down vote accepted

I decided to post this as an answer:

Don't launch all of the processes at once. Let the callback of one request launch the next one. The overall work is still asynchronous, but each request gets run in series. You can then pool up a certain number of the connections to be running simultaneously to maximize I/O throughput. Look at async.eachLimit and replace each of your async.each examples with it.

Your async.parallel calls may be causing issues as well.

share|improve this answer
async.eachLimit looks like exactly what I need. Thanks very much. –  Owen Aug 9 '13 at 17:58
add comment

Your Answer


