Take the 2-minute tour ×

I have a few workers that is listening to an RabbitMQ queue and that is doing some disk I/O intensive work - opening ~18MB files, doing some parsing and writes to some files. While processing one job a worker could take up to 200MB of memory.. and this is fine.

However, my problem is that the worker continues to be idle and still reserving this amount of memory. I have blindly tried to do some garbage collection manually with gc.collect() after the job is done but without any results.

My worker class that receives the job looks like this:

class BuildWorker(worker.Worker):

    def callback(self, ch, method, properties, body):
        fp = FileParseAndStuff()
        fp.execute_job(ch, method, properties, body)
        fp = None

Shouldn't everything that happens inside fp here be contained memory wise and be removed once I set that object to None ? I have tried to use Python's del statement as well but without any improvments.

I'm using Python 2.7 and python-pika to communicute with the RabbitMQ server, if that matters.

share|improve this question
You can try to find out which objects take this space, using objgraph. –  Bogdan Sep 29 '13 at 10:15
See this stackoverflow.com/questions/11957539/… –  kroolik Sep 29 '13 at 10:24

1 Answer 1

put a flag before fp = None, like write to a file "done" or print to console. Your worker may not be there yet when you call del or gc.collect(). If it is, check your execute_job method.

share|improve this answer
I don't get this, I'm not using any asynchronous framework for Python.. so fp.execute_job() is completed before the next line (fp = None) is executed.. so I don't see how this would help.. –  Niklas9 Sep 29 '13 at 11:32
try to pass an dummy job to worker, maybe the reference is kept by worker itself and released when start a new job. –  cox Sep 29 '13 at 11:44
no it's not unfortunately, the memory consumption is growing with the same amount for each job :( –  Niklas9 Sep 29 '13 at 13:46

Your Answer


