Take the 2-minute tour ×

I have set up a pseudo distributed mode cluster setup. The FIFO scheduler got stuck somehow in between and therefore a lot of jobs got piled up which I had scheduler through cron. Now, when I restarted YARN resourcemanager it gets stuck after a while and the jobs keep piling up.

Is there a way I can clear the whole queue. Or, is it that my complete understanding of hadoop scheduling is somewhere flawed. Please help.

share|improve this question
add comment

1 Answer

If you're trying to kill all the jobs in your queue, you can use this shell script:

$HADOOP_HOME/bin/hadoop job -list | awk ' { system("$HADOOP_HOME/bin/hadoop job -kill " $1) } '
share|improve this answer
Also, if you're having trouble with jobs piling up, I'd suggest writing a quick script that you can call through a cron job that checks the size of the job queue (via hadoop job -list, maybe piped into wc or something) and stops new jobs from being created if too many jobs have built up. If you really want to get fancy, you could even write the incoming jobs to a temp file and have your cron job retry them once the queue clears. –  HypnoticSheep Aug 20 '12 at 23:00
add comment

Your Answer


