Take the 2-minute tour ×

This question answers part of my question but not completely.

How do I run a script that manages this, is it from my local filesystem? Where exactly do things like MrJob or Dumbo come into picture? Are there any more alternative?

I am trying to run K-Means where each iterations (a MapReduce job) output will be the input to the next iteration with Hadoop Streaming and Python.

I do not have much experience and any information should help me make this work.Thanks!

share|improve this question

4 Answers 4

If you are not very tightly coupled with Python then you have a very good option. There is one project from Cloudera called "Crunch" that allows us to create pipelines of MR jobs easily. it's a java library that provides a framework for writing, testing, and running MapReduce pipelines, and is based on Google's FlumeJava library.

share|improve this answer

There is another non-python option. Graphlab is an open source project to produce free implementations of scalable machine learning algorithms on multicore machine and clusters. There is an implemented fast scalable version of the Kmeans++ algorithm included in the package. See Graphlab for details.

enter image description here

Clustering API of graphlab can be found here .

share|improve this answer

Seems like a good applications for Spark it has also streaming option but I'm afraid it only works with Scala, but they have Python API, definitively worth a try, it is not that difficult to use ( at least the tutorials) and it can scale at large.

share|improve this answer

It should be possible to use GraphLab Create (in Python) running on Hadoop to do what you describe. The clustering toolkit can help implement the K-Means part. You can coordinate/script it from your local machine and use the graphlab.deploy API to run the job on Hadoop.

share|improve this answer

Your Answer


