Articles / Linux Clustering Software

Linux Clustering Software



Software for building and using clusters
Software used on, and for using, clusters

Software for building and using clusters


High Performance Computing Software







High Availability Software





Load Balancing Software




Software used on, and for using, clusters







Installation and Configuration Software



To install boxes
System Installation Suite
It is the answer


Monitoring and Management Software





Programming and Execution Environments and Tools






Miscellaneous Cluster-Related Projects




Some Comments In Closing



The High Availability Linux Project
The Linux Clustering Information Center
The Beowulf Project
The Sourceforge Clustering Foundry




RSS Recent comments

01 Jun 2002 11:06 mattdm

Don't forget xCAT.

01 Jun 2002 12:37 jgreenseid

Re: xCAT
Don't forget xCAT.

Unfortunately, xCAT isn't registered here on Freshmeat, so I did not include it in the review, but yes, from what little I've seen of it, xCAT looks pretty cool.

02 Jun 2002 08:28 jeffcovey

Re: xCAT

> xCAT isn't registered here on Freshmeat, so I did not include it in
> the review

Just to be clear: In my message inviting you to write this article, I

"We can add or recategorize any projects that you think
need to be listed under this category."

03 Jun 2002 10:57 imipak

Thoughts from the peanut gallery
First, some of the projects are in an indeterminate state. SCE, for example, has no downloads at the moment, and the download link on their entry is broken. (Hey, I'm as guilty as anyone, when it comes to link validation & project validation. FOLK is -hardly- the standard-bearer for Quality Maintenance.) Ideally, though, a write-up would red-flag suspect links and projects in limbo.

But, remember this may be the ideal, but writers have real-lives, are human, and have finite patience with net-lag. The comments can red-flag a project or link just as easily as the master article. If I'd like to see anything in the follow-up comments, it's information from developers and users on what projects mentioned (or missed) are dead, alive, both, or neither.

The second thing I'd mention is that PVM and MPI, as far I am able to understand them, don't actually require underlying clustering software to work. Given that PVM is designed to work with fixed numbers of physical machines, mixing it into a dynamically-resizable virtual architecture might be, ummm, entertaining?

Thirdly, I'd like to mention that OpenMosix is a political fork from MOSIX, and that the two are unlikely to be compatiable. I don't want to get into the issues here - from everything I've heard, there's a LOT of ill-feeling involved, and I've learned that you really don't want to open a can of angry vampire worms.

The technical issue, though, is that heterogenius clustering solutions don't yet exist - there are quite simply no standard frameworks - and project splits can diverge amazingly rapidly. This means that although you can have clusters containing different machines (at least, for some solutions), you can't mix-and-match solutions.

This brings me to my last point. At the present time, inter-communication between clustering methods doesn't exist. This means that whatever method you choose will be a compromise solution. If you have a problem which you want to cluster, but that problem contains elements which each would be better done with a different solution, all you can do is pick the one that is least worst overall.

A trivial, and undeniably lame, example would be clustering a Mandelbrot generator:

Your virtual screen is fixed, so you'd want to use PVM to split the individual operations. PVM is optimal when you've a fixed number of components. Your maths algorithm is also fixed (3 independent operations), but a different size. You still want PVM, but now you're running a virtual machine inside a virtual machine. If you're wanting to generate multiple regions, though, each region wants to be run independently.

If you don't know how many regions you want to generate, at compile time, then you're going to have to use MPI or some other dynamically-resizable clustering methodology.

MPI and PVM are not interoperable. They cannot talk to each other. Mixing them into the same program, therefore, is usually not a good idea.

This leaves two options. (1) Use PVM for the bulk of the work, as before, and then support different regions by having "launchers" on the network, which you can feed the data to independently run those PVM sessions. You then collect the data at the end. Since, in this case, there's no communication between virtual machines, except at the start and end, you can get away with this. If there was a significant amount, you can forget it. This is optimal for any given point, as PVM doesn't need to do the housework of a dynamic solution, so the architecture is much faster for this static problem. Only the work that -needs- to be done is done, and only the traffic that -needs- to be sent is sent.

(2) Use MPI or some other dynamic clustering technology for the whole system. Treat the whole problem space as a single, dynamically-stretchable problem. Because there are likely to be more points being calculated than computers, you can have a mechanism for farming out & re-distributing the work, on the fly. It's sub-optimal for any given point, but optimal for the problem as a whole. This is because your low-level work now has to involve a lot more processing, because you're not making so many simplifying assumptions. On the other hand, your higher layers are vastly better off, as ALL the regions can use ALL the capacity of the cluster. You don't have autonomous clusters sitting idle, because they ran out of work.

All this boils down to: What assumptions can I make, and what clustering solution can exploit those assumptions the best? If you can answer that, then you've answered how you need to cluster your computers to best solve your problem.

03 Jun 2002 11:35 imipak

Re: xCAT

> Don't forget xCAT.

Oh, there were plenty "missed". COSM, for example. There were other aspects of clustering which were overlooked, too. "Fault Tolerent" systems (essentially HA with zero switch-over time) were not discussed.

However, let's be fair on the author, here. However long, complete, or researched an article, it's always going to miss something, for some reason or other. IMHO, that's one of the good things about comments. If there's something vitally important to add, we're able to add it, putting less pressure on authors to "get it perfect", so they can spend more time feeling it's worth doing at all.

Remember that those 300-600 page monster manuals that cost more than your house? Those won't contain everything, either. (In fact, those usually contain less than your average Linux HOWTO, or a Freshmeat article, in terms of actual useful knowledge.)

I'm assuming that the vast number of apps and methods missed were missed:

a) Because the author had run out of Mountain Dew and just wanted to get the damn thing finished

b) Because researching every little app, every side-avenue, every derivative, etc, would take longer than the remaining estimated lifespan of the Universe, and the author felt it more important to post something than nothing

c) Some applications aren't "obvious" clustering apps, but could be used that way. nmap, for example, could be used as an alternative to heartbeat, where the system load is moved onto one machine. (Simply scan your cluster, repeatedly, for a change in port status or machine status.) Anything which allows you to use, manipulate, or process data from, a network connection in a way that can simplify some aspect of clustering could be considered a clustering tool of sorts. In the end, you can either draw the line at some level of indirect obscurity, or you can go quietly insane, burbling to yourself in the corner.

d) Sometimes, a useful tool is going to get nudged out for other reasons. Too many others, of the same ilk, for example. Confusion over whether it meets a given criteria, as seems to have been the case with xCAT.

In my recent article on K5, on extreme tech, I listed maybe a dozen or so corporations involved in technology way outside the normal realms of sane environments. By the next day, I'd researched maybe a dozen more, and had strong pointers to maybe a few hundred more, again. Should I have waited, and published the article as a 10-tonne paperweight guide to companies you'll never ever want or need to know about?

Nah! As I've written elsewhere, the comments section can add all the "fixes" and addendums. That's what it's there for (besides actual comments!! :) Release Early, Release Often is a maxim that works well for code, and works well for weblogs, too.

03 Jun 2002 11:40 mattdm

Re: xCAT
My goodness. That was quite verbose. I wasn't chiding the author for missing something; I was just mentioning another cool clustering tool. You even end by saying that doing this is exactly what the comments are for -- what was your point exactly? To write a lot of words? To refer to your k5 article? I don't get it.

03 Jun 2002 15:50 jgreenseid

Re: xCAT

Just to be clear: In my message
inviting you to write this article, I
said, "We can add or recategorize any
projects that you think
need to be listed under this

Yes, you did, I think that I just didn't realize that you meant that this also included actually registering projects to Freshmeat itself, I thought you meant what we actually did -- added/recategorized some projects that should have been included in this category. I thought that was left to project participants. Apparently a little hiccup in the lines of communication. Though I think no great harm done. :)

03 Jun 2002 16:15 jgreenseid

Re: xCAT





25 Jun 2002 03:53 kisses

Re: xCAT

Unfortunately, xCAT isn't registered
here on Freshmeat, so I did not include
pretty cool.

16 Aug 2002 19:54 Avatar josevnz

What about SNMP monitoring tools?



Some opensource cool tools (SNMP & NMS):

Net-SNMP (open source SNMP agent):


OpenNMS (java based, scalable, good arquitecture):

19 Nov 2002 17:01 rsug



24 Mar 2012 09:24 keratintreatment2012

looks pretty cool.


Project Spotlight

Kwok Information Server

An IT management system.


Project Spotlight


A library to access Generic Tagged Arrays (GTA) files.