Unfortunately, not everybody got the memo. Labels: virtual machines

Sometimes It All Comes Together

Put it all together, and you have this: Compiling Workflows with Weaver Labels: allpairs, makeflow, weaver

From Database to Filesystem and Back Again EXPORT irises WHERE camera='Likon' AS color/subjectid.type

VIEW irises WHERE camera='Likon' AS color/subjectid.type Labels: bxgrid, parrot, roars

Summer REU: Toward Elastic Scientific Applications Posted by Douglas Thain at 12:04 PM No comments: Links to this post The Forty Tribes of Linux

As I have noted in this column before, a perennial challenge of distributed computing in the real world is dealing with the multiplicity of operating systems and related environments. If you are dealing with an uncontrolled environment like a large university or an 'at home' computing environment, there is no telling what you are going to get. If you have a piece of software that depends exactly on the presence of Linux 22.214.171.124.9.2, it just isn't going to work.

You might think that this could be avoided by having a professionally managed environment. At Notre Dame, we have a site license for Red Hat Linux, and our staff are pretty rigorous in keeping everything up to date and on track. But even then, you can't assume everything is identical: there is no way to upgrade everyone simultaneously, and every machine operates on a different schedule (and discipline) for picking up automatic updates. For example, we are currently in the tail of of a general campus migration from Red Hat 4 to Red Hat 5.

Here is some hard evidence. We recently started using the neat 'cron' feature in Condor to make a daily observation of the operating system version, kernel version, and C library version of each machine. With a few variations on condor_status, we can see the upgrade status of the whole system:

The major release numbers (below) aren't too bad. About 3/4 of our cores are running the latest Red Hat, but another 73 machines are behind by a version or two. And, oops, looks like someone plugged in their own personal CentOS machine. Not too hard to deal with, if you are careful to put 'redhat_version' in your requirements:

If we go a little deeper, the picture gets murkier. Below are the distribution of Linux kernel versions. Interesting to note that a few are hand-modified for some unusual hardware, and only two are Xen virtualized. Hope that you don't have any code sensitive to the kernel version.

For completeness, here is the distribution of glibc versions, which has much the same story:

In the good old days, you could just indicate that a program required OpSys=="LINUX" and more or less expect it to run. That certainly isn't possible now. Perhaps we are misleading users by talking about this thing called Linux, which doesn't really exist in any consistent form. Instead, we should be telling our users that a new operating system gets invented every week, and is usually named after a team on Survivor.

The good folks at Sun tried to solve this problem almost 20 years ago with Java. The idea was that they would create a stable platform that could be implemented on any machine. Then, you could write programs that would be universally portable. The problem was, well...

Many people think the grand solution to this problem is virtual machines. Perhaps, but more on that next time.
Posted by Douglas Thain at 12:45 PM No comments: Links to this post Labels: condor, java, linux, virtual machines

Summer REU at Notre Dame

We invite outstanding undergraduates to apply for summer research
positions in scientific and cloud computing at the University of Notre Dame.
Students will build and operate systems that harness hundreds of
machines at once to attack large problems in science and engineering.

Research topics include:
More information is available here:

Applications received by March 1st will be given first consideration.
Posted by Douglas Thain at 1:37 PM No comments: Links to this post Green Cloud Online

The Green Cloud is now online!

The Green Cloud is the invention of Dr. Paul Brenner at the ND Center for Research Computing. It is a containerized data center located at the South Bend city greenhouse, stocked with used servers kindly donated by Ebay, Inc. The first batch of machines was installed in December, and will eventually reach about 400 cores once everything is turned on. What makes the data center unique is that is has no air conditioning. Instead, the data center takes in ambient air, and then exhausts it into the greenhouse. This benefits Notre Dame, since we no longer pay the cost of cooling, but it also benefits the greenhouse, which has significant heating costs during the winter months. (We used to call this idea grid heating.)

Of course, this means the capacity of the system may change with the weather. During the winter, the system can run at full blast and deliver as much heat as possible to the greenhouse. During the spring and fall, the heat may not be needed, and can be vented outdoors. During the hottest part of the summer, we may need to shut some machines down to get the temperature under control. However, recent studies by big data center operators suggest that machine temperature could be safely increased to 80 or 90 degrees, so there may be a fair amount of headroom available. We will see.

For a normal data center that runs web servers and databases, shutting down machines is not really an option. However, the Green Cloud provides fungible computing power for large computations in science and engineering at Notre Dame. If structured correctly, these workloads can adapt to 10 or 100 or 1000 cores. So, turning machines on and off will affect performance, but not correctness.

A good example of a flexible workload is genome assembly. Two of our students, Christopher Moretti and Michael Olson presented initial results on a Scalable Genome Assembler at the MTAGS Workshop held at Supercomputing 2009. Their assembler uses our Work Queue framework to manage a variable workforce, pushing out sequence fragments to whatever machines are available. The system has scaled up to about 1000 cores, spread across the Notre Dame campus, the Green Cloud, Purdue University, and the University of Wisconsin.

We are currently working on a journal paper and an open source release of the assembler, so stay tuned for details.
[null, null, null, null, null, {"document_url": "http://dthain.blogspot.com/2010/", "unformatted_src": "http://2.bp.blogspot.com/_0DkxYiRVGyA/TMg4SIAfgOI/AAAAAAAAAHc/4b8x7Mck_14/s320/iris.jpeg", "src": "http://2.bp.blogspot.com/_0DkxYiRVGyA/TMg4SIAfgOI/AAAAAAAAAHc/4b8x7Mck_14/s320/iris.jpeg", "formatted_filename": "iris"}, null, null, null, {"document_url": "http://dthain.blogspot.com/2010/", "unformatted_src": "http://3.bp.blogspot.com/_0DkxYiRVGyA/TL3rG2o8QFI/AAAAAAAAAHE/ooiIJq3Cpbw/s320/Noname.png", "src": "http://3.bp.blogspot.com/_0DkxYiRVGyA/TL3rG2o8QFI/AAAAAAAAAHE/ooiIJq3Cpbw/s320/Noname.png", "formatted_filename": "Noname"}, null, null, null, null, null, null, {"document_url": "http://dthain.blogspot.com/2010/", "unformatted_src": "http://www.nd.edu/~dthain/images/thain3-small.jpg", "src": "http://www.nd.edu/~dthain/images/thain3-small.jpg", "formatted_filename": "thain small", "alt_text": "Prof. Douglas Thain", "rendered_width": 125, "rendered_height": 160}, null, null, null, null, null, null, null, null, null, null, null]