
Openstack: Open source cloud computing software (openstack.org)
179 points by andyangelos on July 19, 2010 | hide | past | web | favorite | 52 comments

What a great move by rackspace.

That's very impressive, they're essentially risking enabling their competition by doing this and still they've done it.

This actually will allow the development of 'micro clouds', a thing I wrote about a while back because I don't like the centralization issues of clouds for a number of reasons and use cases.


Getting rid of vendor lock-in is also fantastic.

It was touched on in the article, but allowing the actual plumbing of a cloud to become commodity, it allows them to focus on their core business: managed hosting.

Along with other benefits such as a developer community around their infrastructure and ease of friction around moving cloud hosting providers (i.e. Vendor lock-in) makes this a clear win and an excellent move by Rackspace.

This could very well be the biggest announcement of something being open-sourced made this year. I wonder if Amazon will make a comparable countermove or if they will join. (Or ignore it altogether, unlikely).

They won't have to make theirs open, or join OpenStack. All they would have to do is write a set of binaries that translate commands from one to another, and translate return values. Presumably some hackers will sort this out and open source it.

boto, libcloud, etc, etc mostly support talking to multiple clouds. The API is far from the hardest part of migration - it's actually pretty simple in most cases, unless that vendor picks something like SOAP.

You should also check out Ganeti: http://code.google.com/p/ganeti

Its basically a tool for managing virtualization clusters, with a ton of nice features: live migrations, redundant storage over DRBD, support for KVM and Xen (I think support for a few other virtuualization/containerization systems is in the works too), scriptable deployments and an HTTP API. It is, as I understand it, used internally at Google (they developed it) for a few things, and judging by the mailing lists there are a number of other good sized deployments around as well.

In a few hours you can set up your own internal "cloud" with Ganeti, and its fairly easy to scale later as the need arises. Bring a new node online, get on the Ganeti master node and tell it the hostname and root password, and Ganeti will SSH in and take care of most of the setup for you.

I'm a developer on swift (OpenStack storage == Rackspace Cloud Files). We've been pretty excited about opening this, and I'd love to answer any questions you have.

Swift documentation is at http://swift.openstack.org. Check out the "SAIO - Swift All In One" page to see how to set up a VM to run it. With some very small tweaks, you can also run it on a single slice (mount a loopback device rather than another virtual drive).

Although I don't know as much about it, the compute docs are at http://nova.openstack.org

I'd be curious to hear how Swift compares to MogileFS. MogileFS never took off like memcached did, but it's another piece of infrastructure that came out of LiveJournal and similarly allows one to create a storage architecture where files are redundantly stored on commodity nodes.

I'm guessing that both Amazon and Rackspace took a look at MogileFS and its architecture so it would be informative to hear what you liked and what you might have found lacking.

Interestingly, our first implementation of Cloud Files was based on MogileFS. It's a decent design, but it doesn't scale to the levels we need. The central DB that stores the metadata was the bottleneck.

How does openstack compare to Eucalyptus?


Eucalyptus tries to match Amazon's APIs, and (from hearsay) it doesn't scale well.

Openstack doesn't match Amazon's APIs. Nova (compute) is used by NASA and will be used by Rackspace Cloud Servers. Swift (storage) is currently used by Rackspace Cloud Files. Swift (my area of expertise) is production-ready code that is scalable to massive levels (think 100-petabyte clusters and 100000 requests per second).

From http://openstack.org/projects/, most open-source cloud solutions don't scale with the exception of the software behind NASA's Nebula cloud, which as of April was Eucalyptus.

What languages/frameworks/tools did you use to develop swift and other openstack technologies? Also are there benchmarks that compares swift to other cloud computing systems out there?

swift is built entirely in Python using libraries like eventlet, greenlet, and WebOb.

Because benchmarks are very sensitive to deployment details, the only thing I can point you to is comparisons between Rackspace Cloud Files and Amazon S3. Of course, even then there are all kinds of variables like cluster size and network connectivity.

It's open, it's cloud, it's new, it's got NASA involved! So how does this help me move 20 terrabytes of data and a range of web services and customer facing applications from my current cloud provider to a new provider?

The theory is that I can take a data snapshot of my 20 terabytes, move it to the new provider's network by driving it across town (still the fastest way to move data), "drop it in", switch over DNS and excluding a few lost transactions during switch-over everything should work fine.

The reality is that this removes the pain of proprietary configuration from a data center migration but it's still a painful process to switch cloud providers that involves a lot of work, some risk and down-time. So while this is a great marketing play from Rackspace there is still a level of lock-in when you commit to using a provider and so the risk to Rackspace to go 'open' is minimal but the benefits of being the leader are huge.

I think it's more about the worry that if your chosen provider goes away you have no way to run the service yourself. You would have to rewrite everything using google storage to S3 instead of just migrating data centers.

Your big problem seems to be moving the 20TB of data. Why not just have the data get copied over to your alternative provider as you create it. That way all you need to do is spin up your images on the alternative provider. You might still lose a few transactions, though maybe having a message queue to handle that on both providers would be a good idea.

So you're saying that they are maybe deliberately understating the pain of migrating platforms with this solution simply to achieve recognition as being in a leadership position.

If you are correct, this sounds like quite a cynical move to try to hoodwink potential customers into a false sense of security with an apparent promise of platform independence.

If you have questions come join us at irc.freenode.com / #openstack

I'm sad RackSpace chose to market this effort under the term "Open Stack".

A number of us in the open web community have used that term to a great extent to talk about OpenID + OAuth + Portable Contacts + Open Social (http://therealmccrea.com/2008/09/19/joseph-smarr-at-web-20-o...)

Admittedly the initiative has lost momentum (for many reasons and factors) but things are still out there. This just creates confusion and branding collision.

I notice RackSpace is pushing a "TM" on their OpenStack branding, which calls into issue anyone using the term for it's prior and original meaning.

I'm sorry, we didn't think the two communities would cause confusion. Almost everything you can name something these days will cause a "name collision" of sort. The infrastructure and application folks are pretty separated and so are the concepts so hopefully the context in which it is used will make it clear. Detail wise, the OpenID + OAuth... is "Open Stack" and this should be used only as "OpenStack".

Hey Bret, I hear what you are saying. Yes, the infrastructure and application communities are separate. Yes OpenStack != "Open Stack".

But the wider communities; the customer communities and the developer communities are going to be in that overlap between both infrastructure and application.

And we both know that once your brand goes through 'the wash' of the media it's going to be munged into "Open Stack" as much as "OpenStack".

Sure everything has some kind of name collision but that's just indicates why more deeper investigation is required to ensure such collisions occur between two very different usages. It is disappointing in this instance that didn't occur.

(I'm guessing/assuming you are a RS employee...) you also haven't touched on the concern that RS is asserting a TradeMark on "OpenStack" which brings some concern into the existing publicity for the Open Stack concept.

Preamble: thanks for the software, looks awesome.

The problem is that the domains are very close. There's a lot of modern web apps are cloud based and use the open stack (I recall Mike Malone and Joe Stump doing a great talk at PyCon about these topics using that exact term). This makes searching really really annoying.

How about OpenSkies?

OpenSkies is an airline

For some reason your distinction of OpenStack vs Open Stack reminds me of the attempted trademarking of a smiley face by describing it as two dots and a half circle contained by a larger circle.

It's called the Social Federated Web now or OStatus for the set of protocols.



I attended the design summit, and I believe this is going to be a really big deal.

Whether it's a big hosting company running a public cloud, a corporate running a private cloud, or a smaller dev/test cloud, they can now all run the same free software stack, and software can move freely between those environments. It's clear this announcement will have ripple effects throughout the entire cloud ecosystem. I hope the project will also boost adoption of cloud-style infrastructures, now that the fear of supplier lock-in is removed.

Rackspace had a design summit last week in Austin, TX for Openstack. I was an attendee and learned quite a bit from the folks at Rackspace. They've built and re-built their infrastructure enough times they know what they need to look out for. The drop-and-swap component model is great too.

It was fantastic to see what they've put together and the business model is great. Commoditizing architecture and software really is the next step.

If anyone is interested, I could write up a blog post about my experience, maybe provide a little more insight into the buy-in.

Please do write a blog post, and what do you mean by "drop-and-swap component"?

You should get a canonical URL. openstack.com and openstack.org both resolve now, probably better to redirect from openstack.com to .org.

Personally, I find the OpenStack Object Storage component more interesting than OpenStack Compute. I like the idea and implementation of S3 for saving important data. Similarly, the same sort of lazy replication of EBS just makes life easier.

I think that there will be a real sweet spot for smaller organizations who would like an internal system like S3 that offers robust data security with using really cheap disk drives and servers.

Hopefully OpenStack Object Storage will have good support for multi-datacenter replication.

I did downloaded it, looked at the source, but did not install it. OpenStack Object Storage looks like a very interesting project.

Okay I don't know much about Cloud Computing but can anyone tell how is going to help someone like me who has one or two blogs with some web hosting company.

Or should someone like me even care?

Here's a possible advantage: you have your site, some php code, mysql, apache, a ton of data, maybe more stuff on disk. Your hosting service jacks its prices, or announces it's shutting down.

Right now, to move to a new host, you need to handle a number doable but annoying steps. Imagine tho that you could just download "The Site". Web server, blog app, database, everything. Including the OS. All as one (albeit large) file.

And then just upload it to a new host, where it Just Works. No reconfig because you're on a newer os or have to use a dfferent version of mysql. All of that is bundled up for you.

This is what virtualization (i.e. cloud computing) can buy you. And an open standard for infrastructure can make it easier to move your virtual system to other hosts.

Because the hope is that openstack will do for cloud hosting what Apache did for single-server hosting. And now you can install it locally on your own laptop or on a cluster of your own or pay someone else to host it.

While I'm running on AWS I strongly support a push for interoperability and freedom from lockin. We'll just have to see what kind of adoption this project gets.

Does anyone know how this compares with Eucalyptus http://open.eucalyptus.com/ that's the core of Ubuntu Enterprise Cloud http://www.ubuntu.com/cloud/private ?

In April, NASA's Nebula project was based on Eucalyptus


Does this mean they are completely moving away Eucalyptus?

What benefit does Rackspace get from doing this?

They commoditise the software and sell the services.

This reduces the risk that a competitor will come along with either (1) a popular vertically integrated solution that locks you out of market share or (2) some arrangement that allows them to dictate terms to you. Further, if a participant does emerge matching one of those patterns, they are organised respond through their established platform.

IBM does something similar with linux, which competes with non-free unix solutions and Windows. Google have done this with android, which reduces the risk of their being held to ransom by Apple.

What do you mean by "commoditise the software"? If it's open source, how much of a commodity can it be?

Also, and again I apologize for not understanding your language here, but what do you mean by a "vertically integrated solution"? Could you give an example to illustrate what it would mean for a competitor to do this?

It would be interesting if there is a competing solution on the horizon, the success of which Rackspace is trying to preempt with this offering. I guess time will tell.

When I think of commodity I think of oil, or grains, or pork bellies. A good that is generic and interchangeable, 'fungible'. You're not locked in to a source, you can get it from several sources in a competitive marketplace.

    Also, and again I apologize for not understanding
    your language here, but what do you mean by a
    "vertically integrated solution"? Could you give an
    example to illustrate what it would mean for a
    competitor to do this?

Let's start with an example of vertical integration: Apple has a strategy that involves heavy vertical integration. They sell the hardware and the software and the services. If you want OSX, you have to buy the hardware. More so in the phone space: if you want IOS, you need to buy an iphone. Once you've bought an iphone, you have to buy apps for it through the appstore. Even if you hate one part of the Apple solution, you may be forced to take the good with the bad. Even if you only want one thing, you generally have to get everything. If you want to run something that's not in the app store, too bad.

A scenario where this could happen with cloud services: imagine Amazon came up with a really good payment services platform, and then said that in order to use it, you had to live in their cloud. The market would have to interact with their cloud to get access to the payment services. Rackspace would lose access to the segment of the target market that needed this functionality.

By building a capability on the software side of cloud services, Rackspace will have better options in a situation like the one above:

* They will have a platform that they can build a competing payment services platform.

* It's possible that other groups who want the same outcome will contribute to their platform, strengthening it in lots of small ways to compete against other offerings.

* There may be other developers out there who are skilling up for their own ends, that Rackspace can hire at short notice if it needs to skill up quickly.

* They can start building a library of prior art and patents to use in defence against sniping from one proprietary operators in the cloud space.

In the mid-90s, IBM found themselves in an odd situation. They were offering a desktop and corporate operating system platform, but weren't very good at it because it wasn't their focus. Microsoft were good at OS strategy. So IBM found itself in a situation where it needed to invest heavily in Windows NT to avoid losing ground, and this meant that another company was setting its direction to a large extent.

Since then, IBM has got behind linux in a big way. There's far less risk of agenda-setting coming at them from the linux community. It's a stronger proposition for them to offer their customers, because potential customers don't have to fear IBM trying to lock them in to IBM products because in the way that customers might have been wary about this with AIX or OS/2 (or mainframes).

Imagine if Apple move into cloud services, and then get a patent on something ridiculous, and then use that to hound opponents. That's what they're doing in the phone space.

I think the guys at Rackspace have their heads screwed on. I'm looking forward to seeing a free and open cloud platform emerge.

   imagine Amazon came up with a really good payment 
   services platform, and then said that in order to 
   would have to interact with their cloud to get access 
   to the payment services. Rackspace would lose access 
   to the segment of the target market that needed 
   this functionality.
Good example, but I thought the idea with OpenStack is to allow anyone to take this and run with it, customize it as they like and eventually end up with their own platform, potentially distinct from Rackspace's. In your example, Amazon says "in order to use the payment system, you have to live on the Amazon cloud" but I don't see any parallel restrictions in OpenStack, simply because it's open source and can be tailored unpredictably.

It's for this same reason that I question whether the benefits to Rackspace that you pointed out above really are such a sure thing for them. Seems like it could just as easily turn against them if someone else turns their open source solution to better advantage (assuming the open source license permits this).

    Seems like it could just as easily turn against them if
    someone else turns their open source solution to better
    advantage (assuming the open source license permits this).
Sure, but that's their comfort space. They're confident of their skills and economies of scale here, and enthusiastic for this to be the battleground.

My point with the amazon example was partly that if Rackspace found itself needing to build something similar, it has a stronger starting point if it already has a software group. And because the platform is open, they might have a shared interest with another company who has taken on their software to collaborate on new features. Or another party may build an open payment system on top of this open system when they're not even looking, and they benefit from the network effects.

it won't hurt them. the provisioning system isn't the expensive part of becoming a 'cloud' provider.

looking forward to the combined efforts.

we really want to make the cloud that hacker news users would want to deploy on their own infrastructure.

Adding the TC synopsis of the project posted in another thread for continuity. http://techcrunch.com/2010/07/18/openstack-org-rackspace-ope...

Seems right that NASA use Launchpad to host the Openstack code

is there a way to add DNS entries via API yet? I don't see where to get the code except for swift-1.0.0, and I'm not sure what that does.

swift is the storage piece (think S3 or Cloud Files). In fact, it is what is running as Rackspace's Cloud Files right now. Although it is listed as 1.0 and a developer preview release, it is production-ready code that is currently being used.

the API doesn't have DNS support yet

Swift is the storage platform for OpenStack. The compute platform will be released later.

Yay. A marginally competent hosting company just released the code to their mostly unsuccessful cloud! This is like IBM open sourcing OS/2 a year after they admitted defeat to Microsoft in the desktop operating system market.
