Cloudlets: universal server images for the cloud
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Hykes, Solomon
In this talk we will discuss the issue of server images, and how it affects inter-cloud portability. We will describe our vision for a universal format which can be shared and improved as easily as a Git repository, and how we're implementing it with cloudlets. Cloudlets are universal server images for the cloud. They're lightweight, version-controlled, and you can export them to any bootable format known to man: Xen, KVM, Amazon EC2, or just a plain bootable CD. They can be shared and distributed with the semantics of tools such as Git and Mercurial. Our goal is to build the foundations for truly cloud-independent infrastructures. Our roadmap includes: * Multi-image stacks * Auto-scaling * Automated integration tests * In-place image editing * Integration with existing VM generation and configuration management tools
 All right. Thank you. Everybody hear me all right? So I'm Solomon Hikes and let's talk about CloudLits. Before I begin, let me just say this is a conversation. I'm here to talk about a problem that we're trying to solve. It's not a problem that has been solved. So if you relate to the problem at all, if you have anything to contribute, please join the conversation on IRC, Twitter, come see us at the bar, whatever you like. So the problem is sharing. Specifically, how do I share my code when I want it to run in the cloud? So of course, the cloud is just a fancy word to say anything server side. So I'm writing an application. I want it to run on people's servers. How do I share that efficiently? So of course, as open source developers, we know how to share. We know how to share our code. We know how to collaborate with people who want to contribute to our code. And we know how to package our code so that people can install it on their favorite OS. However, if we're writing applications for the cloud, in addition to all that, you also need to provide configurations. Typically, as an example, let's say I'm writing a patchy. People want to see the patchy code. Other people want to install patchy on their systems. But there's a lot of interest in a great working configuration of patchy and all the necessary stuff to run Python apps or Ruby apps. A great stack that just works. And if I want to share that, then that's when things get complicated. I can see three options here. Either I write a tutorial, write in a blog post, and people will comment, edit it. Or I package a virtual machine and put it up for people to download. Or I use a dynamic configuration engine, things like Chef, Puppet, et cetera. We think that all three of these approaches have drawbacks. And the summary would be that we're good at sharing code, good at sharing packages, but we're still pretty bad at sharing configurations that just work. So what we tried to do is improve that situation and provide a format to package a configuration and share it. So a cloudlet is an image that you can use like a virtual machine. Basically, that means it's self-contained. You don't reference anything else. If I give you an image in the cloudlet's format, you have everything you need to boot it. And you have everything you need to boot it anywhere. From a cloudlet image, you can convert that into a Xen image, a VMware image. You can get a raw disk image and then boot it on a physical machine. We've done it with open VZ containers, things like that. So the interesting part, compared to virtual machines, I don't know if everybody has ever tried to share a VMware virtual machine, something like that, or Xen image with people who don't actually use that virtualization technique. That's really tricky. And what we really like is the tools that everybody here, I think, knows, something like JIT, Mercurial, SVN. I can clone someone's code. I can change it. I can commit the changes. I can merge. I can see the differences. All these cool operations that we're used to, we'd like to be able to do that on images as well. And that's basically what the format is made for. You can take someone else's cloudlet image, fork it, make changes and contribute the changes back, things like that. That's a quick example. If anybody here uses Mercurial, you'll recognize the actual Mercurial server interface. This is not our web code. This is an image that a friend of ours made. It's the bare WSGI server. So it's an Apache configuration to serve Python apps. You can see the complete history of the image. And you can inspect the specific change. Let's say, oh, here I see changes in the Apache configuration, things like that. Here, this is a change set where, obviously, as the comment says, he installed the module and you can see all the files that changed. The interesting thing is we didn't replicate the work of a Mercurial. We use it. We take the image and put it in Mercurial. You can actually do HG pool, the image. So that's the workflow we envision. The same way we say, here's the repository for my code. You can say, here's the repository for my image. Do whatever you want with it. And I think that's a really interesting way of seeing things because then you can start building on top of other people's work just like you did with code. And I think that might accelerate the progress we make in not just writing one piece of code here, another one there, but actually plugging them together and sharing entire cloud stacks. So I think that's as far as I'm going to go in terms of lecturing. I'm not going in any details on how it works. I was worried I wouldn't have enough time. But I would love to either answer your questions on email, Twitter, IRC, whatever you'd like. Join us at the bar, like I said. We're going to go there right after this talk and we'll be there again at five. Who here would like to see a demonstration later, just to see it happen? All right. So either right after this at the bar or at five. Is that right? And I think I have time for a few questions. Do you store the entire image? Yes. So binaries and everything? Binarys and everything. Please repeat the question. Oh, I'm sorry. The question was, do you actually store the entire image in the repository? The answer is yes. We store everything. However, we don't work at the block storage level like VMware or Xen, for example. We don't store that. We work at the file system level. So basically we take a file system tree and we say, okay, I'd love to put that in a repository, but what should I take out? So we add metadata so that the image is smart enough to say, if this file changes, you want to record that change. If this changes, you're not really interested in that. Things like that. But yes. Any other question? All right. Well, this is it. Thank you. I guess I'm ahead of time. Yeah. So you store all the bindings with stuff in your hardware image? You probably don't save it. Right. So the question was, you save the file system, but do you save any hardware information? The answer is no. We think in terms of moving from one cloud to the other, let's say you have a Xen image running and you want to share the behavior of that Xen image with someone who's using VMware. If any hardware information you share with him will be useless. So the interesting part is to look at all the information, all the bits you have in that image and only take what's worth moving. Is it possible to move from a VM image to Xen image using only the information you store and what you need to add? The question is, can you actually do that? Can you actually take information from a VMWare image, move that to a physical machine and is it enough? Will it work? The answer is yes. So the way it works is you start from the file system. Let's say you don't have a cloud image. You have a normal VMWare image and you want to use cloudlets to move that. The way you do it is you start with the raw file system and then you have to add the metadata the first time. So you have to, using the cloudlets format, you'll add information saying, this is persistent. This is volatile. This is a template, things like that. And then once you've done that, then you can use the metadata to move the image. And so that means there's an initial step of authoring, adding manual information and we've worked really hard to make that as simple and short as possible. So there's no giant XML file, no scripts to write, anything like that. Typically, the metadata is a 10 line JSON file. No, you don't have to describe the services. You don't have to give any knowledge of how the code works inside or how to construct the image or what are the relationships inside the image. You just focus on the file system, on the final result because whatever the configuration logic is, the end result is always a certain file system. So instead of, that's the big difference with Puppet or Chef, for example, we don't describe how to get to a certain behavior. We let you do that any way you want and once it's done, we let you snapshot that. And it's actually a great combination with tools like Puppet or Chef because if, who here uses something like Puppet Chef, CF Engine or the equivalent homemade scripts, typically you describe steps to take from one state to the final step you want, but you still have the problem of the initial state. A Puppet script describes how to change a base image, but how do you get that base image in the first place? So the typical scenario is here's a base image in the Cloudlet format and then here's a Puppet or Chef script to make it change its behavior dynamically once it's booted. So we do before boots, Chef Puppet, CF Engine do after boot. Great match. Is this something which only works on the kind of public, imperial repositories or can it work in like private bits, like editable private bits? Oh, yeah. The question is, does it only work on public Merciless repositories? I mean, I am guessing you mean things like Bitbucket or open source? Of course, yeah. Obviously here the focus is on sharing for open source developers, but the technology works for anybody who wants to share images with anyone. Just, it's the same workflow options you have as with Mercurial in general. Okay. Sorry. Not on this computer, but yeah, I'd love to, this is basically why I'm here speaking is I would like to interact with people who are interested in this problem. So this is really just to get the conversation started. Any other question? Okay. Thank you very much. All right. Thank you too. Thanks.
 All right. Thank you. Everybody hear me all right? So I'm Solomon Hikes and let's talk  about CloudLits. Before I begin, let me just say this is a conversation. I'm here to talk  about a problem that we're trying to solve. It's not a problem that has been solved. So  if you relate to the problem at all, if you have anything to contribute, please join the  conversation on IRC, Twitter, come see us at the bar, whatever you like. So the problem  is sharing. Specifically, how do I share my code when I want it to run in the cloud?  So of course, the cloud is just a fancy word to say anything server side. So I'm writing  an application. I want it to run on people's servers. How do I share that efficiently?  So of course, as open source developers, we know how to share. We know how to share our  code. We know how to collaborate with people who want to contribute to our code. And we  know how to package our code so that people can install it on their favorite OS. However,  if we're writing applications for the cloud, in addition to all that, you also need to  provide configurations. Typically, as an example, let's say I'm writing a patchy. People want  to see the patchy code. Other people want to install patchy on their systems. But there's  a lot of interest in a great working configuration of patchy and all the necessary stuff to run  Python apps or Ruby apps. A great stack that just works. And if I want to share that, then  that's when things get complicated. I can see three options here. Either I write a tutorial,  write in a blog post, and people will comment, edit it. Or I package a virtual machine and  put it up for people to download. Or I use a dynamic configuration engine, things like  Chef, Puppet, et cetera. We think that all three of these approaches have drawbacks. And  the summary would be that we're good at sharing code, good at sharing packages, but we're  still pretty bad at sharing configurations that just work. So what we tried to do is  improve that situation and provide a format to package a configuration and share it. So  a cloudlet is an image that you can use like a virtual machine. Basically, that means it's  self-contained. You don't reference anything else. If I give you an image in the cloudlet's  format, you have everything you need to boot it. And you have everything you need to boot  it anywhere. From a cloudlet image, you can convert that into a Xen image, a VMware image.  You can get a raw disk image and then boot it on a physical machine. We've done it with  open VZ containers, things like that. So the interesting part, compared to virtual machines,  I don't know if everybody has ever tried to share a VMware virtual machine, something  like that, or Xen image with people who don't actually use that virtualization technique.  That's really tricky. And what we really like is the tools that everybody here, I think,  knows, something like JIT, Mercurial, SVN. I can clone someone's code. I can change it.  I can commit the changes. I can merge. I can see the differences. All these cool operations  that we're used to, we'd like to be able to do that on images as well. And that's basically  what the format is made for. You can take someone else's cloudlet image, fork it, make  changes and contribute the changes back, things like that.  That's a quick example. If anybody here uses Mercurial, you'll recognize the actual Mercurial  server interface. This is not our web code. This is an image that a friend of ours made.  It's the bare WSGI server. So it's an Apache configuration to serve Python apps. You can  see the complete history of the image. And you can inspect the specific change. Let's  say, oh, here I see changes in the Apache configuration, things like that. Here, this  is a change set where, obviously, as the comment says, he installed the module and you can  see all the files that changed. The interesting thing is we didn't replicate the work of a  Mercurial. We use it. We take the image and put it in Mercurial. You can actually do HG  pool, the image. So that's the workflow we envision. The same way we say, here's the  repository for my code. You can say, here's the repository for my image. Do whatever you  want with it. And I think that's a really interesting way of seeing things because then  you can start building on top of other people's work just like you did with code. And I think  that might accelerate the progress we make in not just writing one piece of code here,  another one there, but actually plugging them together and sharing entire cloud stacks.  So I think that's as far as I'm going to go in terms of lecturing. I'm not going in any  details on how it works. I was worried I wouldn't have enough time. But I would love to either  answer your questions on email, Twitter, IRC, whatever you'd like. Join us at the bar,  like I said. We're going to go there right after this talk and we'll be there again at  five. Who here would like to see a demonstration later, just to see it happen? All right. So  either right after this at the bar or at five. Is that right? And I think I have time for  a few questions.  Do you store the entire image?  Yes.  So binaries and everything?  Binarys and everything.  Please repeat the question.  Oh, I'm sorry. The question was, do you actually store the entire image in the repository? The  answer is yes. We store everything. However, we don't work at the block storage level like  VMware or Xen, for example. We don't store that. We work at the file system level. So  basically we take a file system tree and we say, okay, I'd love to put that in a repository,  but what should I take out? So we add metadata so that the image is smart enough to say,  if this file changes, you want to record that change. If this changes, you're not really  interested in that. Things like that. But yes. Any other question? All right. Well,  this is it. Thank you. I guess I'm ahead of time.  Yeah.  So you store all the bindings with stuff in your hardware image? You probably don't  save it.  Right. So the question was, you save the file system, but do you save any hardware information?  The answer is no. We think in terms of moving from one cloud to the other, let's say you  have a Xen image running and you want to share the behavior of that Xen image with someone  who's using VMware. If any hardware information you share with him will be useless. So the  interesting part is to look at all the information, all the bits you have in that image and only  take what's worth moving.  Is it possible to move from a VM image to Xen image using only the information you store  and what you need to add?  The question is, can you actually do that? Can you actually take information from a  VMWare image, move that to a physical machine and is it enough? Will it work? The answer  is yes. So the way it works is you start from the file system. Let's say you don't have  a cloud image. You have a normal VMWare image and you want to use cloudlets to move that.  The way you do it is you start with the raw file system and then you have to add the metadata  the first time. So you have to, using the cloudlets format, you'll add information saying, this  is persistent. This is volatile. This is a template, things like that. And then once  you've done that, then you can use the metadata to move the image. And so that means there's  an initial step of authoring, adding manual information and we've worked really hard to  make that as simple and short as possible. So there's no giant XML file, no scripts to  write, anything like that. Typically, the metadata is a 10 line JSON file.  No, you don't have to describe the services. You don't have to give any knowledge of how  the code works inside or how to construct the image or what are the relationships inside  the image. You just focus on the file system, on the final result because whatever the configuration  logic is, the end result is always a certain file system. So instead of, that's the big  difference with Puppet or Chef, for example, we don't describe how to get to a certain  behavior. We let you do that any way you want and once it's done, we let you snapshot that.  And it's actually a great combination with tools like Puppet or Chef because if, who  here uses something like Puppet Chef, CF Engine or the equivalent homemade scripts, typically  you describe steps to take from one state to the final step you want, but you still have  the problem of the initial state. A Puppet script describes how to change a base image,  but how do you get that base image in the first place? So the typical scenario is here's  a base image in the Cloudlet format and then here's a Puppet or Chef script to make it  change its behavior dynamically once it's booted. So we do before boots, Chef Puppet,  CF Engine do after boot. Great match.  Is this something which only works on the kind of public, imperial repositories or can  it work in like private bits, like editable private bits?  Oh, yeah. The question is, does it only work on public Merciless repositories? I mean, I  am guessing you mean things like Bitbucket or open source?  Of course, yeah. Obviously here the focus is on sharing for open source developers,  but the technology works for anybody who wants to share images with anyone. Just, it's the  same workflow options you have as with Mercurial in general. Okay. Sorry. Not on this computer,  but yeah, I'd love to, this is basically why I'm here speaking is I would like to interact  with people who are interested in this problem. So this is really just to get the conversation  started. Any other question? Okay. Thank you very much.  All right. Thank you too. Thanks.
Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_0_131.png Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_132_179.png Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_180_383.png Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_384_839.png Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_840_1343.png Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_1344_3239.png Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_3240_3479.png Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_3480_3659.png Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_3660_5579.png Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_5580_6155.png Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_6156_6323.png Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_6324_6863.png Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_6864_17543.png Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_17544_17591.png Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_17592_17723.png Cloudlets__universal_server_images_for_the_cloud_kL30gj2KY_k_1_17724_17842.png