Is distribution-level package management obsolete?
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Berkholz, Donnie
Recent trends in software development have raised questions as to whether package management in Linux distributions is still relevant. Whether it's independent package managers in popular Web frameworks and languages (Node.js, Ruby, Python, etc) or bundling and containerization that's become increasingly popular in DevOps culture, it appears that integrated approaches to package management are on the decline. Yet at the same time we've seen package managers in the Windows world such as NuGet grow more popular. This talk from a leader of the Gentoo Linux distribution will explore the reasoning and history behind this shift and whether it's the right move for the FLOSS movement as a whole
 Alright, are we good from the AV crew? Perfect. Thanks. I apologize for my voice. It was a great night last night at the beer event. And I got here about five minutes ago after scrambling to catch the last bus that we've been getting here in time. So I won't have any references to earlier talks from this morning because I was sleeping. And probably too drunk to get to talk anyway, so that was good. It's great to be back. I love coming to FOSM every year. And I've been doing gentey stuff for a long time now. I started... I became a gentey developer. I think it was in June of 2003. So I've been doing this just for a long time. I've been thinking about it for a long time. And now, in the past couple of years, I was fortunate enough to become an employee of that company that lets me do research and think about this stuff full-time. So now I work at Red Monk. My job is called the industry analyst, which nobody really knows what that means. But what it means to me is that I could spend all of my time doing research about the state of the art and software development and how that compares to what people are actually doing, where they actually are. And now I'm able to sort of bring that back to my initial love and my initial open source love of Gentoo. And say, look, where is Gentoo? But not just that. More importantly, where is everybody today in the distro world? And where do we need to go? And why are so many developers running Macs instead of Linux? And part of that is just because the experience sucks. So we're going to just sort of walk through that today. And Joe is going to see some of these slides for the third time, although I'm saying something completely different. Because I've given three talks in the past four days. But it will not be the same content. I promise you that. So some of the questions, and I'm trying to avoid bullet points, but sometimes I just want to start with a few of them. The key questions for me when thinking about, all right, where are distros today? Where do they need to go? Is how are actual software developers in the real world doing things? And how could distros do a better job of catering to them? How are users doing things? How could distros better cater to them? Because there's all of these things happening out there that we, as distros, and I'm curious, how many of you are volunteers for a Linux distribution? Just raise your hand. Excellent. Perfect. How many of you think your distribution has solved all of the difficult problems? Again, good. So you're here for a reason. You want to think about this stuff. And feel free to catch me after, and we can have beers and talk about it even more. But the things I want to go after today is, where are developers? Where are users today? Where do they need to be? And what do we need to do to build the right kinds of things for them? And less importantly, down at the bottom, is what do we want? It's great to build a distro that's friendly for new maintainers to get into the distro, and it's more important to build one that people actually want to run. If you don't have people who want to run the thing, you're never going to get anybody who wants to become a maintainer, because it's this whole funnel thinking about things like nobody becomes a maintainer magically without being a user first. So we need to build software that users and that developers want to run. And that's not happening in the way it has to today. I'm going to start with a little bit of data, and I apologize for the black and white, but I have no excuse. I did a little bit of data mining across a few different data sources, and you may not know it, but when you talk in places like Hacker News and do Google searches, I'm searching for you what you did. I am looking behind the scenes and trying to understand what are people doing and why. And some of this data is more reliable than others, so if you don't like one particular data source, that's fine. What makes it interesting is the trends across multiple data sources, and that's also what makes it more reliable to me. So I can look at this stuff and say, okay, Google Trends, fine, if you don't like that, that's okay. But you look at the trends across Google Trends, across Hacker News, across Stack Overflow, for example, across GitHub. I've even done some mining of IRC logs that I've had on my machine, because I've been doing this stuff for long enough that there's some data in there, and you can do some fun things. And then across, in this example, one of the others is Linux Journal, where they actually ran a survey of what distro are you running. And what you see is not really surprising, there's a ton of Ubuntu users, just loads of them. But what is interesting is Arch, right? And so many pointed out on Twitter, I think it was last night or this morning, I don't think there are any Arch talks here. How many of you are Arch users? So what's going on with that, right? And it's one of those things where you've got this set of traditional distros, the set of new distros, and even the new ones, I don't think, have solved the problems that need to be solved. And that's what we're going to talk about today. But if you look at this stuff, you see some interesting trends emerging. In terms of, if you think about this and try and build some intuition about what are these people like who are using these different data sources. At least to me, I think, okay, Google Trends, that's just normal people. At least insofar as it can be normal and be running Linux at the same time. Now, Linux Journal, alright, these people are interested in technology, they're a little bit more technically savvy. Hacker News, they're actual developers, they're not just users and enthusiasts. And so that was sort of my intuition with this. And what's interesting about looking at this data is that intuition holds true. You get trends that sort of make sense when you think about where are the developers, where are the enthusiasts, and where are the more normal people running Linux. And I don't think any of that last group is here today. So we're just going to look at data from them and talk about them without seeing the rarely observed normal Linux user. But if you look at this stuff, you see some trends that make sense. You see things like, wow, there's so many developers running Arch. But in terms of enthusiasts and normal users, there's a little bit less. Whereas you go look at Mint, and I'd love to make fun of Mint just for, well, we won't get into that today. But in terms of normal people running Mint, there's loads of them. People love Mint, it's super easy, it just tells you, like, here's what you need to do, and we're going to do it for you. And you don't need to think about that, and you don't need flexibility. So, how did you, maybe, so you could get Mint like as a copy? Right. I think all of these searches included Linux. So it was Mint, Linux, Arch, Linux, Debbie, and Linux will have you. And not GNU, Linux, unfortunately. So if RMS is here, I'm sorry for that. Yeah, so there's ways to do searches such that you can keep out some of the stuff you don't want to happen, although it's never perfect. Like, for example, searching for Arch gives you all of the architecture stuff related to Linux as well. You have to deal with it and do some controls to see. You know, if you actually look through the results, you realize that after you look through 100 of them, less than one of them talks about architecture, as in the other 99.9 talk about Arch. But it's interesting to look at these things and say, like, oh, this is where things are in the closest sense we can get to real data. Because almost nobody actually installs Popcon or something like it. We don't know how many Linux users there are. We don't even know what the breakdown looks like across different distros. This is about as close as we can get to that sort of thing. Where you look at multiple data sources, you build this list of, okay, it's pretty consistent across a lot of them. The only one where there's a great deal of confusion is Mint. I don't know. Is this number more right or is that number more right? But in almost every other case, it's fairly consistent and consistent with at least my intuition of so many people used Ubuntu. And unfortunately for me, not so many use Gen2 anymore. But 2004 was a great year for us. And you can look at this a little bit differently and say, okay, if Hacker News is the zero point, what are the other data sources look like in comparison? And again, you see things work out sort of in the way you would expect. Some things are much more biased toward developers. Some are more biased toward normal users. And some of them, what makes it really weird is some of them, like Arch is a great example, apparently enthusiasts don't like Arch, but everybody else does. So people who just enjoy playing around with stuff, I don't know. I'm not sure how to explain that, but most of this stuff does actually make sense when you think about it. But I want to get away from the data and start talking about ideas. And if you look at just trends over time without any axis to show you volume, just look at raw trends over time, you can get some very interesting ideas of which way things are going. And I'm not saying by any means that there's more people running this or that, but which way are things going, what do people care about enough to talk about it? And I put in red everything that's going up or remaining stable, and in gray is everything else. And in fact, it's interesting to me, in part because there's two new distros, we're going to call them new, even though they're not that new. Two of the newer distros are going up. And then sentos is actually pretty stable, which is interesting to me. Yeah. I'm sorry? The scale? Oh, the time scale. This is since 2002, 2003, so it's about the last 10 years or so, which is why it basically doesn't exist until more recently. In art you can see some of the noise that I was talking about, where people are just talking about architectures and Linux back then, and then it picks up. And the same thing with openSUSA, because it was SUSA, until they decided to call it open and make it more open, I guess. But it is of value to look at this thing and think about, wow, people don't seem to care quite so much about things like, well, Slackware, Gentoo, Fedora, Debian as they used to. This doesn't mean they aren't running it, but they are talking about it. It's an assumption. It's just underlies their lives at this point and they don't care. And so that's one of the points of this talk, is how do you keep the distro relevant? How do you keep people caring about it? And one of those things is you have to integrate with the way they want to work. And distros kind of built this idea of, here's how package management should work. We're just going to keep it that way forever, instead of kind of moving forward with the times. And that's created loads of issues. And one of these trends that's happened over the past eight years or so is cloud. How many of you have ever run anything in a cloud? Right. And did your distro do a good job of that? Or were you running a Mac at the time, maybe? Just integrating with this new style of development is something that distros have not done a great job of doing. You know, they don't, for the most part, do things like natively cluster, like share configuration data. Like, you know, this is a guy named Randy Bias, started a company called Cloud Scaling. He has this fantastic analogy of cattle versus pets, where in a cloud world, all of the servers are cattle. And if one gets sick, you just take it out back and you shoot it. In a world we live in, the servers are pets. They're carefully taken care of. They're certainly not kicked, Joe. So he made a joke about kicking a cat in the elevator with him the other day, and I think he just about flipped. But distros are still in that world of naming everything carefully and carefully installing it, for the most part. Although there's these new things that have come along, and I tried, the reason I'm here late is because it took me 15 minutes to get past the configuration management room. And that's something people care deeply about, and that distros, for the most part, don't handle well. You know, there's some modules that are written to sort of interface with the distros and run RPM or whatever behind the scenes, but it's not native. And at the distros side, we're like, oh, we don't really care about that. I mean, sure, people can run that if they want to. But, you know, for example, why don't we install all the distros using puppet or using Chef? Why don't we just set that on the machine, and that's the way you do the package management? Because that's the way people want to run the stuff. And so they slap on their own package management on top after the fact, and you've got this weird mishmash of things happening, or, okay, they've got Cpan, they've got PyPy, they've got, you know, they're installing gems, and none of that stuff talks to each other. And so you've got like 20 copies of the same thing in different versions, and things break, and it's just not a great experience. And that's the sort of configuration management thing. That's the point, right? Is that developers today, maybe not you, but you're not the only developers I do research on, are looking for an experience that uses configuration management. And it's not today either. I mean, even Gentoo's own infrastructure runs, well, still, CF Engine, slowly moving forward to something a little more modern. CF Engine's been around for more than 20 years now. It was 1993 when CF Engine 1.0 came out. And we're still not thinking about, in fact, do any distros here, do you think you do a good job of integrating with config management? Okay. But that's the way people want to manage their systems, because we're working at high scale, we're working with transient things, we don't want to sit around installing things, and, you know, we came up, I think Red Hot initially came up with the idea of Kickstart quite a while ago, and we kind of did that, and then we stopped. And we need to go farther. We need to start integrating these things into the way, not just the way we develop the distro, but the way users install the distro. I mean, why don't you give people an option to upload an encrypted repository configuration management, and we could store it for them? It would be easy. We already managed loads of weird servers doing all kinds of random things for a distro anyways. Why not make it a better experience for the users? When they destroy their install as they inevitably do, and I inevitably do, we've got the data form, and it's ready to replicate. And one great example of this, of how poorly we integrate with configuration management, is Chef, which told the Debian folks, you guys are packaging us in such an out-of-date and terrible way that we don't even want to be in your repo anymore. Just please take us out, because we're doing a better job of being up to date and having all the packages that matter to us. We're the domain experts. Get us out of here. And they did, which was great. They listened. But that's a real problem, and it's a problem for a lot of reasons, because, you know, as distros, we create all of these policies for packaging and guidelines and code for a reason, and so everything works together, and it's all happy. And now we're in this weird sort of middle ground where if you've got ISVs, software vendors, writing their own packaging, it doesn't understand how to talk to the rest of the distro. And it's in between, in this point where it depends on things in the distro. Other things depend on it, but it doesn't integrate right. And so you end up with all kinds of bugs that, well, some of my colleagues would love to talk to you about for hours. And so, you know, we're in this weird place where, you know, as distros, for the most part, we want to be stable, but software like Chef, especially software that's sort of cloud-like or DevOps-like, you know, there's a lot of innovation happening very quickly. How do you keep up with the pace of that? Because users want to run the newest software. They want the latest features, but at the same time, how do we put out something that's stable? And this is something I don't think anybody's really solved yet. You know, some people are taking an effort, like, if you talk to the Red Hat folks, they'll tell you about software collections where they're trying to put, you know, newer software for developers on top of RHEL, but it's not in a place where, you know, that's a normal behavior yet, and it's not in a place that users and the developers running the distros are looking for. And in all honesty, I'm not sure of the right answer for this question, but it's a question that you have to think about and decide what is the right answer for you so you understand what you're optimizing for and what you're giving up. I am curious, how many distros actually have continuous integration setup for all your packages? Which one? Okay, so for the core packages. But almost none of them, but that's the way that, you know, if you're a professional software engineer working at a modern company, which most of them are not, you have CI setup and every commit is tested and built. But we're not doing that with distros and we're not doing things like, how many of you have heard of continuous delivery or continuous deployment? Excellent. Are you doing it? Wow, awesome. Great job, guys. That's a good 25% of you who are doing continuous delivery. But most of us aren't doing it with a distro. And this is one point where I am actually very proud of Genti because we've been shipping software to users within an hour for the past 10 years. But we're not actually testing it. The users are the testers. And that's why they're called betas, right? But we need to start delivering software continuously and tested so that they can get the modern stuff, but it also works and we're not. And so we've got, you know, all of these different package managers all over the place. Every language has its own. Some languages have like three or four. JavaScript is a great example. I mean, there's NPM, there's Bower, there's Jam. I mean, how many different JavaScript package managers do you actually need? And I guess we haven't gotten to a large enough number to answer that question yet. But none of them talk to each other. And I mean, we've put some work in a Genti on trying to integrate this stuff and say, hey, let's download stuff from Plap High and then generate a Genti eBuild with it. And that's going part of the way toward the right answer. But I think the answer that we need is that package managers need to natively understand this stuff. Having a translator isn't good enough. Because people are, you know, installing this stuff from all over the place. And we're actually more modern developers using things like Node and Ruby on Rails as much as we don't want to talk about that here because we're all Libre and we love Copy Left. That's where people are and that's what we need to build for. And we're not. We're doing a terrible job of this stuff. And so people end up with, you know, five different copies of some of the lower level things in Node, for example, installed by five different package managers. And it's just not an ideal situation because you never know what's using what with all kinds of interesting issues. And so what do we do about that? And I think that integration is going to be the answer. You've got normal developers today are using all kinds of crazy data stores. For the most part, we're still storing package management data in flat files. You know, why aren't we keeping it in a more queryable way, in a more reliable way than just sticking stuff in a directory somewhere in a bunch of text files? I mean, it gets corrupted all the time and it creates all kinds of problems. And I know some distros actually do, but a lot of them don't. And if you store things in a way that's more friendly and more transparent to the developers running the stuff, you know, what happens? It lowers the barrier for them to get involved because the distro runs in a way that they are used to working. And so they're more likely to become a distro developer, a distro contributor, because it's friendly to the way they want to develop. And now all of these things are packaged, but, you know, we don't rely on any of them. And there is a point too, and I'm going to agree with the criticism you want to say, that we shouldn't have to rely on this stuff. We never want to get a Cassandra cluster up and running just to install a package. Sure, you're right. But at the same time, you have to optimize for something, and I don't think that something should be, you know, text files at the expense of data consistency. I'm not sure that's the right trade-off. You know, what else is happening out there in the real world where we're not, for the most part, except for the 25% of you doing continuous delivery, people are doing all kinds of containerization, right? I mean, virtualization isn't even interesting anymore. Now it's got to be containerization. It's got to be LXC, and, you know, front-ends on top of LXC, or on top of VirtualBox in the case of that grant. Everything is happening in containers, but none of us are doing anything with containers. I mean, none of the distros, to my knowledge, understand how to work with containers or install anything in containers with, I think, one exception, CoreOS, which is built around Docker. And that's, you know, one of the things, one of the case studies I'm going to bring up over and over, if I have time and I don't just run out. I'm halfway through. Cool. I mean, we need to start taking the software that developers want to use and optimizing for that. Instead of pretending distros are a thing that, like, we figured out where they should be in 1998 and just staying there, and that's what we have done, and it's really a shame. And BenT has done a great job on the UI side for normal users, but for developers and for technical people, you know, even they've kind of fallen short, I think. So why, for example, aren't we able to install every single application and application stack into its own container and just use, you know, copy on way to cross containers to manage all the dependencies? Why is it so tough to start a distro in the first place and just get it installed? Why aren't we shipping a Vagrant box? And that's the way people want to do it. Do you know any of you who work on distros, do you ship a Vagrant image, a Vagrant box? Which one? Ubuntu. Excellent. So we've got the vast majority covered. Unfortunately for the rest of us, we're not Ubuntu. I bet nobody does anything with Docker. Oh, yeah? I know they said we're going to support it. Do they do? Yeah, that's going to be a pressure last four months. All right, because of OpenShift and Paz's are awesome and no ops and all that good stuff. Can you tell me more? What does Fedora do with Docker? And I'll tell everybody else. Yeah, I'm not the right person to answer that. Okay. In the Config Management room? Oh, so nobody will be able to get into here, unfortunately. But perhaps it's on video. So we've got all these things happening out there in the real world. And this is another great example. Well, two examples I want to talk about. One of them is Chef has this thing called omnibus packaging, where what they do is just bundle everything up and ship it. Pretend the distro doesn't exist, basically, which means we have failed. If they have to go that far, we have utterly failed in our purpose. And our purpose should not just be, Gentoo is great, or Fedora is great, Ubuntu is great, but Linux is great. And people in the real world have to run stuff across multiple distros, but we're not talking to each other enough. We come here once a month, once a year, and do it. And then we go home and mostly don't. I've been on this list at freestess.org called the Distributions List for a number of years. I don't actually remember when the last post was. So hopefully some of you will go home and subscribe to it and start collaborating a little bit more. If that was the only thing that happened as a result of the stock, I'd be very happy. So, you know, you've got Chef doing things, and Chef keeps coming up over and over. They first pulled their packages and then they said, screw package managers entirely, we're just going to do this all ourselves. But there's more and more companies going that way. Chef isn't the first example, it's not the last example of just saying, and they say omnibus because it's got everything in it, right? They just ship the entire application stack all the way down to and including Ruby itself. So they're not even using the system Ruby installation. We failed. Another example is FPM. Depending on who you talk to, it stands for various levels of profanity with the second to being package managers. And that's one of those things where people in the real world who have to build packages and even want to integrate are having such a hard time with it that they had to write their own solution to build packages because it's such a painful experience. And so what FPM does is you unpack a tar ball, bail in and install it, and then run FPM and it generates a Debra and RPM for you. Because people have such a hard time writing this stuff, we've made it too difficult. We haven't thought enough about the barriers to entry to creating a package because we've been too focused on making the experience perfect for the experts. And what else is happening now? Well, we've got a lot of people who don't even bother making releases. They don't even make a tag and get. They just say, go clone it from GitHub and run it. And this is a thing that I see people shaking their head. I don't like it. You don't like it. But it's happening and so we have to deal with it. People are doing this stuff. Again, because the experience is so painful, what are we optimizing for? Well, it's not what developers want to do today. We're not optimizing for use of entry into writing packages, into creating packages. We're not optimizing for people who need to run stuff across multiple distros. We're not optimizing for people who need to do configuration management or want to do configuration management and who are sick of editing text files and configuration files in Etsy. But we need to because the distro is becoming irrelevant. There's a lot of people who don't even care. And in fact, it's... No, I'm not... Am I in video? I guess I am. It's actually kind of embarrassing now because Windows is doing a better job with some of this stuff. Windows package management has finally reached the Windows world. I just learned about this myself like six months ago. These are Windows package managers. They all work. They do stuff. There are repositories. And that wasn't the case a couple of years ago. It was like, go install Sigwin and then you can do some kind of package management. And you know the thing that Windows really has going for them is they really understand people who like to click GUIs. They really deeply understand people who like to do points and click and getting a good user experience. And once they're bringing that to the developers who are using this stuff, those are good chance that they'll get this problem solved before we do in the Linux world. And so, you know, what kinds of directions do we need to go? And I put up a few examples here. If you actually know the story behind them, you'll know that two of them are at some level based on Gen2 or its package manager. I'm kind of proud of that that they're in the directions that I think everybody else needs to go. The third one, I think there's actually talk on NixOS at some point. I encourage all of you to go check it out because it's really, really interesting ideas in terms of where we need to go as distros. And so, you know, we need things like, hey, everything is ephemeral. You don't need to care about package managers or, you know, in the case of Chrome, the browser. We've solved that for you so you don't need to think about it and we'll just keep all the data. You can tell us everything. But CoreOS is a very interesting idea. It's super minimal system. It's designed for the cloud. It's built using Portage, which is Gen2's package manager, but nobody who knows, who runs it knows that. Because inside of CoreOS, if you care about packages, you just use Docker. And again, it's one of those things where the developers want to actually do. They want to use things like Docker. They don't want to care about package managers. If you don't make a package manager a good experience for them, they're going to ignore it and go just search. And go just start installing random stuff and using ChefAmdabus and, you know, picking a language level package manager, perhaps. And now this isn't something you have to do. It's not mandatory for distros to matter. It's not mandatory for them to succeed. If they die, it's okay. Nobody else is going to care. But if you and I want to remain relevant, we have to keep up with these things that are happening. And yeah, one of the reasons I put this text down here, I'm not sure if you can read it, but there's this key phrase here that says, use Docker as a package manager to build and push your app. Docker is not a package manager. It doesn't know anything about dependencies. What you do is you write this Docker file script and in that script, you run something like yum install xyz. And that's the concept of package management. It's not the way we think about it, but it's the way people actually want to use it. They don't care about package management. They don't really care about, even about things like eventual consistency, like configuration management does for you. They just want to get stuff up and running. And these, you go check out all of these if you haven't yet, because these are the kinds of ideas that people want today. This coro s thing is just going nuts. People love it because it works in the way people want to work today. It does things like, oh, and they use go, of course, and go is, I don't know if you've heard it yet, but it's the trendiest language ever since node. They use go, they have this clustering demon using the raft algorithm that maintains consistency of configuration data across Etsy on every single installation. It's based on Docker. And these are the technologies people are interested in using, people who aren't distro people. So we need to be there for them. Otherwise, they're going to pick something else. Yeah. And bundling is great. And one of the things that's really key and that most of us haven't done yet, NixOS does, which is really cool, is atomic upgrades. Install stuff somewhere, flip the bit. Suddenly, everything's updated at once. Instead of breaking halfway through the process because you have libraries that change and you have an upgrade. Everything that depends on the library is yet, and we've done that forever and we still haven't fixed it. But they did. And so it's not mandatory for us to succeed because they will. So what kinds of things do we actually need? We need to understand bundling. We need to be able to have some kind of insight into what's happening inside of those bundles so that we can integrate with them and understand what's there. And understand if there are security vulnerabilities in the 27th version of Zlib. Because that's what's happening out there, right? People are bundling in things like Zlib and like OpenSSL and just name your library and it's being bundled so many times you have no idea. That's not going to go away. But what we need to do as a response is to understand those bundles and integrate with those bundles and integrate with the language level package managers. And integrate with things like Docker and things like Vagrant. And some distros are doing some of that, but almost nobody's doing all of it. So our data and ways that people expect it to work, if you want to get more developers, it's not necessary that your community stay healthy or that it grow. It's fine if it shrinks. Nobody else is going to care. I lived it. You know, integrate with configuration management. Why not? It's a better way to deal with stuff. We're still not doing it. We're still in this world of pets where that server over there is named pretty. And you carefully maintain it and groom it and configure it and spend... I used to run Fluxbox a long time ago. And how many hours do you think I spent configuring my menus? You have no idea. And then it turns out I decided I had other things I have to do behind... Besides screw around with my desktop environment, so I just installed GNOME and forgot about it. And now I run two applications, a terminal and a web browser. And I don't care about what else is happening over there. And so there's this difference in worldview between, you know, the developers running us and us, and they're code-centric and we're system-centric. We need to think about what's in the middle. It's applications. And that's where we need to meet. Just thinking about the applications that people want to use and run and live there instead of living at a systems level. And then maybe we'll all get along. So I'm about five or ten minutes short, which is great because I hope I've generated some discussion. Thanks for your time. And you can hit me up on IRC or Twitter or one of my 20 different email addresses or wherever you like. We can take questions too. Yeah, yeah, questions now as well, of course. Thank you. Raise your hand if you have a question. I'll bring you the mic. Anybody? Yes. Wait for Johnny. We need it for the video, I think. Video. So how much time have you spent looking at tools like Juju and OpenStackHeat because they're designed to solve pretty much exactly this problem? Well, yeah, but Juju is, and I don't know, does it run anywhere but you want to right now? Okay, then it's not going to solve this problem because people in the real world have to run and deal with multiple distros. That's true. So maybe it can solve that problem. And I, you know, one of the things that drives me nuts about software in general is NIH syndrome. And, you know, I'm sure it would not be a huge amount of effort to add multi-distro support to Juju. Probably the biggest issue would be getting Mark Shudderworth to say it's okay. So take a look at OpenStackHeat. It's not distro specific. I can actually pull it up right now. So one of the difficulties of packaging software that comes from multiple sources is where to place files. And what are your thoughts on the file system hierarchy standard and what distros should be doing about abiding to it or rethinking it? I think FHS is great. And I like it. And it's another one of those things that where, you know, at some point, roughly 10 years ago, people decided it was good enough and then it was mostly left to fall by the wayside. It's kind of getting picked up again. It's useful to have these kind of things like consistency. It's actually surprisingly hard to parse and figure out where you're supposed to put something even after reading it. And in fact, that's part of our equipment quiz for new developers is figuring out which subdirectory of VAR are you actually supposed to put certain things in. And it's not as clear as you would like it to be. But it's definitely something that's useful. At Gen2, we obey it insofar as we feel like it makes sense or is up to date with current standards. That's the problem with these things that got written 10 years ago and have had very few changes since then. I know it's, like I said, I think it's been picked up again. I hope it goes somewhere because it's great. But you know what's going to happen with it is nobody who's not a distro developer will ever read it. And that's the problem. We need to build tools that encourage the right behavior rather than writing long documents that tell you the right behavior. Some of this that I'm hearing is really developers are the problem. I mean... But no, developers are never the problem. Users are never the problem. It's always your fault. It's always my fault. But we're doing it wrong. I hate to use the devil's advocate phrase, but a lot of the things that developers don't want to do is because they want to treat your operating system as their own little playground and assume that nothing else but their application exists. Which is one of the things that is problematic. Yeah, but I mean, you know, the fact of the matter is these are the users. They're going to do whatever they feel like doing. It doesn't matter what we want them to do. They're just going to go on and keep running Mongo and losing data or whatever. Well, one of the problems that we have is that there's actually... There are two completely different, in many cases, categories of users. And that's blurring a little with some of the DevOps stuff, but you've got the users who are the developers who want to build with all this latest technology. They want to do whatever is the native package management for that system and they want to get it out there quickly. But then you also have the other side of the equation, which is the deployers. And they are almost entirely distinct in the real world right now. You've got... You should... Okay, before we keep going with this, I'm sorry to interrupt. I have a slide deck on SlideShare about this exact topic I gave this talk three days ago in London. So go check it out. I will. But my point is that in the real world, you have these cool technologies that you want to be able to use, but if you want to put them into a business where it's going to be the difference between making a million dollars and losing a million dollars in stock trade, for example, a fly-by-night bundled package with known vulnerabilities is not an acceptable deployment scenario. And so that's a place where the distros are much better at it, but we can't convince the developers that they're not producing something that can be deployed cleanly. Right. Yeah, I agree. Good bye. Thank you. Hi, so I think it touches slowly on the last one. As a developer, do I care about my security updates, find packaging and save Docker? So I ship my code in this little bubble and it's got a bunch of dependencies within that bubble. And as you say, there's security issue with OpenSSL. I don't want my end users to be caught out by that. They're data exposed just because I didn't care about package management. So who as a developer would I deal with that? Right. And that's kind of an interesting quandary is that people care, but what's been happening is that a lot more of the effort has been thrust upon the users rather than the creators of the software. I mean, there's a lot of upstreams nowadays that aren't even bothering to issue security releases when there are security issues. They'll commit them and push out a release, but there won't be any sort of working with search or what have you to do in this stuff. You have time for one more. How can meta distributions help to fill those containers? I think it's, you know, through some of the things that I brought up, which is, you know, and this meta distributions thing, that's been sort of Gentoo's thing for a long time. So I'm not sure if that question is specifically asked for that reason. Thank you for that great marketing opportunity. But no, it's, yeah, people are looking for a certain level of flexibility and they want to do different things with the containers versus outside of the containers. And, you know, most of the distors are not good at being installed inside of a container right now. And so, you know, one of the things that Gentoo and that a number of OSX package managers do is let you install things into a prefix where you don't need the full system. You just need a certain level of it. And so you can have package management even on a system that doesn't care. I think Debian, one of the Debian BSD things might do the same thing. I'm not sure. But just being able to, you know, have a user who has package management in their home directory. And that's, that's an important direction. Not because that's an interesting use case in itself, but because it deals with other use cases like, like the Docker thing. So you can get some level of package management within every single container. It could help for security updates too. Yeah, I agree. So you should all run Gentoo. Yay. All right. Thanks for coming everybody. I really appreciate it. Sure. I have to find some money first.
 Alright, are we good from the AV crew? Perfect. Thanks.  I apologize for my voice. It was a great night last night at the beer event.  And I got here about five minutes ago after scrambling to catch the last bus that we've been getting here in time.  So I won't have any references to earlier talks from this morning because I was sleeping.  And probably too drunk to get to talk anyway, so that was good.  It's great to be back. I love coming to FOSM every year.  And I've been doing gentey stuff for a long time now. I started...  I became a gentey developer. I think it was in June of 2003.  So I've been doing this just for a long time. I've been thinking about it for a long time.  And now, in the past couple of years, I was fortunate enough to become an employee of that company  that lets me do research and think about this stuff full-time.  So now I work at Red Monk. My job is called the industry analyst, which nobody really knows what that means.  But what it means to me is that I could spend all of my time doing research about the state of the art and software development  and how that compares to what people are actually doing, where they actually are.  And now I'm able to sort of bring that back to my initial love and my initial open source love of Gentoo.  And say, look, where is Gentoo? But not just that.  More importantly, where is everybody today in the distro world?  And where do we need to go? And why are so many developers running Macs instead of Linux?  And part of that is just because the experience sucks.  So we're going to just sort of walk through that today.  And Joe is going to see some of these slides for the third time, although I'm saying something completely different.  Because I've given three talks in the past four days.  But it will not be the same content. I promise you that.  So some of the questions, and I'm trying to avoid bullet points, but sometimes I just want to start with a few of them.  The key questions for me when thinking about, all right, where are distros today? Where do they need to go?  Is how are actual software developers in the real world doing things?  And how could distros do a better job of catering to them?  How are users doing things? How could distros better cater to them?  Because there's all of these things happening out there that we, as distros, and I'm curious, how many of you are volunteers for a Linux distribution?  Just raise your hand. Excellent. Perfect.  How many of you think your distribution has solved all of the difficult problems?  Again, good. So you're here for a reason. You want to think about this stuff.  And feel free to catch me after, and we can have beers and talk about it even more.  But the things I want to go after today is, where are developers? Where are users today?  Where do they need to be? And what do we need to do to build the right kinds of things for them?  And less importantly, down at the bottom, is what do we want?  It's great to build a distro that's friendly for new maintainers to get into the distro,  and it's more important to build one that people actually want to run.  If you don't have people who want to run the thing, you're never going to get anybody who wants to become a maintainer,  because it's this whole funnel thinking about things like nobody becomes a maintainer magically without being a user first.  So we need to build software that users and that developers want to run.  And that's not happening in the way it has to today.  I'm going to start with a little bit of data, and I apologize for the black and white, but I have no excuse.  I did a little bit of data mining across a few different data sources, and you may not know it,  but when you talk in places like Hacker News and do Google searches, I'm searching for you what you did.  I am looking behind the scenes and trying to understand what are people doing and why.  And some of this data is more reliable than others, so if you don't like one particular data source, that's fine.  What makes it interesting is the trends across multiple data sources, and that's also what makes it more reliable to me.  So I can look at this stuff and say, okay, Google Trends, fine, if you don't like that, that's okay.  But you look at the trends across Google Trends, across Hacker News, across Stack Overflow, for example, across GitHub.  I've even done some mining of IRC logs that I've had on my machine, because I've been doing this stuff for long enough  that there's some data in there, and you can do some fun things.  And then across, in this example, one of the others is Linux Journal, where they actually ran a survey of what distro are you running.  And what you see is not really surprising, there's a ton of Ubuntu users, just loads of them.  But what is interesting is Arch, right?  And so many pointed out on Twitter, I think it was last night or this morning, I don't think there are any Arch talks here.  How many of you are Arch users?  So what's going on with that, right?  And it's one of those things where you've got this set of traditional distros, the set of new distros,  and even the new ones, I don't think, have solved the problems that need to be solved.  And that's what we're going to talk about today.  But if you look at this stuff, you see some interesting trends emerging.  In terms of, if you think about this and try and build some intuition about what are these people like who are using these different data sources.  At least to me, I think, okay, Google Trends, that's just normal people.  At least insofar as it can be normal and be running Linux at the same time.  Now, Linux Journal, alright, these people are interested in technology, they're a little bit more technically savvy.  Hacker News, they're actual developers, they're not just users and enthusiasts.  And so that was sort of my intuition with this.  And what's interesting about looking at this data is that intuition holds true.  You get trends that sort of make sense when you think about where are the developers, where are the enthusiasts, and where are the more normal people running Linux.  And I don't think any of that last group is here today.  So we're just going to look at data from them and talk about them without seeing the rarely observed normal Linux user.  But if you look at this stuff, you see some trends that make sense.  You see things like, wow, there's so many developers running Arch.  But in terms of enthusiasts and normal users, there's a little bit less.  Whereas you go look at Mint, and I'd love to make fun of Mint just for, well, we won't get into that today.  But in terms of normal people running Mint, there's loads of them.  People love Mint, it's super easy, it just tells you, like, here's what you need to do, and we're going to do it for you.  And you don't need to think about that, and you don't need flexibility.  So, how did you, maybe, so you could get Mint like as a copy?  Right. I think all of these searches included Linux.  So it was Mint, Linux, Arch, Linux, Debbie, and Linux will have you.  And not GNU, Linux, unfortunately.  So if RMS is here, I'm sorry for that.  Yeah, so there's ways to do searches such that you can keep out some of the stuff you don't want to happen, although it's never perfect.  Like, for example, searching for Arch gives you all of the architecture stuff related to Linux as well.  You have to deal with it and do some controls to see.  You know, if you actually look through the results, you realize that after you look through 100 of them,  less than one of them talks about architecture, as in the other 99.9 talk about Arch.  But it's interesting to look at these things and say, like, oh, this is where things are in the closest sense we can get to real data.  Because almost nobody actually installs Popcon or something like it.  We don't know how many Linux users there are.  We don't even know what the breakdown looks like across different distros.  This is about as close as we can get to that sort of thing.  Where you look at multiple data sources, you build this list of, okay, it's pretty consistent across a lot of them.  The only one where there's a great deal of confusion is Mint.  I don't know. Is this number more right or is that number more right?  But in almost every other case, it's fairly consistent and consistent with at least my intuition of so many people used Ubuntu.  And unfortunately for me, not so many use Gen2 anymore.  But 2004 was a great year for us.  And you can look at this a little bit differently and say, okay, if Hacker News is the zero point,  what are the other data sources look like in comparison?  And again, you see things work out sort of in the way you would expect.  Some things are much more biased toward developers.  Some are more biased toward normal users.  And some of them, what makes it really weird is some of them, like Arch is a great example,  apparently enthusiasts don't like Arch, but everybody else does.  So people who just enjoy playing around with stuff, I don't know.  I'm not sure how to explain that, but most of this stuff does actually make sense when you think about it.  But I want to get away from the data and start talking about ideas.  And if you look at just trends over time without any axis to show you volume, just look at raw trends over time,  you can get some very interesting ideas of which way things are going.  And I'm not saying by any means that there's more people running this or that,  but which way are things going, what do people care about enough to talk about it?  And I put in red everything that's going up or remaining stable, and in gray is everything else.  And in fact, it's interesting to me, in part because there's two new distros,  we're going to call them new, even though they're not that new.  Two of the newer distros are going up.  And then sentos is actually pretty stable, which is interesting to me.  Yeah.  I'm sorry?  The scale?  Oh, the time scale.  This is since 2002, 2003, so it's about the last 10 years or so,  which is why it basically doesn't exist until more recently.  In art you can see some of the noise that I was talking about,  where people are just talking about architectures and Linux back then,  and then it picks up.  And the same thing with openSUSA, because it was SUSA,  until they decided to call it open and make it more open, I guess.  But it is of value to look at this thing and think about,  wow, people don't seem to care quite so much about things like,  well, Slackware, Gentoo, Fedora, Debian as they used to.  This doesn't mean they aren't running it, but they are talking about it.  It's an assumption.  It's just underlies their lives at this point and they don't care.  And so that's one of the points of this talk,  is how do you keep the distro relevant?  How do you keep people caring about it?  And one of those things is you have to integrate with the way they want to work.  And distros kind of built this idea of, here's how package management should work.  We're just going to keep it that way forever,  instead of kind of moving forward with the times.  And that's created loads of issues.  And one of these trends that's happened over the past eight years or so is cloud.  How many of you have ever run anything in a cloud?  Right.  And did your distro do a good job of that?  Or were you running a Mac at the time, maybe?  Just integrating with this new style of development is something that distros  have not done a great job of doing.  You know, they don't, for the most part, do things like natively cluster,  like share configuration data.  Like, you know, this is a guy named Randy Bias,  started a company called Cloud Scaling.  He has this fantastic analogy of cattle versus pets,  where in a cloud world, all of the servers are cattle.  And if one gets sick, you just take it out back and you shoot it.  In a world we live in, the servers are pets.  They're carefully taken care of.  They're certainly not kicked, Joe.  So he made a joke about kicking a cat in the elevator with him the other day,  and I think he just about flipped.  But distros are still in that world of naming everything carefully and carefully  installing it, for the most part.  Although there's these new things that have come along,  and I tried, the reason I'm here late is because it took me 15 minutes  to get past the configuration management room.  And that's something people care deeply about,  and that distros, for the most part, don't handle well.  You know, there's some modules that are written to sort of interface with the distros  and run RPM or whatever behind the scenes,  but it's not native.  And at the distros side, we're like, oh, we don't really care about that.  I mean, sure, people can run that if they want to.  But, you know, for example, why don't we install all the distros  using puppet or using Chef?  Why don't we just set that on the machine,  and that's the way you do the package management?  Because that's the way people want to run the stuff.  And so they slap on their own package management on top after the fact,  and you've got this weird mishmash of things happening,  or, okay, they've got Cpan, they've got PyPy,  they've got, you know, they're installing gems,  and none of that stuff talks to each other.  And so you've got like 20 copies of the same thing in different versions,  and things break, and it's just not a great experience.  And that's the sort of configuration management thing.  That's the point, right?  Is that developers today, maybe not you,  but you're not the only developers I do research on,  are looking for an experience that uses configuration management.  And it's not today either.  I mean, even Gentoo's own infrastructure runs, well, still, CF Engine,  slowly moving forward to something a little more modern.  CF Engine's been around for more than 20 years now.  It was 1993 when CF Engine 1.0 came out.  And we're still not thinking about, in fact,  do any distros here, do you think you do a good job of integrating with config management?  Okay.  But that's the way people want to manage their systems,  because we're working at high scale, we're working with transient things,  we don't want to sit around installing things,  and, you know, we came up, I think Red Hot initially came up with the idea of Kickstart  quite a while ago, and we kind of did that, and then we stopped.  And we need to go farther.  We need to start integrating these things into the way,  not just the way we develop the distro, but the way users install the distro.  I mean, why don't you give people an option to upload  an encrypted repository configuration management, and we could store it for them?  It would be easy.  We already managed loads of weird servers doing all kinds of random things for a distro anyways.  Why not make it a better experience for the users?  When they destroy their install as they inevitably do, and I inevitably do,  we've got the data form, and it's ready to replicate.  And one great example of this, of how poorly we integrate with configuration management,  is Chef, which told the Debian folks,  you guys are packaging us in such an out-of-date and terrible way  that we don't even want to be in your repo anymore.  Just please take us out, because we're doing a better job of being up to date  and having all the packages that matter to us.  We're the domain experts.  Get us out of here.  And they did, which was great. They listened.  But that's a real problem, and it's a problem for a lot of reasons,  because, you know, as distros, we create all of these policies for packaging  and guidelines and code for a reason, and so everything works together,  and it's all happy.  And now we're in this weird sort of middle ground where if you've got  ISVs, software vendors, writing their own packaging,  it doesn't understand how to talk to the rest of the distro.  And it's in between, in this point where it depends on things in the distro.  Other things depend on it, but it doesn't integrate right.  And so you end up with all kinds of bugs that, well,  some of my colleagues would love to talk to you about for hours.  And so, you know, we're in this weird place where, you know,  as distros, for the most part, we want to be stable,  but software like Chef, especially software that's sort of cloud-like  or DevOps-like, you know, there's a lot of innovation happening very quickly.  How do you keep up with the pace of that?  Because users want to run the newest software.  They want the latest features, but at the same time,  how do we put out something that's stable?  And this is something I don't think anybody's really solved yet.  You know, some people are taking an effort, like, if you talk to the Red Hat folks,  they'll tell you about software collections where they're trying to put, you know,  newer software for developers on top of RHEL,  but it's not in a place where, you know, that's a normal behavior yet,  and it's not in a place that users and the developers running the distros are looking for.  And in all honesty, I'm not sure of the right answer for this question,  but it's a question that you have to think about and decide what is the right answer for you  so you understand what you're optimizing for and what you're giving up.  I am curious, how many distros actually have continuous integration setup for all your packages?  Which one?  Okay, so for the core packages.  But almost none of them, but that's the way that, you know,  if you're a professional software engineer working at a modern company,  which most of them are not, you have CI setup and every commit is tested and built.  But we're not doing that with distros and we're not doing things like,  how many of you have heard of continuous delivery or continuous deployment?  Excellent.  Are you doing it?  Wow, awesome.  Great job, guys.  That's a good 25% of you who are doing continuous delivery.  But most of us aren't doing it with a distro.  And this is one point where I am actually very proud of Genti because we've been shipping software to users  within an hour for the past 10 years.  But we're not actually testing it.  The users are the testers.  And that's why they're called betas, right?  But we need to start delivering software continuously and tested so that they can get the modern stuff,  but it also works and we're not.  And so we've got, you know, all of these different package managers all over the place.  Every language has its own.  Some languages have like three or four.  JavaScript is a great example.  I mean, there's NPM, there's Bower, there's Jam.  I mean, how many different JavaScript package managers do you actually need?  And I guess we haven't gotten to a large enough number to answer that question yet.  But none of them talk to each other.  And I mean, we've put some work in a Genti on trying to integrate this stuff and say,  hey, let's download stuff from Plap High and then generate a Genti eBuild with it.  And that's going part of the way toward the right answer.  But I think the answer that we need is that package managers need to natively understand this stuff.  Having a translator isn't good enough.  Because people are, you know, installing this stuff from all over the place.  And we're actually more modern developers using things like Node and Ruby on Rails  as much as we don't want to talk about that here because we're all Libre and we love Copy Left.  That's where people are and that's what we need to build for.  And we're not.  We're doing a terrible job of this stuff.  And so people end up with, you know, five different copies of some of the lower level things in Node,  for example, installed by five different package managers.  And it's just not an ideal situation because you never know what's using what  with all kinds of interesting issues.  And so what do we do about that?  And I think that integration is going to be the answer.  You've got normal developers today are using all kinds of crazy data stores.  For the most part, we're still storing package management data in flat files.  You know, why aren't we keeping it in a more queryable way, in a more reliable way  than just sticking stuff in a directory somewhere in a bunch of text files?  I mean, it gets corrupted all the time and it creates all kinds of problems.  And I know some distros actually do, but a lot of them don't.  And if you store things in a way that's more friendly and more transparent to the developers running the stuff,  you know, what happens?  It lowers the barrier for them to get involved because the distro runs in a way  that they are used to working.  And so they're more likely to become a distro developer, a distro contributor,  because it's friendly to the way they want to develop.  And now all of these things are packaged, but, you know, we don't rely on any of them.  And there is a point too, and I'm going to agree with the criticism you want to say,  that we shouldn't have to rely on this stuff.  We never want to get a Cassandra cluster up and running just to install a package.  Sure, you're right.  But at the same time, you have to optimize for something,  and I don't think that something should be, you know, text files at the expense of data consistency.  I'm not sure that's the right trade-off.  You know, what else is happening out there in the real world where we're not, for the most part,  except for the 25% of you doing continuous delivery,  people are doing all kinds of containerization, right?  I mean, virtualization isn't even interesting anymore.  Now it's got to be containerization.  It's got to be LXC, and, you know, front-ends on top of LXC,  or on top of VirtualBox in the case of that grant.  Everything is happening in containers, but none of us are doing anything with containers.  I mean, none of the distros, to my knowledge, understand how to work with containers  or install anything in containers with, I think, one exception, CoreOS,  which is built around Docker.  And that's, you know, one of the things, one of the case studies I'm going to bring up over and over,  if I have time and I don't just run out.  I'm halfway through. Cool.  I mean, we need to start taking the software that developers want to use and optimizing for that.  Instead of pretending distros are a thing that, like, we figured out where they should be in 1998  and just staying there, and that's what we have done, and it's really a shame.  And BenT has done a great job on the UI side for normal users,  but for developers and for technical people, you know, even they've kind of fallen short, I think.  So why, for example, aren't we able to install every single application  and application stack into its own container and just use, you know,  copy on way to cross containers to manage all the dependencies?  Why is it so tough to start a distro in the first place and just get it installed?  Why aren't we shipping a Vagrant box?  And that's the way people want to do it.  Do you know any of you who work on distros, do you ship a Vagrant image, a Vagrant box?  Which one?  Ubuntu. Excellent.  So we've got the vast majority covered.  Unfortunately for the rest of us, we're not Ubuntu.  I bet nobody does anything with Docker.  Oh, yeah?  I know they said we're going to support it.  Do they do?  Yeah, that's going to be a pressure last four months.  All right, because of OpenShift and Paz's are awesome and no ops and all that good stuff.  Can you tell me more?  What does Fedora do with Docker?  And I'll tell everybody else.  Yeah, I'm not the right person to answer that.  Okay.  In the Config Management room?  Oh, so nobody will be able to get into here, unfortunately.  But perhaps it's on video.  So we've got all these things happening out there in the real world.  And this is another great example.  Well, two examples I want to talk about.  One of them is Chef has this thing called omnibus packaging, where what they do is just bundle everything up and ship it.  Pretend the distro doesn't exist, basically, which means we have failed.  If they have to go that far, we have utterly failed in our purpose.  And our purpose should not just be, Gentoo is great, or Fedora is great, Ubuntu is great,  but Linux is great.  And people in the real world have to run stuff across multiple distros, but we're not talking to each other enough.  We come here once a month, once a year, and do it.  And then we go home and mostly don't.  I've been on this list at freestess.org called the Distributions List for a number of years.  I don't actually remember when the last post was.  So hopefully some of you will go home and subscribe to it and start collaborating a little bit more.  If that was the only thing that happened as a result of the stock, I'd be very happy.  So, you know, you've got Chef doing things, and Chef keeps coming up over and over.  They first pulled their packages and then they said, screw package managers entirely, we're just going to do this all ourselves.  But there's more and more companies going that way.  Chef isn't the first example, it's not the last example of just saying, and they say omnibus because it's got everything in it, right?  They just ship the entire application stack all the way down to and including Ruby itself.  So they're not even using the system Ruby installation.  We failed.  Another example is FPM.  Depending on who you talk to, it stands for various levels of profanity with the second to being package managers.  And that's one of those things where people in the real world who have to build packages and even want to integrate are having such a hard time with it  that they had to write their own solution to build packages because it's such a painful experience.  And so what FPM does is you unpack a tar ball, bail in and install it, and then run FPM and it generates a Debra and RPM for you.  Because people have such a hard time writing this stuff, we've made it too difficult.  We haven't thought enough about the barriers to entry to creating a package because we've been too focused on making the experience perfect for the experts.  And what else is happening now?  Well, we've got a lot of people who don't even bother making releases.  They don't even make a tag and get.  They just say, go clone it from GitHub and run it.  And this is a thing that I see people shaking their head.  I don't like it. You don't like it.  But it's happening and so we have to deal with it.  People are doing this stuff.  Again, because the experience is so painful, what are we optimizing for?  Well, it's not what developers want to do today.  We're not optimizing for use of entry into writing packages, into creating packages.  We're not optimizing for people who need to run stuff across multiple distros.  We're not optimizing for people who need to do configuration management or want to do configuration management and who are sick of editing text files and configuration files in Etsy.  But we need to because the distro is becoming irrelevant.  There's a lot of people who don't even care.  And in fact, it's...  No, I'm not... Am I in video? I guess I am.  It's actually kind of embarrassing now because Windows is doing a better job with some of this stuff.  Windows package management has finally reached the Windows world.  I just learned about this myself like six months ago.  These are Windows package managers.  They all work. They do stuff.  There are repositories.  And that wasn't the case a couple of years ago.  It was like, go install Sigwin and then you can do some kind of package management.  And you know the thing that Windows really has going for them is they really understand people who like to click GUIs.  They really deeply understand people who like to do points and click and getting a good user experience.  And once they're bringing that to the developers who are using this stuff,  those are good chance that they'll get this problem solved before we do in the Linux world.  And so, you know, what kinds of directions do we need to go?  And I put up a few examples here.  If you actually know the story behind them, you'll know that two of them are at some level based on Gen2 or its package manager.  I'm kind of proud of that that they're in the directions that I think everybody else needs to go.  The third one, I think there's actually talk on NixOS at some point.  I encourage all of you to go check it out because it's really, really interesting ideas in terms of where we need to go as distros.  And so, you know, we need things like, hey, everything is ephemeral.  You don't need to care about package managers or, you know, in the case of Chrome, the browser.  We've solved that for you so you don't need to think about it and we'll just keep all the data.  You can tell us everything.  But CoreOS is a very interesting idea.  It's super minimal system.  It's designed for the cloud.  It's built using Portage, which is Gen2's package manager, but nobody who knows, who runs it knows that.  Because inside of CoreOS, if you care about packages, you just use Docker.  And again, it's one of those things where the developers want to actually do.  They want to use things like Docker.  They don't want to care about package managers.  If you don't make a package manager a good experience for them, they're going to ignore it and go just search.  And go just start installing random stuff and using ChefAmdabus and, you know, picking a language level package manager, perhaps.  And now this isn't something you have to do.  It's not mandatory for distros to matter.  It's not mandatory for them to succeed.  If they die, it's okay. Nobody else is going to care.  But if you and I want to remain relevant, we have to keep up with these things that are happening.  And yeah, one of the reasons I put this text down here, I'm not sure if you can read it, but there's this key phrase here that says,  use Docker as a package manager to build and push your app.  Docker is not a package manager. It doesn't know anything about dependencies.  What you do is you write this Docker file script and in that script, you run something like yum install xyz.  And that's the concept of package management.  It's not the way we think about it, but it's the way people actually want to use it.  They don't care about package management. They don't really care about, even about things like eventual consistency, like configuration management does for you.  They just want to get stuff up and running.  And these, you go check out all of these if you haven't yet, because these are the kinds of ideas that people want today.  This coro s thing is just going nuts. People love it because it works in the way people want to work today.  It does things like, oh, and they use go, of course, and go is, I don't know if you've heard it yet, but it's the trendiest language ever since node.  They use go, they have this clustering demon using the raft algorithm that maintains consistency of configuration data across Etsy on every single installation.  It's based on Docker. And these are the technologies people are interested in using, people who aren't distro people.  So we need to be there for them. Otherwise, they're going to pick something else.  Yeah. And bundling is great.  And one of the things that's really key and that most of us haven't done yet, NixOS does, which is really cool, is atomic upgrades.  Install stuff somewhere, flip the bit. Suddenly, everything's updated at once.  Instead of breaking halfway through the process because you have libraries that change and you have an upgrade.  Everything that depends on the library is yet, and we've done that forever and we still haven't fixed it.  But they did. And so it's not mandatory for us to succeed because they will.  So what kinds of things do we actually need?  We need to understand bundling. We need to be able to have some kind of insight into what's happening inside of those bundles so that we can integrate with them and understand what's there.  And understand if there are security vulnerabilities in the 27th version of Zlib.  Because that's what's happening out there, right? People are bundling in things like Zlib and like OpenSSL and just name your library and it's being bundled so many times you have no idea.  That's not going to go away.  But what we need to do as a response is to understand those bundles and integrate with those bundles and integrate with the language level package managers.  And integrate with things like Docker and things like Vagrant.  And some distros are doing some of that, but almost nobody's doing all of it.  So our data and ways that people expect it to work, if you want to get more developers, it's not necessary that your community stay healthy or that it grow.  It's fine if it shrinks. Nobody else is going to care.  I lived it.  You know, integrate with configuration management. Why not? It's a better way to deal with stuff.  We're still not doing it. We're still in this world of pets where that server over there is named pretty.  And you carefully maintain it and groom it and configure it and spend...  I used to run Fluxbox a long time ago.  And how many hours do you think I spent configuring my menus?  You have no idea.  And then it turns out I decided I had other things I have to do behind...  Besides screw around with my desktop environment, so I just installed GNOME and forgot about it.  And now I run two applications, a terminal and a web browser.  And I don't care about what else is happening over there.  And so there's this difference in worldview between, you know, the developers running us and us, and they're code-centric and we're system-centric.  We need to think about what's in the middle. It's applications.  And that's where we need to meet.  Just thinking about the applications that people want to use and run and live there instead of living at a systems level.  And then maybe we'll all get along.  So I'm about five or ten minutes short, which is great because I hope I've generated some discussion.  Thanks for your time. And you can hit me up on IRC or Twitter or one of my 20 different email addresses or wherever you like.  We can take questions too.  Yeah, yeah, questions now as well, of course. Thank you.  Raise your hand if you have a question. I'll bring you the mic.  Anybody?  Yes.  Wait for Johnny. We need it for the video, I think.  Video.  So how much time have you spent looking at tools like Juju and OpenStackHeat because they're designed to solve pretty much exactly this problem?  Well, yeah, but Juju is, and I don't know, does it run anywhere but you want to right now?  Okay, then it's not going to solve this problem because people in the real world have to run and deal with multiple distros.  That's true.  So maybe it can solve that problem.  And I, you know, one of the things that drives me nuts about software in general is NIH syndrome.  And, you know, I'm sure it would not be a huge amount of effort to add multi-distro support to Juju.  Probably the biggest issue would be getting Mark Shudderworth to say it's okay.  So take a look at OpenStackHeat. It's not distro specific.  I can actually pull it up right now.  So one of the difficulties of packaging software that comes from multiple sources is where to place files.  And what are your thoughts on the file system hierarchy standard and what distros should be doing about abiding to it or rethinking it?  I think FHS is great.  And I like it. And it's another one of those things that where, you know, at some point, roughly 10 years ago, people decided it was good enough and then it was mostly left to fall by the wayside.  It's kind of getting picked up again. It's useful to have these kind of things like consistency.  It's actually surprisingly hard to parse and figure out where you're supposed to put something even after reading it.  And in fact, that's part of our equipment quiz for new developers is figuring out which subdirectory of VAR are you actually supposed to put certain things in.  And it's not as clear as you would like it to be.  But it's definitely something that's useful.  At Gen2, we obey it insofar as we feel like it makes sense or is up to date with current standards.  That's the problem with these things that got written 10 years ago and have had very few changes since then.  I know it's, like I said, I think it's been picked up again. I hope it goes somewhere because it's great.  But you know what's going to happen with it is nobody who's not a distro developer will ever read it.  And that's the problem. We need to build tools that encourage the right behavior rather than writing long documents that tell you the right behavior.  Some of this that I'm hearing is really developers are the problem.  I mean...  But no, developers are never the problem. Users are never the problem. It's always your fault. It's always my fault.  But we're doing it wrong.  I hate to use the devil's advocate phrase, but a lot of the things that developers don't want to do is because they want to treat your operating system as their own little playground and assume that nothing else but their application exists.  Which is one of the things that is problematic.  Yeah, but I mean, you know, the fact of the matter is these are the users. They're going to do whatever they feel like doing.  It doesn't matter what we want them to do. They're just going to go on and keep running Mongo and losing data or whatever.  Well, one of the problems that we have is that there's actually...  There are two completely different, in many cases, categories of users.  And that's blurring a little with some of the DevOps stuff, but you've got the users who are the developers who want to build with all this latest technology.  They want to do whatever is the native package management for that system and they want to get it out there quickly.  But then you also have the other side of the equation, which is the deployers.  And they are almost entirely distinct in the real world right now.  You've got...  You should...  Okay, before we keep going with this, I'm sorry to interrupt.  I have a slide deck on SlideShare about this exact topic I gave this talk three days ago in London.  So go check it out.  I will.  But my point is that in the real world, you have these cool technologies that you want to be able to use,  but if you want to put them into a business where it's going to be the difference between making a million dollars and losing a million dollars in stock trade, for example,  a fly-by-night bundled package with known vulnerabilities is not an acceptable deployment scenario.  And so that's a place where the distros are much better at it, but we can't convince the developers that they're not producing something that can be deployed cleanly.  Right.  Yeah, I agree.  Good bye.  Thank you.  Hi, so I think it touches slowly on the last one.  As a developer, do I care about my security updates, find packaging and save Docker?  So I ship my code in this little bubble and it's got a bunch of dependencies within that bubble.  And as you say, there's security issue with OpenSSL. I don't want my end users to be caught out by that.  They're data exposed just because I didn't care about package management.  So who as a developer would I deal with that?  Right.  And that's kind of an interesting quandary is that people care, but what's been happening is that a lot more of the effort has been thrust upon the users rather than the creators of the software.  I mean, there's a lot of upstreams nowadays that aren't even bothering to issue security releases when there are security issues.  They'll commit them and push out a release, but there won't be any sort of working with search or what have you to do in this stuff.  You have time for one more.  How can meta distributions help to fill those containers?  I think it's, you know, through some of the things that I brought up, which is, you know, and this meta distributions thing, that's been sort of Gentoo's thing for a long time.  So I'm not sure if that question is specifically asked for that reason.  Thank you for that great marketing opportunity.  But no, it's, yeah, people are looking for a certain level of flexibility and they want to do different things with the containers versus outside of the containers.  And, you know, most of the distors are not good at being installed inside of a container right now.  And so, you know, one of the things that Gentoo and that a number of OSX package managers do is let you install things into a prefix where you don't need the full system.  You just need a certain level of it.  And so you can have package management even on a system that doesn't care.  I think Debian, one of the Debian BSD things might do the same thing.  I'm not sure.  But just being able to, you know, have a user who has package management in their home directory.  And that's, that's an important direction.  Not because that's an interesting use case in itself, but because it deals with other use cases like, like the Docker thing.  So you can get some level of package management within every single container.  It could help for security updates too.  Yeah, I agree.  So you should all run Gentoo. Yay.  All right. Thanks for coming everybody. I really appreciate it.  Sure. I have to find some money first.
_FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_0_155.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_156_371.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_372_767.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_768_911.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_912_1247.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_1248_1403.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_1404_1679.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_1680_2207.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_2208_2447.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_2448_2819.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_2820_2891.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_2892_3047.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_3048_3455.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_3456_3719.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_3720_4031.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_4032_4091.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_4092_4331.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_4332_4487.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_4488_4703.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_4704_5291.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_5292_5387.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_5388_5603.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_5604_5687.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_5688_5843.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_5844_6179.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_6180_6287.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_6288_6455.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_6456_6563.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_6564_6635.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_6636_6791.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_6792_6935.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_6936_7043.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_7044_8171.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_8172_8231.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_8232_8819.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_8820_8903.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_8904_8963.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_8964_9095.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_9096_10463.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_10464_10955.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_10956_11159.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_11160_11327.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_11328_11735.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_11736_11903.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_11904_12323.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_12324_15863.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_15864_65435.png _FOSDEM_2014__Is_distribution-level_package_management_obsolete__FENKd8NT9cY_65436_65916.png