Tumbleweed: A Story of Love and Worry
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Brown, Richard
Rolling Releases are the future of Linux distributions. They are already the better solution for power users & developers. Tumbleweed is the future of Rolling Releases. The methodologies, techniques, and capabilities of Tumbleweed are opening up new doors, creating possibilities, and disrupting existing technologies beyond its borders. This session will explain how and why openSUSE Tumbleweed is paving the way for that future, while already being "the reliable rolling release". The talk will dispel the fears, uncertainties and doubts that many have regarding rolling releases in general and Tumbleweed specifically, and share how you can get involved both using, and improving, this exciting fast moving foundation of the openSUSE Project. But not everything is perfect. This talk will also identify some rough edges in Tumbleweed and suggest collaborative solutions as to how the openSUSE Project could start addressing them, so we can continue the exceptional progress Tumbleweed has made into the future and beyond the year 2020.
 I'm now the after lunch slot, so I hope don't mind if people start trickling in, but I'm going to get things started. So I'm Richard Brown, Chairman of the OpenSuser project, and I'm here to talk to you today about tumbleweed, why I think it is the best thing since sliced bread, why I'm most excited about this and anything else we're doing in OpenSuser, why I think rolling releases are the future of Linux distributions, and some of the bits and pieces where I don't think we're doing everything that we could be doing to make it as smooth as it possibly could be. Because, you know, we're not perfect, we're just great. To talk about rolling releases, I really have to start at the beginning. We have to explain where Linux comes from, what is a distribution, and when we're talking about Linux distributions or traditional distributions, we are talking about regular releases. It's what most Linux distributions follow. It's a model where you collect all of your different upstream packages, you put them all together, you make a cohesive operating system as a distribution, and you're releasing it every X years or months. Depends on your users and your use case of how often that is. Community distributions generally favor slightly faster release schedules, so distributions like Fedora, Ubuntu, or the old OpenSuser would be every six to 12 months, and then, of course, you have things like enterprise distributions where the new major release of an enterprise distribution will be several years away. Once that release is out, once users can download that software and start using it, the general model of a traditional distribution is to not dramatically change the software within that. You know, being very, very conservative from that point, and only very reluctantly upgrading, very reluctantly patching those things you need to patch to keep the operating system working, but you don't want to introduce unexpected changes, you don't want to break anything, so, you know, very, very reluctantly doing that, generally freezing anything, which means when you look at sort of the big wide open source world and everything else going on and packages elsewhere, upstream projects elsewhere, the only choice to maintain a regular release is with heavy use of back porting, you know, so taking patches and fixes from the upstream project and putting them into your stable regular release one. And like I said, this is the traditional model, it's, you know, followed by Debian, Fedora, open-suzer-leap follows it as well, and yeah, Ubuntu. But developing these is tricky. You need something to start this. How do you develop a regular release? And most other Linux distributions rely on a development branch to do this. We used to, ours was called factory, but, you know, other distributions have things like Debian CID or Fedora Vorhide or Ubuntu have something that they kind of call dailies, but it never seems to work. But it's where your developers of a distribution should be using actively putting in their various upstream packages to constantly give you a rolling picture of, you know, where is your code base, what is your next regular release going to look like? Everything's ever frozen in there, it's always moving, and it's almost always broken. This is true of every single one out there, every single development branch, it's typically broken. And this is problematic because developers need their system to be as close as possible to the upstreams they're working on. You know, they need to be able to see where is, you know, both where is their particular part they're used to and everything else around them. They need to be able to see how it's all working. Dev branches accomplish that, but they're completely and utterly unstable and unusable. So you've got this sort of nice deadlock problem of, you know, how do you actually then get a good picture of what's really going on? What generally happens is developers stop using their dev branch, apart from when they really, really have to, to comply with some process somewhere. That means you have very narrow attention being paid to what's actually going on in all those upstreams that you're relying on to build your distribution. So it doesn't work for developers. They just move on and find some other way of hacking together their packages to get into the main regular releases. And that becomes a really big problem as a distribution project. So traditional open SUSE as a distribution project, we used to do this, we used to have factory and as your users and contributors of your dev branch decline, your entire project starts getting slower. There's no doubt about it. You get less indications of bugs before a regular release. You get less new features in your regular releases, you know, less innovation there because nobody knows what's going on. So even the most ambitious features they think of are relatively narrow things compared to what they could possibly be getting if they had a broader view of where the world is. You end up with over time, in particular, sort of increased technical debt stuff like that is lingering around in your dev branch for ages and no one ever fixes it, which then means when you do eventually have a regular release and you do eventually need to fix it, clearing that technical debt makes that release more and more expensive. It's more and more hard work. It's more and more work to get the community involved in doing it. And yeah, it just really, really starts holding the entire project back. It's not just a problem for the distributions trying to get this software into the hands of users, though. It's a problem for upstream developers also because every upstream project, especially in this day and age, wants to get their hands in the, you know, want to get this software in the hands of users as fast as humanly possible. Dev branches technically accomplish this, but they're useless really because no user's going to be using them. Regular releases don't accomplish it. Whatever schedule, whatever distribution picks, it's going to be too slow for that goal of getting it in the hands of users quickly. And then containerized apps, things like App Image, Flap Pack, Snappy, you know, promised to solve this, but it's not quite that easy. You know, they'll make it there one day, and if you want to hear more about my rant about that, you can come see my talk on Sunday at three o'clock in the other room. But yeah, the problem's not ideal there. And then when you start looking at users, and particularly enthusiastic Linux users, or users, you know, the kind of core part of the community that are interested in open source because it's open source, they also want to have that software as fast as possible. They don't want to wait, but when they get it, they want to make sure it works. So Dev branches don't work there either. And there's also the sort of second thing. Users want a consistent experience. They want it to feel like it's well put together, that, you know, collectively themed looks to the right, that feels right. The UX works properly. These are key requirements that users have. And this is another problem, actually, that a lot of these containerized apps are starting to bump into. You know, they're getting out there and running these things, but, you know, getting that feeling of this is a consistently built and a consistently engineered solution just doesn't work with either the regular release model, where it's always too slow, or the development branch model where everything's just moving too fast and breaking. And these are the people we've got to capture, because these are the people that are going to be our contributors in the future. They're the ones who are enthusiastic, they're the ones who are looking at these upstreams, who are keen on what we're doing. We need to find a way of encouraging them to use this software and, you know, get enamored by it so we can then start having them help maintain it and make things even quicker and faster. Rolling releases are the answer to these problems. But what is a rolling release? Well, in basic terms, a rolling release is a Linux distribution without a release schedule. No version numbers, no point releases, no milestone dates, frequently updating all of the packages in the operating system whenever they're ready. So you can just download it, start using it, and you're always going to get the latest ready, stable version of everything. There's other examples, of course, I'm talking about Tumbleweed, but there's sort of two other main distributions advocating this model and really pushing it, is Gentoo and Arch, where, you know, quite popular, especially in that enthusiastic user base area where you have people downloading it and getting the latest of everything in there. And when I talk to people about rolling releases, I always hear the same three complaints or the same three excuses of why they don't like a rolling release. There's a perception that they're unstable. There's a perception that they're unreliable, which is subtly different from being unstable, and they're a perception that they're hard to live with. In the case of unstable, in this case, I'm talking specifically about it's always changing. The way I used my system yesterday is now different from the way I have to use my system today. A fast moving code base is going to include changes. That is kind of part of the point. So some of the, you know, there is always going to be a little bit of a change in there. The question sometimes is how fast, and having different rolling releases at different paces is something that I think the world needs to start thinking about. But to really solve this problem, you need to be making sure that your rolling release is building everything, testing everything, and then integrating everything in a consistent cohesive fashion constantly. And then when that is then delivered to users, delivered in a way that those behavioral changes, that sudden new way that that new application is behaving, doesn't get in the way of the work they need to do that day. It's changed. They're going to have to learn it at some point, but you don't want it to block that work when they need to get their work done. That's somewhat different from the problem of unreliability, of the perception that rolling releases are going to break. It's a fair challenge. A lot of rolling distributions have this problem. It is thousands of moving parts from thousands of different upstream projects, and the distribution has to find some way of them getting it all working together. Just like we're solving the stability problem, you've got to build it consistently. You have to test it consistently, and you have to integrate it consistently. But speaking from experience, it isn't just a case of testing it before you ship it, but actually testing at the point of submission. Finding as early as possible when someone is contributing to a rolling code base, does this work? Will this break the entire build? Will this ruin everything? Testing it there really, really early, getting fast feedback, helps both with the kind of contribution engagement and use. And then, of course, you have a second shield of it. Testing is a whole. Because you can't just think of a Linux distribution like a collection of packages. This is the fatal flaw, which I think so many other distributions get wrong, and we get right, is we think of our distribution like a cohesive single thing that we ship. And we try and make sure that it all works together in one bit. And we're not falling into the trap of distributions like Arch, where, oh, we're shipping this wonderful library right on time, and they're forgetting that about the 20 other things it needs or should go with it aren't there, aren't integrated. And ultimately, the goal there is to make sure that you don't ship something that doesn't work. Talking about testing, most of the distributions, rolling and regular, and formerly us in the past, rely on passive testing with their community distributions. The idea of upstream has released something, we've packaged it, we've thrown it in some testing branch somewhere or something like that. And then we wait a bunch of days, and we just trust that someone in the community is going to look at it and play with it, and then it's good enough we ship. No one ever actually checked, did anyone actually tested it? They just trusted that no one filed enough bugs, it must be fine, so we shipped it. The model works to some degree. The bigger the community is, the more chance you have of finding those bugs quickly enough, and that testing window works out okay. But it's still Russian roulette. At some point, your users are going to get shot by something that slipped by that approach. Passive testing just does not work for distributions. You need to have active testing. You need to have proactively confirming, does this new package in this distribution break something just on its own? Does it work at all? Did the developer completely screw it up? Or does it work when you install it with a context of 20, 30, or 4,000 other packages? And does it change, or at least even if it's working technically speaking, does it change in a way that users don't expect? And you need to be able to answer that question in order to be able to integrate everything quickly and fast and deliver it to the users fast enough before an upstream release has even made that change. So you need to have a way of knowing at least as fast as possible after an upstream check something new in, or really before they check it in. Does this change something? Does this break something? So we can get working on fixing it and shielding our users from those problems. And then the last problem, the hardest to live with it, but when looking at the other rolling releases, Arch, I really respect Arch. They have this mantra of the Arch way, which gets summed up as do-it-yourself, it's a learning exercise. Not my way of doing things, but it works. The Arch Wiki is a wonderful bit of documentation. And then of course you have the Gentoo way, which is basically the same, it just takes longer because you're compiling. But that works for guys at the bleeding edge really, really working on this upstream stuff. It doesn't work even for most enthusiastic users. We have too much other stuff to do. We don't want to be hacking around with the inner workings of our distribution. We want something that we can just install, work with, and get the latest of everything because we want to have our cake and eat it too. And we need to have some way of stopping this. Something's changed and I have to spend three days hacking around my system to fix it and get it working the way it's meant to work. And in OpenSuser, we've asked this question, why do rolling releases need to be difficult? And obviously with Tumbleweed, we think the answer is they don't. But Tumbleweed didn't start out as the Tumbleweed we now know and love. It started originally by Greg Co-Hartman in, well, before 2014. I can't actually find the exact date where it started. And as you know, it's providing the latest updates. The kind of key focus point there is at the pace of contribution. Tumbleweed runs as fast as our community makes it run. Sometimes that means incredibly quickly. Sometimes that means we actively decide to do things at a slower pace because we think that's the best way of handling what that upstream project is doing. It's tested by OpenQA. And in terms of that kind of user-based focus, we're really targeting that developer, contributor, enthusiast part of things. Because that's really where rolling releases really, really shine. But old Tumbleweed wasn't like that at all. Tumbleweed originally started as a model of taking the base system of OpenSuser we were shipping at the time and putting rolling updates on top of that. So instead of having a separate release, it was really an add-on for an existing release. It had a very particular focus. Obviously, Greg Co-Hartman's a kernel hacker, so he started with the kernel and then the community started building up on that and things like KDE and GNOME and some applications got in. But that model of sitting the two meant that the only way of delivering software would overwrite the packages from the original base system, which meant every time we released a new base system, a new version of OpenSuser, your only choice was resetting to zero everything in Tumbleweed, which was a really dramatic change because any customizations that had been sitting in the Tumbleweed bit suddenly disappeared and vanished. Sometimes had their packages roll backwards or just with different config or yeah, it was always a painful mess. That wasn't the only lesson we really learned from that. I think the key lesson we learned was partially rolling distributions don't really work, not in a general sense. In a very narrow sense, in very specific, you know, small narrow use case with small narrow changes, I think you can make it work. But in terms of a general broad general purpose distribution, it just constantly fails because that rolling top constantly needs new requirements from that stable base and you can't change your stable base, it's stable. So you end up having to come around with nasty little hacks, you end up tinkering with the stable base, you end up linking stuff in weird and wonderful ways and it just falls apart every single time and even if it didn't, you still got that reset to zero every eight months with a new release, which is brutally disruptive for users. And I kind of sum up this lesson as my rolling release rule, which I think Tumbleweed really solved better than anything else. To be able to move a Linux distribution where you want to be able to move any one thing in there quickly, you know, you've got this massive code base and you want to move some weird library on the far end of something, you've got to have your tools, your processes, the technology in place where you can change everything. Just be prepared to throw out the entire distribution and start again just to get that one new library in there. And we've done that with Tumbleweed, not just on its own because we've done that with the tools we have. We couldn't do this without the build service. It's a key part of everything we're doing, we just had a presentation about it earlier, so I won't go on about it again. But the build service, the way it works, the fact that we can track all of these different dependencies in all these different locations, link them all together, rebuild them when they're needed, it makes sure that you have that consistent view of the distribution built quesively together all of the time. But building is fine. You need to make sure it works as well. And of course, we have OpenQA. Originally started 2009 for testing the basic installation part of OpenSuser. And it's become now an absolutely key part of the Tumbleweed release process. A single Tumbleweed update doesn't happen until it's been tested by OpenQA. It's also a key part of the leap process. It's now also a key part of the SLEZ development process. And it's even used by Red Hat for testing for Dora. I haven't got them using it for Red Hat, but working on that one. Hopefully one day. I'd like to steal their tests. But with OpenQA, you get these nice dashboards of all these different scenarios. So we're not just testing one basic, boring use case, but different ways of installing the distribution, different raid configurations, different desktop environments, different architectures, although that isn't shown on the screen here. In a deeper view, the tests break down to exactly the steps being done by the test. And the kind of key part here is it's not testing artificially and just poking around some APIs or calling very particular scripts to do very particular things. The tests are written in a way to actually test the software the same way a user is going to use it. So when you're talking about that kind of conceptual problem of making sure that your software, you know, you're aware when your software is changing in a way that a user might be impacted, that's exactly how OpenQA is testing this. So it can be a trivial change like a wallpaper or a login screen where we've changed the color. OpenQA at least makes us aware that that's happened so it flags it up so we can decide is this the right thing we want or not. If it is, we click next. It moves on. Everything's fine. And that is all then tied together into the factory development process where we have this kind of pipeline of code submissions going to tumbleweed, initially getting automatically reviewed in the build service, then getting tested in a process we call staging where we make sure in isolation does this one little thing work on its own? If it works fine on its own, that's the point when we then start involving humans and actually have someone looking at it and doing a proper review of the submission, does this actually work? At that point, it is then put into factory. So you have the sort of full, large, big code base, 10,000 plus packages all built together, all consistently integrated in one big pool, which we then test again in OpenQA in a much more intensive fashion. And that then gets done and pumped out at the end as tumbleweeding. That means if you want to change something in tumbleweed, you've basically got two very easy ways of doing that. One, just contribute to OpenQA. Writing a test to make sure that OpenQA is checking that one thing you really care about means that every single snapshot of every single tumbleweed will make sure that it behaves the way you want it to behave about. So you don't really even have to worry about packaging anything or coding anything. If you can just describe in an OpenQA test what you want to make sure stays that way, it will stay that way. Or at least it will stay that way until it breaks and then we'll figure out the best way of getting around the problem. But at least means we're aware that that use case has changed. Or if you're more packaging aware, the factory submission process obviously contribute to factory. That is how tumbleweed works. When I talk about all these toolings and processes, especially to upstream developers or new people to the project, I always get the same responses. That's cool. It's great. You're doing all this stuff fast. But I don't want to wait for that build or the test nonsense. I'm an upstream developer. I've just got my tarball. How can I run it really, really quickly? I don't want to wait for all this testing and stuff. Happens a lot with especially these people embracing stuff like Snappy. The process works at a ridiculous pace. I just realized I still haven't fixed this slide. I got the number wrong on it. Nome 322 is an example. The upstream release of Nome came out and within less than 48 hours, we had it fully integrated, fully tested and shipped in tumbleweed, every single package. And it worked. We had very few bugs, very few issues, users universally happy. When we did have a few issues, they got fixed the next day. In the case of KDE Plasma 5.9, not 4.9, we even actually shipped it on the upstream release day. The upstream release process for KDE, we actually get the tarballs a few days earlier. So we were able to do all of that testing and pre-work in place there and then we just hit the button when it was ready and done straight out. If that's still too slow for you, thanks to the build service, we've got these kind of separate incubator style projects where we can have derivatives of tumbleweed testing straight from the get of these upstream projects. So things like Nome Next or OpenSUSE or Krypton, where every single commit from KDE or from Nome immediately spins out new versions, tumbleweed style, tested tumbleweed style, and they're right away. That's not nice to brag about two simple examples. It's not just a case of the specific stacks where we're interested in. Is Dominic here? I guess not, because he knows all this stuff anyway. But Dominic Lemberg, our release manager for tumbleweed, every week he does a report to the community of what's been going on in tumbleweed this week. A year ago now, he made this comment of it's been a quiet week. There wasn't really that much and the report was shorter than usual. And that got me curious of what is a quiet week for Dominic. And that week was this, three snapshots. That's three different software releases. Basically the equivalent of a point release of a regular distribution. Collectively all those releases put together. We've included 146 new package updates. It included a new kernel. He changed a whole bunch of stuff on the DVDs that we ship. That's quiet. It's ludicrous. It's an insane amount of change for one week. In fact, a couple of weeks later, it was twice as much. Last week, in fact, two weeks ago, it was three times as much as that. The pace is still accelerating. The process still works. It scales out because we have more and more people using it. The tooling works so we can do huge amounts of changes in a relatively short time, keep pace with all of that and still make sure we're shipping something to the users that actually works. That works from our perspective and it works from the perspective of what upstreams are trying to deliver. Users might have different opinions. But this is where we have BTIFS and Snapper. Because we've got BTIFS as our default file system, because we ship Snapper as our tool for taking a snapshot every time you do an update, that means the whole problem of something changed in a way I don't like is immediately immunized. You can roll down tumbleweed every single day and if you then find out it's not working the way I wanted it to, you can always roll back to yesterday's snapshot and just work from there. Get your job done. Even if we break your machine, not that it happens, but you can even do that from grub. Even when the system is booting, it just works. That's great. But what about the development? What about open-suser factory? What about the dev branch approach? Once we started putting tumbleweed together in this way as an end of rolling release, that became the next question. What do we do about a factory? We don't need a development branch. We don't have a development branch in the purest sense anymore. There is no crazy rolling untested head for someone to mess around with because tumbleweed is keeping up so well, we can give our developers something that actually works all the time. They're always going to be close enough to be able to know what the hell is going on and everything they're doing. They don't need to have a factory anymore. It's there still in terms of the process. The process is the factory process, but the output of that is tumbleweed. That's what people use. I've been talking about rolling releases all of this time. What about an open-suser regular release? It's a simple truth that a couple of years ago, a huge amount of our community were very much focused on the kind of concepts I've been talking about so far today. Rolling releases, delivering quickly, factory or tumbleweed, and the enthusiasm for a traditional rolling release was fading away. The cool thing is, because of tumbleweed, we've been able to do really exciting things with the regular release that we were too scared or blind to think about in the first place. With Leap, we're able to have that nice, stable, Susan X Enterprise codebase at that nice, stable, regular release, the polar opposite of tumbleweed, the completely different use case for completely different people, and appealing also to completely different contributors. Then we can still, because we're using the build service, because we're using OpenQA, because we have all of these tools and techniques that we've been doing for years, take parts from tumbleweed that make sense, layer them on top, but still do so in a process that you're not having this nasty issue of rolling and stable breaking, and ship a nice, cohesively tested loop. In the past, OpenSuser used to always seem to be split. We had a community where some people wanted us to go faster, some wanted us to go slower. We can do both now. Tumbleweed is the fast road. Leap is a stable one. They both serve different users. They both work perfectly fine. They both actually help each other quite well, because it also helps that Slee is based on what we're doing in tumbleweed. All Slee engineers are developing on tumbleweed, and then it ends up filtering into both leap and Slee from that point of view. Just like that slide there. From a cross-section of the whole thing, you end up with a pitch like this, where obviously tumbleweed is over 8,000 packages. It's actually over 10,000 packages now. As a rolling base system, it's own unique code base, and then leap and Slee sharing a shared core and overlapping between the two. This is how things are developed. I really should have updated the slides, because I've just noticed the numbers wrong. Tumbleweed is rolling along constantly, constantly changing at its own pace. Then next year, like we already announced, there will be a new code base to replace the current one we are using for Slee 12, Slee 3, and leap 42. There will be core 15, not 13, and there will be Slee 15 and leap 15, all originating from what we're doing in tumbleweed, being frozen from there, polished up, tightened up, et cetera. It's great. It's wonderful. It's great from a rolling release perspective of getting things in the hands of users quickly and working with upstreams fast. It's a key part of how we're building the more stable enterprise-focused stuff we're doing at SUSE and Open SUSE. It all starts in tumbleweed. That's the main code base when this stuff all works. It's wonderful. It's not perfect. There's a few things in tumbleweed we really need to get fixed. To start with, this is the only sensible way of patching your tumbleweed machine. Zippered up, no allow vendor change. Not enough people know that, even though it's the first thing you read on the documentation these days. That's mainly a knock-on effect of how tumbleweed is built and how users are using tumbleweed with OBS. The traditional zipper up command is way too conservative. It only changes. It always assumes an upgrade. It doesn't work when upstreams change their version numbering. It doesn't work very comfortably if dependency change dramatically. It works fine for the regular release upgrade approach, but it isn't good enough for a rolling release. Zippered up, which you would think would be the right solution for doing the same thing but changing a whole operating system the whole time, ends up being a little bit too liberal. Quite often, especially in the presence of additional repositories, there's nothing stopping zippered up from grabbing packages from another OBS repo that you've set up on your machine and then using that to overwrite all of the stuff you have from tumbleweed. Zippered up, no allow vendor change is that happy medium in between of having the more loose zippered up approach of look at the latest version, get the latest version, put it on the system, but the no allow vendor change means it's going to try its best to do that for the packages from the repositories the user has chosen to install those repositories from. So if you're only using tumbleweed repositories, zippered up only pulls tumbleweed packages in. So if you're picking one thing from OBS, zippered up, no allow vendor change will keep pulling that one thing from OBS and not accidentally pull through the 20 other things that happen to live in the same repository. But it's way too obscure and way too long to type. I know there's millions of tumbleweed users out there who don't do this. And then they're machine break. And then they're on the forums. Yes. Yes. And there's a Zippered Conf option for that. I'm just getting to that part. So it is obscure and too long to type. I'd like us to think about changing either changing the default behavior of zippered up, because, for example, inside SLEE, we also use a variation of this for the ZIPP migration routine. There's a few extra variables there for SLEE. But it's clear that zippered up on its own is too liable to break stuff. I'd like us to think about changing that. Or having a variation, maybe specifically for tumbleweed, something like Zippered Twup. Or like Derek said, you can actually change a single line in your config file to make this the default behavior. Maybe we should be doing that in tumbleweed. Even when we solve that problem, and it's not that hard to solve, we just need to decide how to do it, we then need to also fix or remove the graphical update tools that tumbleweed is using. YAS and package kit. Because right now, you go to YAS, you try and do an update, it doesn't know what to do really with it. It doesn't have an equivalent of DUP, it doesn't have an equivalent of DUP and no allow render change. It tries to do a zipper up, which will work most of the time when things are quiet, but when there's big changes and big dependency changes, it falls over, it breaks, it doesn't work. And package kit, also, you know, a whole nother layer of problems there. So I'd really like to see us fixing those tools, getting them done, or if we can't get them fixed, remove them. So users don't get confused and start wondering, you know, why does my gnome update app keep on telling me to update and then not working properly? Yes. Yeah. The question is why don't we just make a script and do that? That's definitely one option we could do. You know, the problem there is people have quite a lot of update scripts now already, like for example, perfectly here, in the case of transactional updates. So transactional updates are a new technology which we've got in Tumbleweed. You can learn much more about it in this room at 5.15 today because Torsten's going to talk about it. But the short and simple version of it is transactional updates is taking the update model we currently have and the snapshot model we currently have where you're changing your system, taking a snapshot before, what's that, you're updating your system, you're taking a snapshot before, then updating your system and taking a snapshot after so you can roll back before and after and figure out what went wrong. That has quite a few negative side effects. One of them being you end up with a load of snapshots on your machine. And also it's not the cleanest and crispest way of keeping track of what's really going on in our system. With transactional updates in the simplest form, it's doing the update into a snapshot on BTRFS. So not touching the root file system, creating a snapshot, making all the changes in there and then when you reboot, that's the snapshot you boot into so all your updates are there. So it's completely atomic. It either all works or if it doesn't, you just roll back the whole snapshot and nothing got changed. It's already in tumbleweed and it's implemented as a script. So if we change the zip ad up, we have to change this or we can keep on using this. Yes. Yes. Quite often, in fact, the last time I rolled back was when I accidentally did RM minus RF in the wrong folder. But I haven't had to roll back because of a snapshot, because of a patching issue, but yeah, rolling back happens quite often with me, actually. But for transactional updates to really work, which I really think, actually, this is a technology in tumbleweed that's really exciting me. I'd actually like to see us think about doing this as the main update mechanism in tumbleweed one day to get there, because Torsten's talking about it in the context of cubic, a variation of tumbleweed, to get there for all of tumbleweed. We really need to get to the point where the packages are a little bit more compliant with our own packaging guidelines. Because the things that break this approach are things like packages putting stuff in SRV, packages messing around with user data, something that isn't going to be in that snapshot when it's patching in the snapshot, and then the whole thing falls apart and everything dies. We're not that far away from it, in fact. With the cubic stuff we're talking about later today, tumbleweed is already testing it alongside so we can see when they break. But yeah, long road to go. Snapshots are great. Funny we were talking about all of that. But it's only a temporary workaround. Once you've rolled back, you're back to exactly how you were yesterday, but tumbleweed still moved on. So you can't install anything, because you've only got the new packages in tumbleweed and not the old ones that you wanted to use yesterday. I'd like us to look at the possibility of retaining old snapshots in the tumbleweed repos. Not entirely sure how we'd do that, but maybe have fancy sim links, fancy snapshotted versions. Like I said, I'm not sure. It could be a logistical nightmare. But if we find a solution to this, we could really have this utopian vision of tumbleweed moving at full pace and users able to pick what pace actually suits them for tumbleweed. If they don't want to necessarily upgrade to the latest of everything, but want to maybe take a week or two or months or two longer to keep up with stuff, if we're keeping those old snapshots around in the build service and keeping those old snapshots around in download.openSuzo.org, users could still then have the version of the snapshot they want, get the packages that were built for that version of tumbleweed, and best of both worlds, everybody's happy or works. Obviously, despite the logistical issues and the fact I haven't got all the answers of how we could do this, it will also mean our mirror host would have to host a huge pile more packages. But tumbleweed is actually surprisingly small when it comes to our mirror infrastructure. We've got four terabytes now, I think, generally speaking. No, down to check. There's terabytes of stuff in our mirrors. Nine? Nine terabytes. Okay. Nine terabytes. Not four. Nine terabytes for most of our mirror infrastructure, that's all of OBS, all of our distributions, et cetera. Tumbleweed is about 60 gig of that. It's relatively small. It's a very fast-changing 60 gig, but it's a tiny part in comparison. So doubling or tripling it, especially if adding the additional size isn't going to change the churn rate too much because these are old snapshots, they're not going to be changing. Our mirror host shouldn't be that concerned with a little bit of the extra pain of having a few extra gig because they're holding nine terabytes already. And that's something else I'd like to see fixed in tumbleweed. We need more tests. I already gave you the sales pitch earlier, but we really, really need more tests. If you're finding something in tumbleweed you don't like, a bug, a behavior you dislike, whatever, write a test for it. The documentation is there, we accept poor requests, it's all in GitHub. We will merge that test, we will run that test on tumbleweed, and that problem will never happen again without us knowing about it first. That's fine for the generic stuff. There is sort of the second problem of how the heck do we test in video because Open QA is typically VM-based. So they don't have Nvidia cards in VMs. We do have some options for testing on bare metal. That's normally using stuff like IPMI, which means you're talking over serial and VNC, which isn't looking at an Nvidia card. So I'd love to have people thinking about how the heck can we test the Nvidia drivers, or any graphics drivers, but Nvidia is the one that breaks all the time on tumbleweeds. I'm just going to pick on them. It's theoretically possible. Like I said, we have to support in Open QA for handling different architectures, handling real hardware. It's all flexible on the back-end side of things. We just need a little bit of help with figuring out how to make that happen. Because if we can get stuff like third-party hardware drivers being tested in Open QA, tumbleweed will just have a whole other class of user who right now can't really touch it because they're unfortunate to have a laptop or desktop that has an Nvidia card in them. And then last but not least, the kind of non-technical thing I'd like. I'd like to hear how people are using tumbleweed. We've got to do a better job of marketing this. I can talk for hours about how I think the process is wonderful. I can talk about hours of what I'm using it for, what is the rest of the world using it for. Because with this, you've got a way of getting open source software in the hands of users quicker than anyone else, more cohesively engineered than anyone else. It's not just a server OS. It's not just a desktop OS. I know we've got people running it on Raspberry Pi. I know we've got people doing crazy robots and stuff like the other talk that's happening in the room right now. I want to hear about this so I can help get people writing case studies about it, writing blog posts, spreading the word. Because that's what's really exciting about tumbleweed. It's really a unique platform for doing cool stuff with. So please, there's my email address. Send me stuff. I promise I will help spread the word about it. So in review, who should be using tumbleweed? Developers. Any developer. Whatever upstream project you're working with, you want to be getting the latest and greatest stuff. Use tumbleweed. It just works. It's stable. If it doesn't work the way you want, just roll back using snapper. And if it's not quite perfect, you're a developer. You can help us fix it. If you're an upstream developer in particular, targeting tumbleweed is a great way of making sure your software is getting in the hands of users quicker. And our tools with the build service, with open QA, are there to be able to help you not just do it with us. You can do it with us first and then build it and test it on every other distribution too. Open QA isn't just an open suzer thing. OBS is building for every other distribution. We're even building app images and other containerized stuff in there now as well. Ultimately those tools are more work at the moment than traditional packaging when done right in OBS. So why avoid traditional packaging when you can do them right in OBS? And then as user, you want the latest and greatest of everything. Tumbleweed just works. And we would love you as a contributor. And then when you become a contributor, like I've already said twice now, the open QA process and the factory process, one is sort of reactive, one is proactive, you can make sure that tumbleweed is shaped exactly the way you want it to be. And if you do that and you join us and you become a direct contributor to tumbleweed, it's not just going to be helping you out. The tumbleweed user downloads are going higher and higher and higher. This is the last few years. So just to kind of explain the graph, because I notice it doesn't do that well. That blue line along the bottom is our old development branch. The orange line is old fashioned tumbleweed. The green line is the sum of both of them because they have branch and rolling. We kind of did the two things in parallel. So we had a few thousand users on it, but not a huge amount on those two different platforms. And then we merged them together, then we started doing this. And as you can see, since the end of November 2014, it's gone crazy. And I wanted to keep on going crazy. I want to have more users on that. I want to be able to show this graph next year that's twice as high. So please, thank you. Thank you and help me. Does anybody have any questions? Yeah. There's a microphone there if you want to go to actually understand. So it's a question about answering how to test NVIDIA. Have you tried PCI pass through with the virtualization solutions? In theory, I've messed around with it a little bit, but all the open QA hardware I have doesn't have an NVIDIA card in it. Haven't got the hardware for it. But yeah, in theory, that would work. Any more questions, comments, et cetera? Cool. Thank you very much.
 I'm now the after lunch slot, so I hope don't mind if people start trickling in, but I'm  going to get things started.  So I'm Richard Brown, Chairman of the OpenSuser project, and I'm here to talk to you today  about tumbleweed, why I think it is the best thing since sliced bread, why I'm most excited  about this and anything else we're doing in OpenSuser, why I think rolling releases are  the future of Linux distributions, and some of the bits and pieces where I don't think  we're doing everything that we could be doing to make it as smooth as it possibly could  be.  Because, you know, we're not perfect, we're just great.  To talk about rolling releases, I really have to start at the beginning.  We have to explain where Linux comes from, what is a distribution, and when we're talking  about Linux distributions or traditional distributions, we are talking about regular releases.  It's what most Linux distributions follow.  It's a model where you collect all of your different upstream packages, you put them  all together, you make a cohesive operating system as a distribution, and you're releasing  it every X years or months.  Depends on your users and your use case of how often that is.  Community distributions generally favor slightly faster release schedules, so distributions  like Fedora, Ubuntu, or the old OpenSuser would be every six to 12 months, and then, of course,  you have things like enterprise distributions where the new major release of an enterprise  distribution will be several years away.  Once that release is out, once users can download that software and start using it, the general  model of a traditional distribution is to not dramatically change the software within  that.  You know, being very, very conservative from that point, and only very reluctantly upgrading,  very reluctantly patching those things you need to patch to keep the operating system  working, but you don't want to introduce unexpected changes, you don't want to break anything,  so, you know, very, very reluctantly doing that, generally freezing anything, which means  when you look at sort of the big wide open source world and everything else going on  and packages elsewhere, upstream projects elsewhere, the only choice to maintain a regular release  is with heavy use of back porting, you know, so taking patches and fixes from the upstream  project and putting them into your stable regular release one.  And like I said, this is the traditional model, it's, you know, followed by Debian, Fedora,  open-suzer-leap follows it as well, and yeah, Ubuntu.  But developing these is tricky.  You need something to start this.  How do you develop a regular release?  And most other Linux distributions rely on a development branch to do this.  We used to, ours was called factory, but, you know, other distributions have things like  Debian CID or Fedora Vorhide or Ubuntu have something that they kind of call dailies,  but it never seems to work.  But it's where your developers of a distribution should be using actively putting in their various  upstream packages to constantly give you a rolling picture of, you know, where is your  code base, what is your next regular release going to look like?  Everything's ever frozen in there, it's always moving, and it's almost always broken.  This is true of every single one out there, every single development branch, it's typically  broken.  And this is problematic because developers need their system to be as close as possible  to the upstreams they're working on.  You know, they need to be able to see where is, you know, both where is their particular  part they're used to and everything else around them.  They need to be able to see how it's all working.  Dev branches accomplish that, but they're completely and utterly unstable and unusable.  So you've got this sort of nice deadlock problem of, you know, how do you actually then get  a good picture of what's really going on?  What generally happens is developers stop using their dev branch, apart from when they  really, really have to, to comply with some process somewhere.  That means you have very narrow attention being paid to what's actually going on in  all those upstreams that you're relying on to build your distribution.  So it doesn't work for developers.  They just move on and find some other way of hacking together their packages to get into  the main regular releases.  And that becomes a really big problem as a distribution project.  So traditional open SUSE as a distribution project, we used to do this, we used to have  factory and as your users and contributors of your dev branch decline, your entire project  starts getting slower.  There's no doubt about it.  You get less indications of bugs before a regular release.  You get less new features in your regular releases, you know, less innovation there because nobody  knows what's going on.  So even the most ambitious features they think of are relatively narrow things compared to  what they could possibly be getting if they had a broader view of where the world is.  You end up with over time, in particular, sort of increased technical debt stuff like  that is lingering around in your dev branch for ages and no one ever fixes it, which then  means when you do eventually have a regular release and you do eventually need to fix  it, clearing that technical debt makes that release more and more expensive.  It's more and more hard work.  It's more and more work to get the community involved in doing it.  And yeah, it just really, really starts holding the entire project back.  It's not just a problem for the distributions trying to get this software into the hands  of users, though.  It's a problem for upstream developers also because every upstream project, especially  in this day and age, wants to get their hands in the, you know, want to get this software  in the hands of users as fast as humanly possible.  Dev branches technically accomplish this, but they're useless really because no user's  going to be using them.  Regular releases don't accomplish it.  Whatever schedule, whatever distribution picks, it's going to be too slow for that goal of  getting it in the hands of users quickly.  And then containerized apps, things like App Image, Flap Pack, Snappy, you know, promised  to solve this, but it's not quite that easy.  You know, they'll make it there one day, and if you want to hear more about my rant about  that, you can come see my talk on Sunday at three o'clock in the other room.  But yeah, the problem's not ideal there.  And then when you start looking at users, and particularly enthusiastic Linux users,  or users, you know, the kind of core part of the community that are interested in open  source because it's open source, they also want to have that software as fast as possible.  They don't want to wait, but when they get it, they want to make sure it works.  So Dev branches don't work there either.  And there's also the sort of second thing.  Users want a consistent experience.  They want it to feel like it's well put together, that, you know, collectively themed looks  to the right, that feels right.  The UX works properly.  These are key requirements that users have.  And this is another problem, actually, that a lot of these containerized apps are starting  to bump into.  You know, they're getting out there and running these things, but, you know, getting that  feeling of this is a consistently built and a consistently engineered solution just doesn't  work with either the regular release model, where it's always too slow, or the development  branch model where everything's just moving too fast and breaking.  And these are the people we've got to capture, because these are the people that are going  to be our contributors in the future.  They're the ones who are enthusiastic, they're the ones who are looking at these upstreams,  who are keen on what we're doing.  We need to find a way of encouraging them to use this software and, you know, get enamored  by it so we can then start having them help maintain it and make things even quicker and  faster.  Rolling releases are the answer to these problems.  But what is a rolling release?  Well, in basic terms, a rolling release is a Linux distribution without a release schedule.  No version numbers, no point releases, no milestone dates, frequently updating all of  the packages in the operating system whenever they're ready.  So you can just download it, start using it, and you're always going to get the latest  ready, stable version of everything.  There's other examples, of course, I'm talking about Tumbleweed, but there's sort of two  other main distributions advocating this model and really pushing it, is Gentoo and Arch,  where, you know, quite popular, especially in that enthusiastic user base area where  you have people downloading it and getting the latest of everything in there.  And when I talk to people about rolling releases, I always hear the same three complaints or  the same three excuses of why they don't like a rolling release.  There's a perception that they're unstable.  There's a perception that they're unreliable, which is subtly different from being unstable,  and they're a perception that they're hard to live with.  In the case of unstable, in this case, I'm talking specifically about it's always changing.  The way I used my system yesterday is now different from the way I have to use my system  today.  A fast moving code base is going to include changes.  That is kind of part of the point.  So some of the, you know, there is always going to be a little bit of a change in there.  The question sometimes is how fast, and having different rolling releases at different paces  is something that I think the world needs to start thinking about.  But to really solve this problem, you need to be making sure that your rolling release  is building everything, testing everything, and then integrating everything in a consistent  cohesive fashion constantly.  And then when that is then delivered to users, delivered in a way that those behavioral changes,  that sudden new way that that new application is behaving, doesn't get in the way of the  work they need to do that day.  It's changed.  They're going to have to learn it at some point, but you don't want it to block that  work when they need to get their work done.  That's somewhat different from the problem of unreliability, of the perception that rolling  releases are going to break.  It's a fair challenge.  A lot of rolling distributions have this problem.  It is thousands of moving parts from thousands of different upstream projects, and the distribution  has to find some way of them getting it all working together.  Just like we're solving the stability problem, you've got to build it consistently.  You have to test it consistently, and you have to integrate it consistently.  But speaking from experience, it isn't just a case of testing it before you ship it, but  actually testing at the point of submission.  Finding as early as possible when someone is contributing to a rolling code base, does  this work?  Will this break the entire build?  Will this ruin everything?  Testing it there really, really early, getting fast feedback, helps both with the kind of  contribution engagement and use.  And then, of course, you have a second shield of it.  Testing is a whole.  Because you can't just think of a Linux distribution like a collection of packages.  This is the fatal flaw, which I think so many other distributions get wrong, and we get  right, is we think of our distribution like a cohesive single thing that we ship.  And we try and make sure that it all works together in one bit.  And we're not falling into the trap of distributions like Arch, where, oh, we're shipping this  wonderful library right on time, and they're forgetting that about the 20 other things  it needs or should go with it aren't there, aren't integrated.  And ultimately, the goal there is to make sure that you don't ship something that doesn't  work.  Talking about testing, most of the distributions, rolling and regular, and formerly us in the  past, rely on passive testing with their community distributions.  The idea of upstream has released something, we've packaged it, we've thrown it in some  testing branch somewhere or something like that.  And then we wait a bunch of days, and we just trust that someone in the community is going  to look at it and play with it, and then it's good enough we ship.  No one ever actually checked, did anyone actually tested it?  They just trusted that no one filed enough bugs, it must be fine, so we shipped it.  The model works to some degree.  The bigger the community is, the more chance you have of finding those bugs quickly enough,  and that testing window works out okay.  But it's still Russian roulette.  At some point, your users are going to get shot by something that slipped by that approach.  Passive testing just does not work for distributions.  You need to have active testing.  You need to have proactively confirming, does this new package in this distribution break  something just on its own?  Does it work at all?  Did the developer completely screw it up?  Or does it work when you install it with a context of 20, 30, or 4,000 other packages?  And does it change, or at least even if it's working technically speaking, does it change  in a way that users don't expect?  And you need to be able to answer that question in order to be able to integrate everything  quickly and fast and deliver it to the users fast enough before an upstream release has  even made that change.  So you need to have a way of knowing at least as fast as possible after an upstream check  something new in, or really before they check it in.  Does this change something?  Does this break something?  So we can get working on fixing it and shielding our users from those problems.  And then the last problem, the hardest to live with it, but when looking at the other  rolling releases, Arch, I really respect Arch.  They have this mantra of the Arch way, which gets summed up as do-it-yourself, it's a learning  exercise.  Not my way of doing things, but it works.  The Arch Wiki is a wonderful bit of documentation.  And then of course you have the Gentoo way, which is basically the same, it just takes  longer because you're compiling.  But that works for guys at the bleeding edge really, really working on this upstream stuff.  It doesn't work even for most enthusiastic users.  We have too much other stuff to do.  We don't want to be hacking around with the inner workings of our distribution.  We want something that we can just install, work with, and get the latest of everything  because we want to have our cake and eat it too.  And we need to have some way of stopping this.  Something's changed and I have to spend three days hacking around my system to fix it and  get it working the way it's meant to work.  And in OpenSuser, we've asked this question, why do rolling releases need to be difficult?  And obviously with Tumbleweed, we think the answer is they don't.  But Tumbleweed didn't start out as the Tumbleweed we now know and love.  It started originally by Greg Co-Hartman in, well, before 2014.  I can't actually find the exact date where it started.  And as you know, it's providing the latest updates.  The kind of key focus point there is at the pace of contribution.  Tumbleweed runs as fast as our community makes it run.  Sometimes that means incredibly quickly.  Sometimes that means we actively decide to do things at a slower pace because we think  that's the best way of handling what that upstream project is doing.  It's tested by OpenQA.  And in terms of that kind of user-based focus, we're really targeting that developer, contributor,  enthusiast part of things.  Because that's really where rolling releases really, really shine.  But old Tumbleweed wasn't like that at all.  Tumbleweed originally started as a model of taking the base system of OpenSuser we were  shipping at the time and putting rolling updates on top of that.  So instead of having a separate release, it was really an add-on for an existing release.  It had a very particular focus.  Obviously, Greg Co-Hartman's a kernel hacker, so he started with the kernel and then the  community started building up on that and things like KDE and GNOME and some applications  got in.  But that model of sitting the two meant that the only way of delivering software would  overwrite the packages from the original base system, which meant every time we released  a new base system, a new version of OpenSuser, your only choice was resetting to zero everything  in Tumbleweed, which was a really dramatic change because any customizations that had  been sitting in the Tumbleweed bit suddenly disappeared and vanished.  Sometimes had their packages roll backwards or just with different config or yeah, it  was always a painful mess.  That wasn't the only lesson we really learned from that.  I think the key lesson we learned was partially rolling distributions don't really work, not  in a general sense.  In a very narrow sense, in very specific, you know, small narrow use case with small  narrow changes, I think you can make it work.  But in terms of a general broad general purpose distribution, it just constantly fails because  that rolling top constantly needs new requirements from that stable base and you can't change  your stable base, it's stable.  So you end up having to come around with nasty little hacks, you end up tinkering with the  stable base, you end up linking stuff in weird and wonderful ways and it just falls apart  every single time and even if it didn't, you still got that reset to zero every eight months  with a new release, which is brutally disruptive for users.  And I kind of sum up this lesson as my rolling release rule, which I think Tumbleweed really  solved better than anything else.  To be able to move a Linux distribution where you want to be able to move any one thing  in there quickly, you know, you've got this massive code base and you want to move some  weird library on the far end of something, you've got to have your tools, your processes,  the technology in place where you can change everything.  Just be prepared to throw out the entire distribution and start again just to get that one new library  in there.  And we've done that with Tumbleweed, not just on its own because we've done that with the  tools we have.  We couldn't do this without the build service.  It's a key part of everything we're doing, we just had a presentation about it earlier,  so I won't go on about it again.  But the build service, the way it works, the fact that we can track all of these different  dependencies in all these different locations, link them all together, rebuild them when  they're needed, it makes sure that you have that consistent view of the distribution built  quesively together all of the time.  But building is fine.  You need to make sure it works as well.  And of course, we have OpenQA.  Originally started 2009 for testing the basic installation part of OpenSuser.  And it's become now an absolutely key part of the Tumbleweed release process.  A single Tumbleweed update doesn't happen until it's been tested by OpenQA.  It's also a key part of the leap process.  It's now also a key part of the SLEZ development process.  And it's even used by Red Hat for testing for Dora.  I haven't got them using it for Red Hat, but working on that one.  Hopefully one day.  I'd like to steal their tests.  But with OpenQA, you get these nice dashboards of all these different scenarios.  So we're not just testing one basic, boring use case, but different ways of installing  the distribution, different raid configurations, different desktop environments, different  architectures, although that isn't shown on the screen here.  In a deeper view, the tests break down to exactly the steps being done by the test.  And the kind of key part here is it's not testing artificially and just poking around  some APIs or calling very particular scripts to do very particular things.  The tests are written in a way to actually test the software the same way a user is going  to use it.  So when you're talking about that kind of conceptual problem of making sure that your  software, you know, you're aware when your software is changing in a way that a user  might be impacted, that's exactly how OpenQA is testing this.  So it can be a trivial change like a wallpaper or a login screen where we've changed the  color.  OpenQA at least makes us aware that that's happened so it flags it up so we can decide  is this the right thing we want or not.  If it is, we click next.  It moves on.  Everything's fine.  And that is all then tied together into the factory development process where we have this  kind of pipeline of code submissions going to tumbleweed, initially getting automatically  reviewed in the build service, then getting tested in a process we call staging where  we make sure in isolation does this one little thing work on its own?  If it works fine on its own, that's the point when we then start involving humans and actually  have someone looking at it and doing a proper review of the submission, does this actually  work?  At that point, it is then put into factory.  So you have the sort of full, large, big code base, 10,000 plus packages all built together,  all consistently integrated in one big pool, which we then test again in OpenQA in a much  more intensive fashion.  And that then gets done and pumped out at the end as tumbleweeding.  That means if you want to change something in tumbleweed, you've basically got two very  easy ways of doing that.  One, just contribute to OpenQA.  Writing a test to make sure that OpenQA is checking that one thing you really care about  means that every single snapshot of every single tumbleweed will make sure that it behaves  the way you want it to behave about.  So you don't really even have to worry about packaging anything or coding anything.  If you can just describe in an OpenQA test what you want to make sure stays that way,  it will stay that way.  Or at least it will stay that way until it breaks and then we'll figure out the best  way of getting around the problem.  But at least means we're aware that that use case has changed.  Or if you're more packaging aware, the factory submission process obviously contribute to  factory.  That is how tumbleweed works.  When I talk about all these toolings and processes, especially to upstream developers or new people  to the project, I always get the same responses.  That's cool.  It's great.  You're doing all this stuff fast.  But I don't want to wait for that build or the test nonsense.  I'm an upstream developer.  I've just got my tarball.  How can I run it really, really quickly?  I don't want to wait for all this testing and stuff.  Happens a lot with especially these people embracing stuff like Snappy.  The process works at a ridiculous pace.  I just realized I still haven't fixed this slide.  I got the number wrong on it.  Nome 322 is an example.  The upstream release of Nome came out and within less than 48 hours, we had it fully  integrated, fully tested and shipped in tumbleweed, every single package.  And it worked.  We had very few bugs, very few issues, users universally happy.  When we did have a few issues, they got fixed the next day.  In the case of KDE Plasma 5.9, not 4.9, we even actually shipped it on the upstream  release day.  The upstream release process for KDE, we actually get the tarballs a few days earlier.  So we were able to do all of that testing and pre-work in place there and then we just  hit the button when it was ready and done straight out.  If that's still too slow for you, thanks to the build service, we've got these kind  of separate incubator style projects where we can have derivatives of tumbleweed testing  straight from the get of these upstream projects.  So things like Nome Next or OpenSUSE or Krypton, where every single commit from KDE or from  Nome immediately spins out new versions, tumbleweed style, tested tumbleweed style, and they're  right away.  That's not nice to brag about two simple examples.  It's not just a case of the specific stacks where we're interested in.  Is Dominic here?  I guess not, because he knows all this stuff anyway.  But Dominic Lemberg, our release manager for tumbleweed, every week he does a report to  the community of what's been going on in tumbleweed this week.  A year ago now, he made this comment of it's been a quiet week.  There wasn't really that much and the report was shorter than usual.  And that got me curious of what is a quiet week for Dominic.  And that week was this, three snapshots.  That's three different software releases.  Basically the equivalent of a point release of a regular distribution.  Collectively all those releases put together.  We've included 146 new package updates.  It included a new kernel.  He changed a whole bunch of stuff on the DVDs that we ship.  That's quiet.  It's ludicrous.  It's an insane amount of change for one week.  In fact, a couple of weeks later, it was twice as much.  Last week, in fact, two weeks ago, it was three times as much as that.  The pace is still accelerating.  The process still works.  It scales out because we have more and more people using it.  The tooling works so we can do huge amounts of changes in a relatively short time, keep  pace with all of that and still make sure we're shipping something to the users that  actually works.  That works from our perspective and it works from the perspective of what upstreams are  trying to deliver.  Users might have different opinions.  But this is where we have BTIFS and Snapper.  Because we've got BTIFS as our default file system, because we ship Snapper as our tool  for taking a snapshot every time you do an update, that means the whole problem of something  changed in a way I don't like is immediately immunized.  You can roll down tumbleweed every single day and if you then find out it's not working  the way I wanted it to, you can always roll back to yesterday's snapshot and just work  from there.  Get your job done.  Even if we break your machine, not that it happens, but you can even do that from grub.  Even when the system is booting, it just works.  That's great.  But what about the development?  What about open-suser factory?  What about the dev branch approach?  Once we started putting tumbleweed together in this way as an end of rolling release,  that became the next question.  What do we do about a factory?  We don't need a development branch.  We don't have a development branch in the purest sense anymore.  There is no crazy rolling untested head for someone to mess around with because tumbleweed  is keeping up so well, we can give our developers something that actually works all the time.  They're always going to be close enough to be able to know what the hell is going on  and everything they're doing.  They don't need to have a factory anymore.  It's there still in terms of the process.  The process is the factory process, but the output of that is tumbleweed.  That's what people use.  I've been talking about rolling releases all of this time.  What about an open-suser regular release?  It's a simple truth that a couple of years ago, a huge amount of our community were very  much focused on the kind of concepts I've been talking about so far today.  Rolling releases, delivering quickly, factory or tumbleweed, and the enthusiasm for a traditional  rolling release was fading away.  The cool thing is, because of tumbleweed, we've been able to do really exciting things  with the regular release that we were too scared or blind to think about in the first  place.  With Leap, we're able to have that nice, stable, Susan X Enterprise codebase at that  nice, stable, regular release, the polar opposite of tumbleweed, the completely different use  case for completely different people, and appealing also to completely different contributors.  Then we can still, because we're using the build service, because we're using OpenQA,  because we have all of these tools and techniques that we've been doing for years, take parts  from tumbleweed that make sense, layer them on top, but still do so in a process that  you're not having this nasty issue of rolling and stable breaking, and ship a nice, cohesively  tested loop.  In the past, OpenSuser used to always seem to be split.  We had a community where some people wanted us to go faster, some wanted us to go slower.  We can do both now.  Tumbleweed is the fast road.  Leap is a stable one.  They both serve different users.  They both work perfectly fine.  They both actually help each other quite well, because it also helps that Slee is based on  what we're doing in tumbleweed.  All Slee engineers are developing on tumbleweed, and then it ends up filtering into both leap  and Slee from that point of view.  Just like that slide there.  From a cross-section of the whole thing, you end up with a pitch like this, where obviously  tumbleweed is over 8,000 packages.  It's actually over 10,000 packages now.  As a rolling base system, it's own unique code base, and then leap and Slee sharing  a shared core and overlapping between the two.  This is how things are developed.  I really should have updated the slides, because I've just noticed the numbers wrong.  Tumbleweed is rolling along constantly, constantly changing at its own pace.  Then next year, like we already announced, there will be a new code base to replace the  current one we are using for Slee 12, Slee 3, and leap 42.  There will be core 15, not 13, and there will be Slee 15 and leap 15, all originating from  what we're doing in tumbleweed, being frozen from there, polished up, tightened up, et cetera.  It's great.  It's wonderful.  It's great from a rolling release perspective of getting things in the hands of users quickly  and working with upstreams fast.  It's a key part of how we're building the more stable enterprise-focused stuff we're doing  at SUSE and Open SUSE.  It all starts in tumbleweed.  That's the main code base when this stuff all works.  It's wonderful.  It's not perfect.  There's a few things in tumbleweed we really need to get fixed.  To start with, this is the only sensible way of patching your tumbleweed machine.  Zippered up, no allow vendor change.  Not enough people know that, even though it's the first thing you read on the documentation  these days.  That's mainly a knock-on effect of how tumbleweed is built and how users are using tumbleweed  with OBS.  The traditional zipper up command is way too conservative.  It only changes.  It always assumes an upgrade.  It doesn't work when upstreams change their version numbering.  It doesn't work very comfortably if dependency change dramatically.  It works fine for the regular release upgrade approach, but it isn't good enough for a rolling  release.  Zippered up, which you would think would be the right solution for doing the same thing  but changing a whole operating system the whole time, ends up being a little bit too  liberal.  Quite often, especially in the presence of additional repositories, there's nothing stopping  zippered up from grabbing packages from another OBS repo that you've set up on your machine  and then using that to overwrite all of the stuff you have from tumbleweed.  Zippered up, no allow vendor change is that happy medium in between of having the more  loose zippered up approach of look at the latest version, get the latest version, put  it on the system, but the no allow vendor change means it's going to try its best to  do that for the packages from the repositories the user has chosen to install those repositories  from.  So if you're only using tumbleweed repositories, zippered up only pulls tumbleweed packages  in.  So if you're picking one thing from OBS, zippered up, no allow vendor change will keep  pulling that one thing from OBS and not accidentally pull through the 20 other things that happen  to live in the same repository.  But it's way too obscure and way too long to type.  I know there's millions of tumbleweed users out there who don't do this.  And then they're machine break.  And then they're on the forums.  Yes.  Yes.  And there's a Zippered Conf option for that.  I'm just getting to that part.  So it is obscure and too long to type.  I'd like us to think about changing either changing the default behavior of zippered  up, because, for example, inside SLEE, we also use a variation of this for the ZIPP migration  routine.  There's a few extra variables there for SLEE.  But it's clear that zippered up on its own is too liable to break stuff.  I'd like us to think about changing that.  Or having a variation, maybe specifically for tumbleweed, something like Zippered Twup.  Or like Derek said, you can actually change a single line in your config file to make this  the default behavior.  Maybe we should be doing that in tumbleweed.  Even when we solve that problem, and it's not that hard to solve, we just need to decide  how to do it, we then need to also fix or remove the graphical update tools that tumbleweed  is using.  YAS and package kit.  Because right now, you go to YAS, you try and do an update, it doesn't know what to  do really with it.  It doesn't have an equivalent of DUP, it doesn't have an equivalent of DUP and no allow render  change.  It tries to do a zipper up, which will work most of the time when things are quiet, but  when there's big changes and big dependency changes, it falls over, it breaks, it doesn't  work.  And package kit, also, you know, a whole nother layer of problems there.  So I'd really like to see us fixing those tools, getting them done, or if we can't get  them fixed, remove them.  So users don't get confused and start wondering, you know, why does my gnome update app keep  on telling me to update and then not working properly?  Yes.  Yeah.  The question is why don't we just make a script and do that?  That's definitely one option we could do.  You know, the problem there is people have quite a lot of update scripts now already,  like for example, perfectly here, in the case of transactional updates.  So transactional updates are a new technology which we've got in Tumbleweed.  You can learn much more about it in this room at 5.15 today because Torsten's going to talk  about it.  But the short and simple version of it is transactional updates is taking the update  model we currently have and the snapshot model we currently have where you're changing  your system, taking a snapshot before, what's that, you're updating your system, you're  taking a snapshot before, then updating your system and taking a snapshot after so you  can roll back before and after and figure out what went wrong.  That has quite a few negative side effects.  One of them being you end up with a load of snapshots on your machine.  And also it's not the cleanest and crispest way of keeping track of what's really going  on in our system.  With transactional updates in the simplest form, it's doing the update into a snapshot  on BTRFS.  So not touching the root file system, creating a snapshot, making all the changes in there  and then when you reboot, that's the snapshot you boot into so all your updates are there.  So it's completely atomic.  It either all works or if it doesn't, you just roll back the whole snapshot and nothing  got changed.  It's already in tumbleweed and it's implemented as a script.  So if we change the zip ad up, we have to change this or we can keep on using this.  Yes.  Yes.  Quite often, in fact, the last time I rolled back was when I accidentally did RM minus RF  in the wrong folder.  But I haven't had to roll back because of a snapshot, because of a patching issue, but  yeah, rolling back happens quite often with me, actually.  But for transactional updates to really work, which I really think, actually, this is a  technology in tumbleweed that's really exciting me.  I'd actually like to see us think about doing this as the main update mechanism in tumbleweed  one day to get there, because Torsten's talking about it in the context of cubic, a variation  of tumbleweed, to get there for all of tumbleweed.  We really need to get to the point where the packages are a little bit more compliant  with our own packaging guidelines.  Because the things that break this approach are things like packages putting stuff in  SRV, packages messing around with user data, something that isn't going to be in that snapshot  when it's patching in the snapshot, and then the whole thing falls apart and everything  dies.  We're not that far away from it, in fact.  With the cubic stuff we're talking about later today, tumbleweed is already testing it alongside  so we can see when they break.  But yeah, long road to go.  Snapshots are great.  Funny we were talking about all of that.  But it's only a temporary workaround.  Once you've rolled back, you're back to exactly how you were yesterday, but tumbleweed still  moved on.  So you can't install anything, because you've only got the new packages in tumbleweed and  not the old ones that you wanted to use yesterday.  I'd like us to look at the possibility of retaining old snapshots in the tumbleweed  repos.  Not entirely sure how we'd do that, but maybe have fancy sim links, fancy snapshotted versions.  Like I said, I'm not sure.  It could be a logistical nightmare.  But if we find a solution to this, we could really have this utopian vision of tumbleweed  moving at full pace and users able to pick what pace actually suits them for tumbleweed.  If they don't want to necessarily upgrade to the latest of everything, but want to maybe  take a week or two or months or two longer to keep up with stuff, if we're keeping those  old snapshots around in the build service and keeping those old snapshots around in download.openSuzo.org,  users could still then have the version of the snapshot they want, get the packages that  were built for that version of tumbleweed, and best of both worlds, everybody's happy  or works.  Obviously, despite the logistical issues and the fact I haven't got all the answers of  how we could do this, it will also mean our mirror host would have to host a huge pile  more packages.  But tumbleweed is actually surprisingly small when it comes to our mirror infrastructure.  We've got four terabytes now, I think, generally speaking.  No, down to check.  There's terabytes of stuff in our mirrors.  Nine?  Nine terabytes.  Okay.  Nine terabytes.  Not four.  Nine terabytes for most of our mirror infrastructure, that's all of OBS, all of our distributions,  et cetera.  Tumbleweed is about 60 gig of that.  It's relatively small.  It's a very fast-changing 60 gig, but it's a tiny part in comparison.  So doubling or tripling it, especially if adding the additional size isn't going to  change the churn rate too much because these are old snapshots, they're not going to be  changing.  Our mirror host shouldn't be that concerned with a little bit of the extra pain of having  a few extra gig because they're holding nine terabytes already.  And that's something else I'd like to see fixed in tumbleweed.  We need more tests.  I already gave you the sales pitch earlier, but we really, really need more tests.  If you're finding something in tumbleweed you don't like, a bug, a behavior you dislike,  whatever, write a test for it.  The documentation is there, we accept poor requests, it's all in GitHub.  We will merge that test, we will run that test on tumbleweed, and that problem will never  happen again without us knowing about it first.  That's fine for the generic stuff.  There is sort of the second problem of how the heck do we test in video because Open  QA is typically VM-based.  So they don't have Nvidia cards in VMs.  We do have some options for testing on bare metal.  That's normally using stuff like IPMI, which means you're talking over serial and VNC,  which isn't looking at an Nvidia card.  So I'd love to have people thinking about how the heck can we test the Nvidia drivers,  or any graphics drivers, but Nvidia is the one that breaks all the time on tumbleweeds.  I'm just going to pick on them.  It's theoretically possible.  Like I said, we have to support in Open QA for handling different architectures, handling  real hardware.  It's all flexible on the back-end side of things.  We just need a little bit of help with figuring out how to make that happen.  Because if we can get stuff like third-party hardware drivers being tested in Open QA,  tumbleweed will just have a whole other class of user who right now can't really touch it  because they're unfortunate to have a laptop or desktop that has an Nvidia card in them.  And then last but not least, the kind of non-technical thing I'd like.  I'd like to hear how people are using tumbleweed.  We've got to do a better job of marketing this.  I can talk for hours about how I think the process is wonderful.  I can talk about hours of what I'm using it for, what is the rest of the world using  it for.  Because with this, you've got a way of getting open source software in the hands of users  quicker than anyone else, more cohesively engineered than anyone else.  It's not just a server OS.  It's not just a desktop OS.  I know we've got people running it on Raspberry Pi.  I know we've got people doing crazy robots and stuff like the other talk that's happening  in the room right now.  I want to hear about this so I can help get people writing case studies about it, writing  blog posts, spreading the word.  Because that's what's really exciting about tumbleweed.  It's really a unique platform for doing cool stuff with.  So please, there's my email address.  Send me stuff.  I promise I will help spread the word about it.  So in review, who should be using tumbleweed?  Developers.  Any developer.  Whatever upstream project you're working with, you want to be getting the latest and  greatest stuff.  Use tumbleweed.  It just works.  It's stable.  If it doesn't work the way you want, just roll back using snapper.  And if it's not quite perfect, you're a developer.  You can help us fix it.  If you're an upstream developer in particular, targeting tumbleweed is a great way of making  sure your software is getting in the hands of users quicker.  And our tools with the build service, with open QA, are there to be able to help you  not just do it with us.  You can do it with us first and then build it and test it on every other distribution  too.  Open QA isn't just an open suzer thing.  OBS is building for every other distribution.  We're even building app images and other containerized stuff in there now as well.  Ultimately those tools are more work at the moment than traditional packaging when done  right in OBS.  So why avoid traditional packaging when you can do them right in OBS?  And then as user, you want the latest and greatest of everything.  Tumbleweed just works.  And we would love you as a contributor.  And then when you become a contributor, like I've already said twice now, the open QA  process and the factory process, one is sort of reactive, one is proactive, you can make  sure that tumbleweed is shaped exactly the way you want it to be.  And if you do that and you join us and you become a direct contributor to tumbleweed,  it's not just going to be helping you out.  The tumbleweed user downloads are going higher and higher and higher.  This is the last few years.  So just to kind of explain the graph, because I notice it doesn't do that well.  That blue line along the bottom is our old development branch.  The orange line is old fashioned tumbleweed.  The green line is the sum of both of them because they have branch and rolling.  We kind of did the two things in parallel.  So we had a few thousand users on it, but not a huge amount on those two different platforms.  And then we merged them together, then we started doing this.  And as you can see, since the end of November 2014, it's gone crazy.  And I wanted to keep on going crazy.  I want to have more users on that.  I want to be able to show this graph next year that's twice as high.  So please, thank you.  Thank you and help me.  Does anybody have any questions?  Yeah.  There's a microphone there if you want to go to actually understand.  So it's a question about answering how to test NVIDIA.  Have you tried PCI pass through with the virtualization solutions?  In theory, I've messed around with it a little bit, but all the open QA hardware I have doesn't  have an NVIDIA card in it.  Haven't got the hardware for it.  But yeah, in theory, that would work.  Any more questions, comments, et cetera?  Cool.  Thank you very much.
openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_0_263.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_264_311.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_312_383.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_384_455.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_456_503.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_504_563.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_564_827.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_828_1235.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_1236_1571.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_1572_4523.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_4524_5735.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_5736_7211.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_7212_12611.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_12612_12779.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_12780_14159.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_14160_14207.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_14208_14471.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_14472_18671.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_18672_20135.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_20136_21635.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_21636_23267.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_23268_23435.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_23436_23579.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_23580_24839.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_24840_26435.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_26436_27791.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_27792_28595.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_28596_29507.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_29508_30539.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_30540_31067.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_31068_31931.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_31932_33971.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_33972_35315.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_35316_35879.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_35880_38555.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_38556_39299.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_39300_39395.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_39396_40271.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_40272_41471.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_41472_41831.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_41832_42839.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_42840_43595.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_43596_45275.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_45276_45755.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_45756_45911.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_45912_48131.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_48132_48335.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_48336_48515.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_48516_48587.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_48588_48863.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_48864_54851.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_54852_65123.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_65124_66623.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_66624_66683.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_66684_67283.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_67284_68459.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_68460_68627.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_68628_69251.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_69252_70595.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_70596_72167.png openSUSE_Conference_2017_How_I_Learned_to_Stop_Wor_RbP9lNvmWKk_72168_72286.png