Infrastructure round table
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Angenendt, Ralph
Seeing that all distributions face the problem of maintaining their infrastructure (server landscape, keeping the services running, managing downtimes), this round table is an open discussion forum for infrastructure maintainers from all distributions and maybe other large projects which are running their own infrastructure. After a short overview about the infrastructure issues a linux distribution can have, these issues can be discussed between the infrastructure maintainers of different distributions and other interested parties. Besides learning about how other people handle infrastructure issues, this round table should also be a means of getting to know the people behind the infrastructure of other distributions.
 There is a project called DysNix which is used to distribute one configuration of our network. So you can define with the configuration file how the server interacts. But Sundar's show come soon to describe this project. Well then you have the machine deployment which normally probably is the same, it goes in the same area as conflict management. We get machines which are donated normally. So they sit in some data center we don't know of and then somebody puts a center sign on there and then we do the stuff we do. I think it might be different with other people, other distributions I know. I'm not sure about that. I think Demian probably gets all the stuff donated. We get all the stuff donated, yeah. Some of it is for the build system especially some of them are privately owned although we're facing those out. But most of the stuff gets donated in many cases by HP because HP is a very big sponsor of them. And presumably they get installed by them in person. They do that stuff, I don't know. With the GEN2 ones I'd say less than 15% of the machines are actually owned by us mainly it's a machine sitting in a data center somewhere and we ask them please bring it on a live CD and tell us the IP and we'll install it. Actually Nokia runs all the hardware from Mamo which is kind of uncool but that's kind of an interesting side problem. How many machines does CentOS have then? Perch infrastructure? Without counting all the debt ones I guess around 150? 150. How about GEN2? What do you do with such large number of machines? Mirrors for large part. So you count mirrors too? It's a machine. They are owned by you all the mirrors? No. No. Okay. Let's rephrase the question there. How many machines does your infrastructure team directly manage? Yeah. Like you have control over 150 that's what I'm saying. So you have control of those 150 machines? You manage 15 machines? You have a control of what? You can... I can access those. Those are all machines. Okay, they get pulled from time to time with new ones. If you need a test machine because I want to test out something new, you can get a machine from that pool to do that. If you want to do test that as you can get. But normally many of those are mirrors to feed other mirrors from. So that's primarily... Most of the time they are doing mirror stuff but on demand they can be used for something else. How many does GEN2 have? In GEN2 we have about 60 main infrastructure machines at the moment and then another 20 odd built machines. So that's... Yeah, well I'm not... In the infrastructure team as I said, on a blog of a guy who actually is in the infrastructure team, for Debbie and they were talking about 100 machines. I don't know where that figure comes from but 100 is whatever. How many do you guys have? I think it's much smaller than that. Up until November, Mamo had one and then they pushed everything out to Akamai. Yeah, yeah. No keep pays that bill. To build? No. No, they had another machine that they built internally but because of NDA restrictions that kind of firewall. It's a really broken system. I don't recommend it. And then you do have the management side of infrastructure. So where do those machines come from? Who do I have to say thank you for? Which machines do I have to look after, read while listening to this company when boom? Debbie actually had set up an LDAP directory over a decade ago where all that kind of stuff is managed. So after we have a file or several files. You really want to have something that's more internal with the version control. Right. Well, we have another directory where not only all users are stored in but also all machines with some extra schema stuff added to it so you can actually add stuff like who's the local administrator, who's the donator, who's the, what's the, what's the HKEY store in there as well. Who do you find when the machine goes down? Yeah, exactly. That kind of stuff is stored. Does anybody use something? What's my code? GDI, no, not GDI. GDI, something different from that. I don't know what you're saying. Being database for doing something bad. No, no. You're sort of making that a different system. This thing where this LDAP directory was set up before I became a development developer and I became a development developer in 2001. So it's quite old. It's working not too bad. It has issues. It's something home-roon as is the case for machine-debuted. So yeah, it doesn't jump. Okay, and then you have the planning side updates, new machines, how do you do testing, how do you deploy stuff out there? Does the neural network now go to centers five completely from centers four or how do you plan those updates? Well, the Debian Merit Network is not owned by Debian actually. It's the people who run a general mirror and oh yeah, we can test that. True, true. Those are most of the more mirrors too. But you have the internal stuff. Well, Debian probably has the same much as we have pretty long running distributions. So it's like, well, we have seven years. Well yes. So the machine is just a general center for one. RSA, release of the next distribution plus one year actually. So that's currently about three years, so not as long. Do you think there's a possibility to share some of the infrastructure across distro, particularly mirrors? I mean, since there are basically two major packaging formats, you would think that a lot of RPMs are duplicated across the various mirrors across the various distros. Well, they're not necessarily the same RPS. No, not necessarily the same version. They may depend on different versions of libraries. Yeah, that's true. But there's probably some redundancy. I mean, do you think you could have multiple mirrors, multiple package repos? Yeah, before trying to share binary packages, it would mean starting to share either spec file for RPMs, sharing macros, sharing batches. It's only a regular. I mean, if you look at Debian and Ubuntu, they sometimes have packages with the same name because they were built from the exact same source, but the version in Debian was built against this library and the version in Ubuntu was built against that library. So if you try installing the Ubuntu version on a Debian machine, everything goes to Kaboom. So don't try that. Well, you could have separate repos. Well, yeah, okay, but then you're not really sharing infrastructure anymore. You're just sharing the hardest, that's all. Yeah. And when it comes down to just sharing hardest, I think you may end up getting better results out of file systems with deduplication. Because they don't have to know that the file is the same thing. Well, canonical or at least my shuttle was proposed there and said, well, at least for the enterprise institutions, let's all have the same kernel. Let's all have the same glitzy, which doesn't work either, I guess, which something is happening there between the two. That would be one possibility to move in a direction like that. But on the other hand, it's just, well, you have machines, we have machines, we can let us have machines, because I don't think that you can really share mirrors, because it's just too dependent on what distribution is on there. I mean, when we were using RPMs, I won't get any when we were RPMs installed on centers moment without force. With force, yes, sure, it doesn't work there. Oh, that does pretty good. Keep those pieces. Yeah. Oh, actually six pieces. Okay, that's not really infrastructure, but also has to happen. Okay. How do you think about your man? Do you internal stuff? Yes? Wiki. We also have a lock down internal access only Wiki. No, public. Public Wiki and then policy documents. Well, we have private numbers in there even. No, we don't have that maybe. So, where do you share that information? Well, it's on an internal mailing list that people might have down. Everything we do in Wiki is just the public stuff, phone numbers and stuff. We should organize that pattern. That's why an LLBerge is good at it. That's why an LLBerge is a good idea. You just to held up search and you have the information and you can lock that down with all sorts if you only want your own developers to have that information. In that, you actually have a web front end as well where you can search and update stuff. So, that's what we want in our opinion. For documentation, we actually have three modes. Mainly, we've got people, authority, bankers with documentation, for our policy and stuff like that. We've also got the Wiki, presumably a lot of these documents. And we have the, in the web page, which is mainly a public CPS repository that gets automatically converted to web and stuff. So, basically, I think everybody does it that way. Yeah, Wiki seems to be... I just want to mention that within the federal, for example, something like a task which is in the counting system where people can subscribe and manage their own personal information. This lets the user decide which information he or she wants to share with others. You can simply say I want to share my number, share my phone number. So, between the development cycle, and these information is put in the music query from IOC. You can also use the passports in IOC. You can simply put one of them in the query and information there. And this is a very useful process for direct communication. What I want to say is which information we want to hide. Actually, I've started this one. Let the user decide which information he or she would want to hide. And to do not generally set up a hidden process. Because that's always something strange. I have to throw open process where the users or the developers or members of the community can either decide if they want to share or not to share their information. You can do that. For example, we have an LDSO and all its stuff to be denied something like that. There has to be infrastructure stuff which is not public. Because I'm not going to tell anybody the IP address of the servers. Mostly, yeah. Don't want to have a list on who's admin on the machine or who's responsible for a machine. Why not? Why would that be... We share on the infrastructure team, on the five and one team. All the infrastructure team has access to that. Why shouldn't it be public though? Obviously, the passwords are not going to be public. That makes sense. Little later, it's not going to have to be on there. Why not? If they want to, yes. Some of it is not as explicitly requested. That's a different matter. In general, they should be there. That would be silly. Some people would be explicitly requested that they are listed there. But what I'm thinking is, yeah, much of this information, as long as it wouldn't compromise their security, they don't think there's any harm in making it public. And it actually makes it easy for people, if they see something is going wrong, it makes it easy to figure out where to go. Absolutely. Absolutely. There's a page on the main website. It's quite far beyond some others. Fedora has standard operating procedures that they publicly list. Everybody who saw a server go down, reads the standard operating procedures and follows those procedures to bring the server back up. Right? Why not share them on the presentation? Yeah, a lot of other distros don't do that. I think that's a good idea. It is indeed a major problem within Debian to get more volunteers to work on that. Finding infrastructure people is very difficult. But standard operating procedures, I think, would really help. Yes. There are a bunch of procedures on the internal wiki that we really should publish. Like how we go about installing a new machine from scratch. That would be good to know. You do it yourself? No, no, no. Like based on your initial shell, this is what you have to do to fire up the automation. Okay. So I have to largely agree with what these guys said. It's some sort of the spirit of open source and it's not, it won't compromise the security. So you just put it online in a public wiki, just protect the page from being edited by anyone. And then you write there while we do it this way. Any user can read how you do it, but he won't be able to just do it because he doesn't have the right credentials to access anything. And it wouldn't be really transparent for someone who stands for open source software when he just says, well, our internal structure is hidden from you. We can't see how we work internally. So it's a natural thing for me. And the information that has actually to be kept secret is just a few access credentials and phone numbers and the rest can be made public because there's no secrets among those. There's an extension request about that there. We've been discussing it with you at the moment. Should all users be able to see negative status of what's down? Why not? Yes. What's a harm? Exactly. Because if you go to... I think it's actually a little bit of a fit to it because if you try to access something and it doesn't respond, then it might be at your end, it might be at the remote end. And if you can actually access the monitoring system, you can see, oh, yes, this server is down. Then you know you don't need to look further. You just need to tell the people that, hey, this server is down. You might want to reboot it. Whereas if you don't know that the server is down, you might be assuming that, oh, it's probably going to be running because these guys are good. So it has to be on my end. And you're wasting half an hour or half a day to figure out actually the server was down. So it's actually a good thing to make those kinds of things public. And I don't think it would compromise your... how do you say that? Your reputation or anything, because if your server goes down occasionally, because I mean everybody who's server goes down occasionally, it's not a big deal. But the only project we can find is doing this already is Missila. Debian Semi does that. Not really. There's something... some website where you can actually go. Yeah, the Missila one is not exactly public. Well, you have to do it. Yeah, the same goes for ours. And it comes up with a password, which is guest or something. Yes, yes, yes. So it's semi-public. But I don't think there's any harm in having it public as long as you don't put... It depends on the TES. The Wikipedia Foundation, the Wikia guys, publish both the Nagios and the CactiGraph in public. Okay. I think a part of the Yahoo open source group do as well. But these are both commercial companies. Most distributions publish the mirrored stuff. Most distributions publish the starts from mirrored stuff. Yes. Which certainly makes sense. Because... Also because those distributions you can't control all the time. I think the hard part is to find what I mentioned before, to find new members for the infrastructure. And there's something I've already discussed with you. How do we want to decide? How do we want to make sure that one who got the knowledge to manage the machines, who is totally trusted person, and who knows what he or she does, will get into the infrastructure. And this is something I'm not yet sure how to filter. I haven't found a way on filter that these aspects. Because what... how do we want to filter? That's one of the old infrastructure members, for example, have to know the new infrastructure member personally, what if he or she comes from India and nobody knows him or her, how can you make sure that you can trust him or her? How do you handle these things? How do you... I think the trust thing is the largest thing. Because seeing what people can do, they can vocalize themselves and say, well, what you're doing there is playing wrong, because if you want to do it correctly, do it that way. You get from people who know the stuff they're doing and they're interested in doing stuff. But I think the trust thing really goes very, very deep. In Debian, trust problem is basically solved by saying that only Debian developers can become infrastructure people. And then you have a whole trust... You have to develop a process which is rather harsh. Well, harsh is... I wouldn't... Do you know that I'm involved with that developer process? I wouldn't use that word harsh, but I understand... It's getting better. It's getting better. It's getting much better. I didn't say it was bad, so... You said harsh. You said harsh. It's the same thing. I know what you mean. Is anybody that's joined Debian explicitly to work on the infrastructure team? Or did they join as developers? Probably. I mean, I don't think so. I'm not entirely sure about that. We did get one person who joined just to document... documentation, not packaging. And in theory, there's no problem with joining just as an infrastructure member. We don't have a procedure for that yet. If somebody were to suggest that they wanted to do that, they would have to figure out how to come up with that. But basically, if you want to be part of the infrastructure team, you have to be Debian developer. That's basically our approach currently. So by definition, then you have to become a Debian developer before you can be considered to contribute that way. Become a Debian developer, you either have to do packaging work or documentation. And people don't know you, don't they? You can get the code for Debian build systems and for DAC and for rep repro and all that stuff and you can submit patches and that's one way to get in. And I'm sure if you did that, you'd get sponsored and it would be a very easy way in. So in that regard, it's all open. They're not necessarily going to let you log on to the machines and start hacking code, but the code's all available. Yeah. Do you have any of the old infrastructure team needs to know the new infrastructure members personally? Not personally, but you need to know who he is and you need to have... Right, right, right. I mean, not personally as in person, but in the case of the GEN2 stuff, there's two cases. First of all, are you explicitly joining as infrastructure? I've only had, I think, three people from that and they all happened to work at OSL's open source lab. So they already knew when I infrastructure that was there and that's why... They've grown up by trying machines already. That's why they were joining. But then you have to have been a developer for at least three months and somebody else that is an already an infrastructure member must vouch for you. That's the same as yours. What do you mean? You have to have some e-builds and operations in the system. No, in our cases, even more generally, moderators on the forums... on the forums, the GEN2 Wiki is not in-house. It's actually an external entity that runs it. So it doesn't really count. But there have been some knowledgeable people there that we've said, you know what you're doing. So they're sort of documentation, but not really. Right. So the next logic is that it would be kind of a mentoring process to let the new maintainer know how to treat the problem because they might be not very well-documented, some kind of special. Is there something like a mentorship? Isn't that the only way to do it? Because all the build systems are different. You know, knowing how to build packages on Debian is not going to help you know how to build packages for CentOS. And the infrastructure is catering to those tools. And the infrastructure is probably custom-made. So this isn't probably the only way to say, recruit people and mentor them through. That's one of the reasons why I wanted to have this round table here to learn how others do it. And if they're probably, maybe there are processes where we can say, well, let's do that together. Or let me learn from you because you seem to be doing the right way. Well, there is a name, right or wrong way. But distribution has its own rules. I think that these are the causes, the causes like mentorship. There's not one in any kind of distribution. And I'm listening to all of you. And it's really, really interesting for me. Because I'm a user. And I notice that there is an explanation of the role of every person on the interest action. Debian said that you have to be a developer. So this is the top. I understand? What should we do? The top, as in? The infrastructure. So you have to... WN develop is just a person with the ability to upload packages into Debian. So we can actually... And convert. To generate, there is a specific role of a member. Not only in Debian, on every distribution. And I think this is really, really wrong. I think that every person on the distribution on the interest action has to have a specific role. And not to do all the things. So, okay. Well, we do have that. We do have that. It's not...we mentioned it, but it doesn't mean we don't have it. At least in Debian, we've got people who do mail lists. We've got people who do system administration. We've got people who do FTP master, which is the mirror network. People in charge of the bills. And then we have a technical team who governs all of Debian too. So there's a lot of very well-defined roles. That's totally true. And we're crazy not to do it, because otherwise, some people would have to do everything and it just doesn't work. That's a problem. And they're also not actually interested in doing everything. Right. They're interested in one small thing. Like we've got two guys that are very interested in our mirror side. And they run it pretty damn well. I think at the last year, I said that we have to count the pass-on. And they mentioned that if five or six people are killed now, the picture is dead, or some guys are going to continue. I think this is something very specific on the role of the persons who are not actually interested in it. Yeah, that's why we posted the set for the DT-85. Yeah. Documentary. I think the only thing that we can do to make over here which sort of counters the argument for the gentleman there's made is that in a volunteer-based organization, folding people down to specific roles creates black holes. Yeah. Unless you've got a significant in large pool of resources, you cannot have people working within silos. Because if you do, you're creating black holes and you don't know how you're going to get your job. In Devin, I'm guessing the numbers are not in single digit. I'm guessing there's more than nine people doing infrastructures. It's probably possible to have a couple of people doing it in single digit. Much more. If you count builds administrators and system administrators and FDB masters and list masters, I'm sure you'll get in at about 30 or 40 people, something like that. So it only says, did you infrastructure team? Did you enter 27 people? But it changes. I mean, you're pointing to a very fundamental fact that these are volunteer organizations. You cannot tell somebody, are you going to work from nine to five this week? You can't say, well, can you even dedicate 40 hours a month? You have to sort of take what they can give. And that's a fundamental problem. I mean, and when there's a downtime or a break, and we've had two in Devin, then yeah, these people get, need to dedicate a lot of time that they may or may not have. So it is something to worry about, actually. So by the way, one more thing that I noticed, of course it does not have to be twice specific to here, is that developers make very bad decisions. Oh, yeah. Very bad work, too. But we do find that the people who tend to be sysadmins don't tend to have a lot of packages in Devin, in my opinion. They do sysadmin and they may have one or two packages, maybe something that they just happen to use. But indeed, what's up they use as sysadmin? Yeah, exactly. That's what I mean. There's something that they happen to use for their sysadmin work in Devin then. But usually they don't have a lot of packages, at least where I've looked up. Is this because they spend so much time sysadmin that they don't have time to do packaging work, or is this some other reason I don't know? No, I don't think so. One of the two ones is that they're not interested heavily in the packaging process. Right, yeah. Developers tend not to care about packaging at all. They see it as a necessary evil that they'd like to get out of. When you look at Python and Ruby, they both have pretty... Are you calling those packages? So yeah. I've heard a lot of people saying that Ruby has some of the most brilliant people working on it. And Gens is a very fine example of how a brilliant community creates a ship product. Yeah, definitely. But these are problems for the distributions, because I hear from developers, pro developers all the time, they say, oh Devin's too old. Well you're using Devin's stable. It's not old, it's stable, you're testing. Oh, but that's too cutting edge. No, not unstable, testing. But that's an issue for all distributions. What's your stable period? I want to patch it to 2-2. It's Fedora. What's that? You don't get 7 years out of that. 13 months are enough. What? 13 months are enough. 13 months. I'm standing in the middle of a machine. But yes, there's a problem. Developers don't want to do packages because they want to develop. Right. So they don't want to do packages because they want to do but they depend on the packages. Because I don't want to system the machines which aren't packaged. Right. Thank you. So somebody has some packages. Aren't most developers in all distributions doing mostly packaging work? Most of them, Devin at least I know. We have it rather easy. Packages are done for us. You guys just steal from me. Maybe you're right that you've grown too, sir. Yes, a little bit of that. Now, indeed, of course you can't see infrastructure, software, it gets confusing. You've got things like your package manager for Devin as the package which we develop ourselves, not we but some of us that develop ourselves. And we do some other software that we need to do. Most I think the developers for distribution will be packages. I think Susie's open build service will present a very interesting challenge because I think that it can build both devs and RPMs. And if you have an automated system, you're taking away the human element of building packages. And that's going to... You have to have rules for the Devin packages, you have to have a backup for the RPMs and you have to have a different backup for Susie RPMs in there. You have to keep in mind the difference between distributions that are not just only packaging formats, they are more than just different format for packaging. Precisely, and when you have an automated tool that builds packages, then you don't... you're hopping over a human and there's a difference in quality in the package. So does it have to be a human? Does it have the internal knowledge of difference between Devin and Red Hat, for example? I don't know if open Susie has this, but I do know that there are distributions that use open Susie's build system to build their distro. What you still need to do is you still have to know, some human has to decide how do I configure this package, do I enable this feature, do I disable it, do I build two packages that... that's disabled and one that's enabled or do I just... or how does this build system... does this package have a plug-in infrastructure, do I ship the plug-ins in the same package, do I split them in separate packages? That's all... that's things that a human has to do. And after that, I mean, you don't compile it yourself. After that, after that site, you just run your compile tool and if a bot does it or if you just type one command to compile, there's no huge difference from me. So decisions still have to be made by a human who puts these specifications into the build system. I mean, even if it's not a spec file, it's going to be a configure file or whatever... It's any specification that's helping you build... It's just a specification of how to build something and indeed, whether the bots or a human builds it, if I type make or the bot types make, the result is going to be the same, hopefully. Yeah, well, I do maintain a build system, so I do say hopefully. Yeah, I mean... yeah, anyway. I mean, for the open-source of build servers, it's of course the case that every package has a maintainer and more specifically a maintainer group. So we divided up the packages into projects where you can have groups of maintainers and it's also true that we build for WNN and RPM-based distributions. And I think that's a chance because we agreed, we just agreed that nobody really wants to do packaging work. So we don't want to, this is a bad start for us, so we have to seek out specifically for packages. Which I don't think that's the most effective thing to do in our world. So maybe not the studio. You still have to know what the, for example, naming convention is. Yes, of course. The packaging is extremely a difficult thing, but it's also true that most people come in as developers or as users and then they have to be turned or turned and sell them to packages. So you mean enabling users to build packages for, I mean, enabling people to collaborate on packaging might be how to reduce the pain of packaging. Would you put those packages officially in your distribution? Would you take those packages by people you don't know and put them officially in your distribution? Of course not. No. Is it okay for you if you create a service where users can put these packages into one centralized searchable repository? That's OBS. Or in ARCH? We have some public and searchable repositories, but we always have to review process. So we always have to maintain a group and these people are responsible to review this. Yes, of course not include this in the official repository. I think we just do, okay, sorry. Yeah. About the packaging being the hard work, do you think that part of this is that most packaging formats are hard for users to write? Like spec files, e-builds, I don't say that one of them is harder than another one, it's not the other one. But do you think that it depends on the clarity and simplicity of format to improve number of users who would like to write it? Yeah, if you ask me personally, I would say yes, it's a tough and hard job to package, no matter if it's spec or dead end. I would appreciate a more simple thing that covers all, for example. But I know that it's completely often future because it's hard to do it. I'm rather fascinated by all these statements because we actually have exactly the reverse problem. So over 90% of people in ARCH just do packaging. So people package and nobody wants to do the other stuff. So there's people making packages and so on and then there's nobody taking care of that mirrors are in shape and then this task is given to someone and he gets lost in some work and disappears or anything. And nobody wants to do development except the package manager, there's an active development but all other things that need development, well, they just get stuck and get lost and so on. But packaging is actually the part that has enough people on it. But the rest is just, well, it's just not improving as fast as it should, considering the number of, increasing number of users, increasing number of packages and so on. So it's just a reverse problem of most people. I don't think I said that. What we have in Debian Mainpalm is not people wanting to develop because that does happen although there are some cases where we need some more people. But I think the main issue we have in Debian is that because the effort for development to maintain is about four or five packages, people start complaining if you do more than 15 or something. Yeah, except, I think I've got my name on a couple of hundred. Yeah, exactly. But if you do more than 15 or 30 packages, people start really complaining. But the main problem we have with that is that people fail to see the big picture and I think that's one of our main issues for packaging. I mean, it's very easy to get focused on those four or five or six packages that you're doing and not really care about what other packages have in track with yours and stuff like that. So in Debian, packages, in average, have a small number of packages? Small number of packages, yeah, absolutely. There's one of the team maintainers. Is this encouraged or is it just an effect? That's a difficult question. It used to be the case that every package only had one maintainer. And every maintainer only had up to ten packages. And it's still the case that there is one main maintainer who is the, who's the distant maintainer. But more often than not these days for large things like KDE or Coke, no, the maintainer is a mailing list. It's a group of people and then there's people who are listed as upload or co-maintening basically. So we have team maintain packages and if you look at team maintained packages, then you do see that people are often member of a team where the team maintains like fifty hundred packages. What I meant with the thirty packages is if you're listed as the sole maintainer of a package and you have more than thirty then people sign up. But in teams the issue is not as big because then there's more people in the team who can actually look after the packaging care for it. Drifting a little bit into the packaging. Yeah, sorry. It belongs to you too. Yeah, it does. It's kind of at the heart. I'm sorry that the HR talk is directly besides this talk here. We were talking about HR and HR is a new resource. Yes, but we were talking about that all the time at the moment because probably people get into distribution and people get into teams and distributions. That does have to do with the structure too, sure. Is there any distribution here that pays for the infrastructure? It's a kind of pay. If I need to turn servers, I'll go somewhere and buy turned servers. Basically 15% of our GEN2 machines are owned by the GEN2 Foundation. We do not have a pay for any hosting. In Debian, the case mostly that things like our main servers, if we need them, we just ask B.L.Garby who is Linux CTO at HP and he just hands us a bunch of servers. So that's not the issue. In the case of things like architecture specific hardware such as tool systems, that's harder because HP doesn't have all of them. That's about it. We need to be prepared to which happens with the other architecture quite often. I guess we are the only ones who buys stuff that is a little specific because Mandriva is both a company and Mandriva Linux is a community project. I would guess for Fedora the same would be true. It's very datastic by an ACT하발ạt. also has a lot of connected machines. And I know there are at least a dozen hosting companies at Dell, for example, and actually even who have donated equipment which is hosted in construction. And I think also the interface stuff have got to the cloud and Amazon hosts. Oh, that one. OK. And they're there. And it's more spondering ship between us. That's the other reasons that we mix up all. This is a very brief question. Do you feel you have an over-concentration of machines at any one single sponsor that could actually threaten things if that sponsor or its own went away? Good question. I don't think we have that. We have two or three core machines and with one sponsor, that could be a problem. You're a commercial organization. CentOS is, though. We're not commercial. You're not. We see it's not commercial. We see it's community. But your target is certainly business sense, as opposed to, say, ordinary users. Although we want people to say, well, I need the stability of an enterprise distribution but don't have the means to pay for that. But I don't get the support. And I know I don't get support for that. So no, not really. If there are business that want to use that, yes, sure. Many hosts use it. Not here or in Europe, probably, but in America. Lots of hosts are using CentOS. Here you get more, Debbie and Osuzi, I guess. In Europe. Or at least in Germany. I think just to come to think about this, I'd sort of like to reply to your question about the single point of view. I think the problem that we have within CentOS is something which is completely on the extreme other end. And that we would really like to have more than one machine hosted at the same place. I think we've only got two sites where you've got more than one machine. So now if you take the number of machines you've got and think about the fact that you don't have more than two machines in any one place, you'll see how many different locations we spread over. And every person has their own process for support. Everybody has a new ticketing system that you have to improve. We don't have that in Debbie, actually. We've got a number of machines hosted at an HP data center. And then some at Brain Food, I guess, though. I'm not sure that's in Texas. So we've got like some 10 locations. And then of course the old small ones. Houston and Foxconn, it's orange. Sorry? Houston and Foxconn. Three places for folks. Right, right, yeah. Houston and Foxconn. Right. Yeah, but I was talking about Brain Food, which is Texas. It's a small company. They also have some. They used to host masks in the open, some other things. Right, we're generally, I'd say, 50% of our machines are between four big sponsors. They just have machines there. Right. Is it all problems we can't do failover? Because machines aren't sitting beside each other. Could be anything. If those machines don't see each other, that could be anything. The data center's on fire. We can't do failover for other reasons. But yeah. Yeah, OK. There's actually a larger problem than have these different locations. Like the G2 forums, it's the most popular thing. We keep having two increase the amount of hardware for that. If you're doing database replication and stuff, latency is a massive problem. You have to put all in one location. Right, yeah. So at the moment, that's four machines at Hives NL, the Dutch social network. They said, you guys can have four old machines. Oh, sorry. Forums was at Hives. And now, Bugs Miller is on those old machines. And forums has moved to Gossamer, Trades, and Vancouver. Because forums didn't fit on the old machines anymore. Right, right. So you had to ship a whole bunch of data. That's why it sets more things, mainly in this. Because I didn't come in the forums. We have forums. We have more forum subscribers than mainly subscribers. But I think we have more active mainly users. We have some more users. We do forums. We do forums. They've been not met, but they've been not all. So it's not official. That's the distinction. They've been not met and they've been not all. Every they've been not all, they can set up a they've been not all. That's post name. And they've been all. Orc is just maybe something interesting. So if you have them in the orc, that can only be modified by the Degasus administration team. I think in the last time, look at the Gen2 forums versus the mailing list. They're getting 5,000 forum posts today. And the lists are only getting 3,000 or 4,000. I think we're kind of the opposite. The lists get a lot of posts, and the forums are kind of dead. Well, that's not true. The forum was, Yerun and Teess had set that up. And then they really weren't in uniform thing. But they set it up because people were. Some people were. And at some point they got compromised. And that was very good. And then they disabled it. And then they came complaints. And then somebody else volunteered to maintain it. It was more intuitive. I think we have more forum traffic than we have. But I don't know how much forum. But the Degasus mailing list is possibly huge. Your mailing list is way busier than the Gen2. You have 12,000 dev-in-developers on there. Last but people who are interested. Right. We've got massive amounts of posts today. And that's not even counting to panel. For each 90% gets thrown away. I think we have 100, 200 mailing lists post a day on max days. So that's not that much. I get that much. I'm merely, wait. I don't want to think about it. So here's another interesting statistic to be in about location. As of last Sunday, we were in 21 different countries. But on Monday, we lost to Xi and Russia, which we don't know if you want to get back again. So we're down to 20. But we are talking to people with two different data centers in China. So if that goes through the next week, we may be back up to 21. Yeah, or another thing that's, we have a machine in China. In China. Is that an amnesia? Is that a great firewall? Actually, it is an issue because both of these data centers have multi-gigabit connectivity within China. Right. But one of them, and this is the truth, this guy actually only has 128 kilobyte length for the rest of the world. How? Because it is the maximum packet filtering cable that the government of China is going to give them. It's got the capacity, but it can't push more than 128k at a time because of all the different. Maybe throttle the traffic to that rate. So the problem is he started syncing his main data in the Google Lodge. He hasn't finished yet. Ha ha ha. Wow. We're at the same thing in Africa. He supposedly has a, well, OK, only 10 and we're doing 22 Africa or into Egypt, of course. But we couldn't see the machine. It was just the initial r-sync wasn't done after four days, after five days. Just didn't finish. So there's an open office update. He's always syncing for three or five days. He might have better called, he might have better connectivity into Egypt or into neighboring countries. But it was out. But it was blocked. You don't get the stuff there. So, OK, another question I would be interested in. Who actually has infrastructure in South America? Besides Brazil, sorry. I think, I don't know. Do we have Brazil? I think we have two Brazil machines. Do we have one, please? Let me check. OK. Or in Africa? We had one in Chile. Africa? We had one in Chile. We had one in Chile. Anyone else having machines in Africa? We're looking it up. OK. You can't see no more machines. So I know we don't. Slash. What? Egypt is the sole machine. Asia is another point. Well, yes, Taiwan probably, Singapore probably. Yeah. South Korea, I think there's one stuff in Asia. I don't have anything like that. Hard to say because it only says the name of the sponsor on the location. So we have to check all the, it's not that far. Sorry. Four minutes. OK. You have to decide who wins. We actually have a break afterwards. So we can run a little over if we want to. Yeah. But thanks. Yeah, we're supposed to finish it in three minutes. I show off hands, who wins. What was that? Who wins. I win. What do you win? You get to maintain your server? I don't know. No one's got to win. Let's do a great one. The next stop starts in half an hour. We're in Columbia. Yeah, we are. We're in Columbia. We're in the same one. Well, yeah, we, I think everybody has mirrors all over the world. That doesn't count. So with the Chinese thing, we should finish that discussion. What we're working on is that Hong Kong has a lot of capacity. So what we're working on is a couple of USB hard drives. So we actually get it into Hong Kong and we ship using post-school mail messages. So it might be funny. It actually sounds really funny. But what happened is because we don't have any seats in China and this machine that we run ourselves, there are actually three or four external mirrors who mirror us in China. We estimate that we've got six digits of users in China. But these guys want to pick traffic. But these guys want to hit update mirrors. In other parts of the world, they generate a phenomenal amount of traffic out of China. And a lot of mirrors don't like it, whether it's because of politics or whether it's because of policies. We've had mirrors going around. I think I set up a mirror for St. Paul's in Germany. I don't want, how much was it? You got four pedabytes of traffic in China and one in China? The little traffic was because it didn't have a model of I.O. on there. They're getting hit from some download software, there's some download manager, which hits the mirrors in parallel. And if you kill the IP address, it just comes back with a nice IP address, just because most people can't get the software in a different way. So they try to spread as many mirrors as possible from as many IP addresses as possible. That seems to be some Chinese-based download management software. I've never heard of that before. DDOS attack. You guys have well-defined policies as to how often users should hit mirrors before they get banned? Because we don't control most mirrors. But you guys are seeing mirrors. OK. With the Async Mirror site, there's actually two separate tiers. The core set in async.engin.org is strictly in house. We guarantee the content on those mirrors. We validate that. And they typically run a few minutes ahead of the community mirrors, because the normal community mirrors are well maintained and very fast. All the actual core ones actually serve directly for ramps. It doesn't touch this. Wow. How large are your Async data? That Async said, it's 280 bytes of data that you have to watch out for. If you don't watch out for 280 megabytes of data, I'm mistaken. If you don't watch out for high nodes, it loads up to about 700 megs. We at the moment have our internal sync machines open. If you know which machine it is, you can sync from there, which we reserve the right to publicize your machine there. The Async panel says, well, only sync from here if you are an official mirror. It's not that bad. It's on the house. In Debbie, I think it's all based on IPX and lists. So if you want to be a primary mirror, you talk to the mirror admins, and then they give you an account. And then you get it. We have that for DVD at the moment. So if you want to serve the DVD, find another mirror who already has the DVD or RRs. And then they work with SSH triggers. So once we finish you, SSH can do there, and it styles the process, and you can disconnect again. So it's a really wonderful model. Yeah, we do have restricted mirrors for the community mirrors to access. The rest of the gentubers are also in.country code. Those are all the community-run ones. They have a specific mirror that only they can access. They have specific gates. So does that mean that all of your R-sync mirrors have to have a dedicated IP? No. Because if it's R-sync.country code, you can't actually be who's R-sync. Unless you just need to look for a part. Yeah. We do have a bunch of them that they regularly send us the IP data every 60 minutes. If you haven't seen it, we expire it out the list. So kind of dynamic DNS. Right. Mocked up. You're not going to slide. OK. I had some more slides, but we talked about that between some of the last one. In that case, there is a wrapper called PowerPay that checks just the mirrors that you have on the mirror list. And then download a piece of program from one mirror, another piece from another mirror, another piece from another mirror. So you don't download all the software. And it is on the bandwidth on a specific mirror. And it is based on ARIA, too, for the one. And it's really fast in the... Well, I think it's a good idea. Like you, though. You should have been looking at the mirror brain talk just before this one. Yeah, exactly. You should have been looking at it. I should have been showing it. Yeah. Yeah, I had to. Yeah, the talk right before this one was talking about a program mirror brain, which actually does what you just described me. No. I think you should have been talking to the city because I had to with the metaline. Yeah. Yeah, that was coming in the previous talk. Whatever. There was a reason why I tried to block them together. Yeah, there was a reason why I didn't want to do a mirror in Ashkark here because there's a whole kind of different problems, I guess. Right, yeah, sure. Sure. I was going to say something. You know, you were talking about bringing people in to the infrastructure things. Does anybody do interviews? Or tasks or do you do some sort of a selection based on something like that? Definitely when new people join the infrastructure team, there's a couple of things off the top of the easy items to do on the to-do list that here are something that needs doing. Yeah, I think if you start selecting people, I think you'll find out most usually you've got too few volunteers as it is and if you start selecting them, you end up with even less and then you've got more age problems. It's usually best to give them some low-hung fruit and see how they do and then if they do well, then give them some more responsibilities. I think that's the best way to do it because otherwise you just start to get people. And if they do the low-hung fruit and either mess up or it takes way too long or something else goes wrong, then you know you don't want to give them anything else and you move on and they move on and they've had a chance and you've seen it didn't work out. So I know I'm done at that point. I think that's the best way to do it, basically. So what, as an example, what would some of these tasks be? One that comes up pretty frequently for Gen2 is actually the initial setup of a machine just prior to firing the automation. In our case with Gen2, it's that you log into the machine, you're going to end up, you have to be partitioning the machine's menu. That's not part of the automation just because the machine tends to vary a fair bit. So you have to be able to do that. You have to partition the machine and set up the LVM. And then you have to read enough of our internal CF engine documentation and setup to add the new machine to CF engine so that when you fire the automation, that the automation is going to work. And they get to read the logs and say, CF engine failed to add this machine. Why? Because you forgot something in the configuration. I'm not the last person to receive the information. Yes. I'm not going to move to puppet. I don't know how good CF engine is with role isolation where puppet and chef and VCFG are really bad at it. So if you have to give somebody access to CF engine and manifest behind it, that person could pretty much do anything on any machine anywhere. Yes, I know they can change a whole batch of stuff on almost any machine. And then you boot them out and you don't talk to them anymore? In our entire time with the Gintro infrastructure team, we've had a single person that went abusive and he was dealt with within six hours. Exactly. That's a question I have. How do you throw people out? Not that we have a luxury at the moment. We've been able to throw people out. We've had that problem within Debian and we've actually booted a few people out mostly for not being abusive in infrastructure kind of things but more socially abusive. Yeah, I've been in Mexico City some years ago. Yeah, that was one case. That was a very well-publicized case and there was actually a bit like on the spot and figured out how to get built. So I guess key distribution and possible distribution is not automatically so you can revoke on key success. Yeah, yeah. What's behind my question? That's what I said. All right. Sorry. What we have is we've got to move on to the other actually. So you just remove them from the other actually and your account is gone within a few minutes actually because it actually contains a local cash of the account. So 15 minutes later your account is gone. And for uploads, there's a central GPG key server which contains your key if it goes out of there. It sinks into the maintenance software and so a few minutes or a couple of minutes. Yeah, kicking on a regular developer is effective within a couple of minutes. Right. The specific case was we're having to boot somebody out of the infrastructure team. Right. And just the basic automation side of it is within half an hour but all this stuff you have to go and check by hand since the person has had root access to the machines. Right. It's 6,000 so of work. Right. Yeah, sure. That takes a bit more. What do you do with people that are in the team, whether it's in the infrastructure team, whether that's half responsibilities, they've worked well but certainly during their life, they don't have a priority anymore to work on that. They just don't hear anything. How do you process getting those people out of the project and getting people in because often they don't take responsibility by saying that they don't have the time and they just disappear? I'll give a quick answer to this one but one of the other junior two guys that's probably in the remover was actually just going to be discussing it more fully. We have a list, a mail that goes out twice a week to an internal team that we have called the retirement team. After 60 days with no commits, we'll send you an email and we'll ask you what's up. And if you don't respond to us within another 60 days, you're gone. You wait 60 days. 60 days to notification and you get 60 days to respond. 120 days. 120 days, four months. Even if all your software is bug free. Pardon? If your software is bug free. Well, you respond to your response. Within Debian it's slightly less automated. We've got a team which we don't call retirement. We call it the MI-18 but it's basically the same thing. What we do there is if somebody's bug pages is like many bugs there and nothing's happening or stuff like that then you get a mail from the MI-18 what's up and a few more emails if nothing happens. It does go a lot faster than 60 days from that first moment but there's a few reminders because maybe the mail didn't arrive or stand full to go in between stuff like that. We'd actually send a few more emails. Yeah, sure. Anyway. 60 days and 60 days. Right. So, it needs eventually if we don't get any response or the mail bounces which happens. People forgot to update their email. We actually give people local email accounts. Right. We do love you that you've changed it. Let me know. Mail is forwarded usually. So, if you don't update that then the mail bounces then you're gone. In the case if people are not active and don't respond to the MI-18 then eventually they get thrown out. Yeah, particularly if they've been around for a long time the mail starts bouncing we will make an attempt to contact you via other means. Yeah, so will the MI-18. They will try but if you don't find them then it's gone. But this is in case that the people are off the internet or something like that. But what in the case that they're still around, they're still interested in helping or when they don't have the time, answering email is easy. Staying in the team is easy. You know, because it gives you respect to stay in the team but they're not really... What you mean is they're still active, semi-active on the team because as much as they should be... They're active within the community but not necessarily the team. Right, some of that. But you can't go... Yeah, and we separately encourage that, we encourage the teams to trim out their dead weight per se. I don't think... Most of them do. I don't think we have a process for that in Bebion actually. We don't. And maybe we should because... Yeah, the Perl team has probably 150 members. But at most 30. Yeah. Oh, yeah. Yeah. Yeah. Yeah. I think the sort of thing is probably easier to quantify this code or this activity because you can see it should be done eventually. Yeah, I would like, within the infrastructure... I can see that you're making commits to the infrastructure repository that controls those. So should we sum up, maybe take one more question and finish out so people can get something to eat or something? That's a question. From your experience, what would be the most important lessons for a new distribution startup? That's a good question. Are we going to take five minutes? I think first and start. Because many distributions probably started by just one guy or two guys, just started something and then it grew and it grew and it grew and in the end you end up with a pile of packages and a pile of infrastructure and some people and nothing is organized and you sit there and make... No, don't give in. Don't give in to your processes. Nothing is documented, nothing is organized and then you think, now it doesn't work anymore, it's just stuck, it's just in some kind of a deadlock phase. So you just have to think, how do we do it, then do it and then just say, well, let's start by choosing a package format, let's start by choosing some basic packages that I've started. Basically what you're saying is you need a plan. You need a plan, yes, you need a concrete plan. How do I organize my distribution? Because certainly we had that problem. There was one guy showing here that more people and more people and more people and suddenly the founder of the project and the main lead and the only one who had root access on every server suddenly went MIA and the distribution was basically dead so there was a package and they were new releases, there was nothing really new for over a year and still, until he came there and said, well, I don't have time for anything, just someone else do it and then we started, well, we need some processes, we need some things and I think it was in 2005 after four years or maybe 2006 after five years and maybe that's a bit late. Another thing, think of automation early because it's easy to do this administration and you have ten machines. As soon as you have twenty machines, you're completely lost. It's easy to set up automation when you have ten machines and it's much harder when you have... It's harder to get that automation when the machines are already running because they might look completely different from what you're running at the moment. They will look different from the one you get. I want to be positive about that. Well, the thing is, if you don't do automation in the beginning, then your machines will diverge. You will accumulate differences and when you implement automation, when you will eventually implement automation, you need to work away those differences and that makes an order. Even with automation, I guarantee you, they'll think you're going to miss it again, carry on diverging. True, absolutely. It's easier to repair because I repair it at a central point. You have to notice it. I guess documentation, in case nobody mentioned that documentation is... Wait, did we say documentation? Don't really use that. Documenting is very hard sometimes. Yes, indeed. On that note, maybe we'll conclude the round table. Thanks very much.
 There is a project called DysNix which is used to distribute one configuration of our  network.  So you can define with the configuration file how the server interacts.  But Sundar's show come soon to describe this project.  Well then you have the machine deployment which normally probably is the same, it goes  in the same area as conflict management.  We get machines which are donated normally.  So they sit in some data center we don't know of and then somebody puts a center sign  on there and then we do the stuff we do.  I think it might be different with other people, other distributions I know.  I'm not sure about that.  I think Demian probably gets all the stuff donated.  We get all the stuff donated, yeah.  Some of it is for the build system especially some of them are privately owned although  we're facing those out.  But most of the stuff gets donated in many cases by HP because HP is a very big sponsor  of them.  And presumably they get installed by them in person.  They do that stuff, I don't know.  With the GEN2 ones I'd say less than 15% of the machines are actually owned by us mainly  it's a machine sitting in a data center somewhere and we ask them please bring it on a live  CD and tell us the IP and we'll install it.  Actually Nokia runs all the hardware from Mamo which is kind of uncool but that's kind  of an interesting side problem.  How many machines does CentOS have then?  Perch infrastructure?  Without counting all the debt ones I guess around 150?  150.  How about GEN2?  What do you do with such large number of machines?  Mirrors for large part.  So you count mirrors too?  It's a machine.  They are owned by you all the mirrors?  No.  No.  Okay.  Let's rephrase the question there.  How many machines does your infrastructure team directly manage?  Yeah.  Like you have control over 150 that's what I'm saying.  So you have control of those 150 machines?  You manage 15 machines?  You have a control of what?  You can...  I can access those.  Those are all machines.  Okay, they get pulled from time to time with new ones.  If you need a test machine because I want to test out something new, you can get a machine  from that pool to do that.  If you want to do test that as you can get.  But normally many of those are mirrors to feed other mirrors from.  So that's primarily...  Most of the time they are doing mirror stuff but on demand they can be used for something  else.  How many does GEN2 have?  In GEN2 we have about 60 main infrastructure machines at the moment and then another 20  odd built machines.  So that's...  Yeah, well I'm not...  In the infrastructure team as I said, on a blog of a guy who actually is in the infrastructure  team, for Debbie and they were talking about 100 machines.  I don't know where that figure comes from but 100 is whatever.  How many do you guys have?  I think it's much smaller than that.  Up until November, Mamo had one and then they pushed everything out to Akamai.  Yeah, yeah.  No keep pays that bill.  To build?  No.  No, they had another machine that they built internally but because of NDA restrictions  that kind of firewall.  It's a really broken system.  I don't recommend it.  And then you do have the management side of infrastructure.  So where do those machines come from?  Who do I have to say thank you for?  Which machines do I have to look after, read while listening to this company when boom?  Debbie actually had set up an LDAP directory over a decade ago where all that kind of stuff  is managed.  So after we have a file or several files.  You really want to have something that's more internal with the version control.  Right.  Well, we have another directory where not only all users are stored in but also all  machines with some extra schema stuff added to it so you can actually add stuff like who's  the local administrator, who's the donator, who's the, what's the, what's the HKEY store  in there as well.  Who do you find when the machine goes down?  Yeah, exactly.  That kind of stuff is stored.  Does anybody use something?  What's my code?  GDI, no, not GDI.  GDI, something different from that.  I don't know what you're saying.  Being database for doing something bad.  No, no.  You're sort of making that a different system.  This thing where this LDAP directory was set up before I became a development developer  and I became a development developer in 2001.  So it's quite old.  It's working not too bad.  It has issues.  It's something home-roon as is the case for machine-debuted.  So yeah, it doesn't jump.  Okay, and then you have the planning side updates, new machines, how do you do testing,  how do you deploy stuff out there?  Does the neural network now go to centers five completely from centers four or how do  you plan those updates?  Well, the Debian Merit Network is not owned by Debian actually.  It's the people who run a general mirror and oh yeah, we can test that.  True, true.  Those are most of the more mirrors too.  But you have the internal stuff.  Well, Debian probably has the same much as we have pretty long running distributions.  So it's like, well, we have seven years.  Well yes.  So the machine is just a general center for one.  RSA, release of the next distribution plus one year actually.  So that's currently about three years, so not as long.  Do you think there's a possibility to share some of the infrastructure across distro,  particularly mirrors?  I mean, since there are basically two major packaging formats, you would think that a  lot of RPMs are duplicated across the various mirrors across the various distros.  Well, they're not necessarily the same RPS.  No, not necessarily the same version.  They may depend on different versions of libraries.  Yeah, that's true.  But there's probably some redundancy.  I mean, do you think you could have multiple mirrors, multiple package repos?  Yeah, before trying to share binary packages, it would mean starting to share either spec  file for RPMs, sharing macros, sharing batches.  It's only a regular.  I mean, if you look at Debian and Ubuntu, they sometimes have packages with the same  name because they were built from the exact same source, but the version in Debian was  built against this library and the version in Ubuntu was built against that library.  So if you try installing the Ubuntu version on a Debian machine, everything goes to Kaboom.  So don't try that.  Well, you could have separate repos.  Well, yeah, okay, but then you're not really sharing infrastructure anymore.  You're just sharing the hardest, that's all.  Yeah.  And when it comes down to just sharing hardest, I think you may end up getting better results  out of file systems with deduplication.  Because they don't have to know that the file is the same thing.  Well, canonical or at least my shuttle was proposed there and said, well, at least for  the enterprise institutions, let's all have the same kernel.  Let's all have the same glitzy, which doesn't work either, I guess, which something is happening  there between the two.  That would be one possibility to move in a direction like that.  But on the other hand, it's just, well, you have machines, we have machines, we can let  us have machines, because I don't think that you can really share mirrors, because it's  just too dependent on what distribution is on there.  I mean, when we were using RPMs, I won't get any when we were RPMs installed on centers  moment without force.  With force, yes, sure, it doesn't work there.  Oh, that does pretty good.  Keep those pieces.  Yeah.  Oh, actually six pieces.  Okay, that's not really infrastructure, but also has to happen.  Okay.  How do you think about your man?  Do you internal stuff?  Yes?  Wiki.  We also have a lock down internal access only Wiki.  No, public.  Public Wiki and then policy documents.  Well, we have private numbers in there even.  No, we don't have that maybe.  So, where do you share that information?  Well, it's on an internal mailing list that people might have down.  Everything we do in Wiki is just the public stuff, phone numbers and stuff.  We should organize that pattern.  That's why an LLBerge is good at it.  That's why an LLBerge is a good idea.  You just to held up search and you have the information and you can lock that down with  all sorts if you only want your own developers to have that information.  In that, you actually have a web front end as well where you can search and update stuff.  So, that's what we want in our opinion.  For documentation, we actually have three modes.  Mainly, we've got people, authority, bankers with documentation,  for our policy and stuff like that.  We've also got the Wiki, presumably a lot of these documents.  And we have the, in the web page, which is mainly a public CPS repository  that gets automatically converted to web and stuff.  So, basically, I think everybody does it that way.  Yeah, Wiki seems to be...  I just want to mention that within the federal, for example, something like a task which is in the counting system  where people can subscribe and manage their own personal information.  This lets the user decide which information he or she wants to share with others.  You can simply say I want to share my number, share my phone number.  So, between the development cycle, and these information is put in the music query from IOC.  You can also use the passports in IOC.  You can simply put one of them in the query and information there.  And this is a very useful process for direct communication.  What I want to say is which information we want to hide.  Actually, I've started this one.  Let the user decide which information he or she would want to hide.  And to do not generally set up a hidden process.  Because that's always something strange.  I have to throw open process where the users or the developers or members of the community  can either decide if they want to share or not to share their information.  You can do that.  For example, we have an LDSO and all its stuff to be denied something like that.  There has to be infrastructure stuff which is not public.  Because I'm not going to tell anybody the IP address of the servers.  Mostly, yeah.  Don't want to have a list on who's admin on the machine or who's responsible for a machine.  Why not?  Why would that be...  We share on the infrastructure team, on the five and one team.  All the infrastructure team has access to that.  Why shouldn't it be public though?  Obviously, the passwords are not going to be public.  That makes sense.  Little later, it's not going to have to be on there.  Why not?  If they want to, yes.  Some of it is not as explicitly requested.  That's a different matter.  In general, they should be there.  That would be silly.  Some people would be explicitly requested that they are listed there.  But what I'm thinking is, yeah, much of this information,  as long as it wouldn't compromise their security,  they don't think there's any harm in making it public.  And it actually makes it easy for people,  if they see something is going wrong,  it makes it easy to figure out where to go.  Absolutely.  Absolutely.  There's a page on the main website.  It's quite far beyond some others.  Fedora has standard operating procedures  that they publicly list.  Everybody who saw a server go down,  reads the standard operating procedures  and follows those procedures to bring the server back up.  Right?  Why not share them on the presentation?  Yeah, a lot of other distros don't do that.  I think that's a good idea.  It is indeed a major problem within Debian  to get more volunteers to work on that.  Finding infrastructure people is very difficult.  But standard operating procedures, I think, would really help.  Yes.  There are a bunch of procedures on the internal wiki  that we really should publish.  Like how we go about installing a new machine from scratch.  That would be good to know.  You do it yourself?  No, no, no.  Like based on your initial shell,  this is what you have to do to fire up the automation.  Okay.  So I have to largely agree with what these guys said.  It's some sort of the spirit of open source  and it's not, it won't compromise the security.  So you just put it online in a public wiki,  just protect the page from being edited by anyone.  And then you write there while we do it this way.  Any user can read how you do it,  but he won't be able to just do it  because he doesn't have the right credentials  to access anything.  And it wouldn't be really transparent for someone  who stands for open source software  when he just says, well, our internal structure  is hidden from you.  We can't see how we work internally.  So it's a natural thing for me.  And the information that has actually to be kept secret  is just a few access credentials and phone numbers  and the rest can be made public  because there's no secrets among those.  There's an extension request about that there.  We've been discussing it with you at the moment.  Should all users be able to see  negative status of what's down?  Why not?  Yes.  What's a harm?  Exactly.  Because if you go to...  I think it's actually a little bit of a fit to it  because if you try to access something  and it doesn't respond,  then it might be at your end,  it might be at the remote end.  And if you can actually access the monitoring system,  you can see, oh, yes, this server is down.  Then you know you don't need to look further.  You just need to tell the people that, hey, this server is down.  You might want to reboot it.  Whereas if you don't know that the server is down,  you might be assuming that, oh, it's probably going to be running  because these guys are good.  So it has to be on my end.  And you're wasting half an hour or half a day  to figure out actually the server was down.  So it's actually a good thing to make those kinds of things public.  And I don't think it would compromise your...  how do you say that?  Your reputation or anything,  because if your server goes down occasionally,  because I mean everybody who's server goes down occasionally,  it's not a big deal.  But the only project we can find is doing this already is Missila.  Debian Semi does that.  Not really.  There's something...  some website where you can actually go.  Yeah, the Missila one is not exactly public.  Well, you have to do it.  Yeah, the same goes for ours.  And it comes up with a password, which is guest or something.  Yes, yes, yes.  So it's semi-public.  But I don't think there's any harm in having it public  as long as you don't put...  It depends on the TES.  The Wikipedia Foundation, the Wikia guys,  publish both the Nagios and the CactiGraph in public.  Okay.  I think a part of the Yahoo open source group do as well.  But these are both commercial companies.  Most distributions publish the mirrored stuff.  Most distributions publish the starts from mirrored stuff.  Yes.  Which certainly makes sense.  Because...  Also because those distributions you can't control  all the time.  I think the hard part is to find what I mentioned before,  to find new members for the infrastructure.  And there's something I've already discussed  with you.  How do we want to decide?  How do we want to make sure that one who got the knowledge  to manage the machines, who is totally trusted person,  and who knows what he or she does,  will get into the infrastructure.  And this is something I'm not yet sure how to filter.  I haven't found a way on filter that these aspects.  Because what... how do we want to filter?  That's one of the old infrastructure members,  for example, have to know the new infrastructure member  personally, what if he or she comes from India  and nobody knows him or her,  how can you make sure that you can trust him or her?  How do you handle these things?  How do you...  I think the trust thing is the largest thing.  Because seeing what people can do,  they can vocalize themselves and say,  well, what you're doing there is playing wrong,  because if you want to do it correctly, do it that way.  You get from people who know the stuff they're doing  and they're interested in doing stuff.  But I think the trust thing really goes very, very deep.  In Debian, trust problem is basically solved  by saying that only Debian developers  can become infrastructure people.  And then you have a whole trust...  You have to develop a process which is rather harsh.  Well, harsh is... I wouldn't...  Do you know that I'm involved with that developer process?  I wouldn't use that word harsh, but I understand...  It's getting better.  It's getting better. It's getting much better.  I didn't say it was bad, so...  You said harsh.  You said harsh.  It's the same thing.  I know what you mean.  Is anybody that's joined Debian explicitly to work on the infrastructure team?  Or did they join as developers?  Probably.  I mean, I don't think so.  I'm not entirely sure about that.  We did get one person who joined just to document...  documentation, not packaging.  And in theory, there's no problem with joining  just as an infrastructure member.  We don't have a procedure for that yet.  If somebody were to suggest that they wanted to do that,  they would have to figure out how to come up with that.  But basically, if you want to be part of the infrastructure team,  you have to be Debian developer.  That's basically our approach currently.  So by definition, then you have to become a Debian developer  before you can be considered to contribute that way.  Become a Debian developer, you either have to do packaging work or documentation.  And people don't know you, don't they?  You can get the code for Debian build systems  and for DAC and for rep repro and all that stuff  and you can submit patches and that's one way to get in.  And I'm sure if you did that, you'd get sponsored  and it would be a very easy way in.  So in that regard, it's all open.  They're not necessarily going to let you log on to the machines  and start hacking code, but the code's all available.  Yeah.  Do you have any of the old infrastructure team  needs to know the new infrastructure members personally?  Not personally, but you need to know who he is  and you need to have...  Right, right, right.  I mean, not personally as in person,  but in the case of the GEN2 stuff,  there's two cases.  First of all, are you explicitly joining as infrastructure?  I've only had, I think, three people from that  and they all happened to work at OSL's open source lab.  So they already knew when I infrastructure that was there  and that's why...  They've grown up by trying machines already.  That's why they were joining.  But then you have to have been a developer  for at least three months  and somebody else that is an already an infrastructure member  must vouch for you.  That's the same as yours.  What do you mean? You have to have some e-builds  and operations in the system.  No, in our cases, even more generally,  moderators on the forums...  on the forums,  the GEN2 Wiki is not in-house.  It's actually an external entity that runs it.  So it doesn't really count.  But there have been some knowledgeable people there  that we've said, you know what you're doing.  So they're sort of documentation, but not really.  Right.  So the next logic is that  it would be kind of a mentoring process  to let the new maintainer know how to treat the problem  because they might be not very well-documented,  some kind of special.  Is there something like a mentorship?  Isn't that the only way to do it?  Because all the build systems are different.  You know, knowing how to build packages on Debian  is not going to help you know how to build packages for CentOS.  And the infrastructure is catering to those tools.  And the infrastructure is probably custom-made.  So this isn't probably the only way to say,  recruit people and mentor them through.  That's one of the reasons why I wanted to have this round table here  to learn how others do it.  And if they're probably, maybe there are processes  where we can say, well, let's do that together.  Or let me learn from you because you seem to be doing the right way.  Well, there is a name, right or wrong way.  But distribution has its own rules.  I think that these are the causes, the causes like mentorship.  There's not one in any kind of distribution.  And I'm listening to all of you.  And it's really, really interesting for me.  Because I'm a user.  And I notice that there is an explanation of the role  of every person on the interest action.  Debian said that you have to be a developer.  So this is the top.  I understand?  What should we do?  The top, as in?  The infrastructure.  So you have to...  WN develop is just a person with the ability to upload packages  into Debian.  So we can actually...  And convert.  To generate, there is a specific role of a member.  Not only in Debian, on every distribution.  And I think this is really, really wrong.  I think that every person on the distribution on the interest action  has to have a specific role.  And not to do all the things.  So, okay.  Well, we do have that.  We do have that.  It's not...we mentioned it, but it doesn't mean we don't have it.  At least in Debian, we've got people who do mail lists.  We've got people who do system administration.  We've got people who do FTP master, which is the mirror network.  People in charge of the bills.  And then we have a technical team who governs all of Debian too.  So there's a lot of very well-defined roles.  That's totally true.  And we're crazy not to do it, because otherwise,  some people would have to do everything and it just doesn't work.  That's a problem.  And they're also not actually interested in doing everything.  Right.  They're interested in one small thing.  Like we've got two guys that are very interested in our mirror side.  And they run it pretty damn well.  I think at the last year, I said that we have to count the pass-on.  And they mentioned that if five or six people are killed now,  the picture is dead, or some guys are going to continue.  I think this is something very specific on the role of the persons  who are not actually interested in it.  Yeah, that's why we posted the set for the DT-85.  Yeah.  Documentary.  I think the only thing that we can do to make over here  which sort of counters the argument for the gentleman there's made is that  in a volunteer-based organization,  folding people down to specific roles creates black holes.  Yeah.  Unless you've got a significant in large pool of resources,  you cannot have people working within silos.  Because if you do, you're creating black holes  and you don't know how you're going to get your job.  In Devin, I'm guessing the numbers are not in single digit.  I'm guessing there's more than nine people doing infrastructures.  It's probably possible to have a couple of people doing it in single digit.  Much more.  If you count builds administrators and system administrators  and FDB masters and list masters,  I'm sure you'll get in at about 30 or 40 people, something like that.  So it only says, did you infrastructure team?  Did you enter 27 people?  But it changes.  I mean, you're pointing to a very fundamental fact  that these are volunteer organizations.  You cannot tell somebody, are you going to work from nine to five this week?  You can't say, well, can you even dedicate 40 hours a month?  You have to sort of take what they can give.  And that's a fundamental problem.  I mean, and when there's a downtime or a break,  and we've had two in Devin,  then yeah, these people get, need to dedicate a lot of time that they may or may not have.  So it is something to worry about, actually.  So by the way, one more thing that I noticed,  of course it does not have to be twice specific to here,  is that developers make very bad decisions.  Oh, yeah.  Very bad work, too.  But we do find that the people who tend to be sysadmins  don't tend to have a lot of packages in Devin, in my opinion.  They do sysadmin and they may have one or two packages,  maybe something that they just happen to use.  But indeed, what's up they use as sysadmin?  Yeah, exactly.  That's what I mean.  There's something that they happen to use for their sysadmin work in Devin then.  But usually they don't have a lot of packages,  at least where I've looked up.  Is this because they spend so much time sysadmin  that they don't have time to do packaging work,  or is this some other reason I don't know?  No, I don't think so.  One of the two ones is that they're not interested heavily in the packaging process.  Right, yeah.  Developers tend not to care about packaging at all.  They see it as a necessary evil that they'd like to get out of.  When you look at Python and Ruby, they both have pretty...  Are you calling those packages?  So yeah.  I've heard a lot of people saying that Ruby has some of the most brilliant people working on it.  And Gens is a very fine example of how a brilliant community creates a ship product.  Yeah, definitely.  But these are problems for the distributions,  because I hear from developers,  pro developers all the time, they say,  oh Devin's too old.  Well you're using Devin's stable.  It's not old, it's stable,  you're testing.  Oh, but that's too cutting edge.  No, not unstable, testing.  But that's an issue for all distributions.  What's your stable period?  I want to patch it to 2-2.  It's Fedora.  What's that?  You don't get 7 years out of that.  13 months are enough.  What?  13 months are enough.  13 months.  I'm standing in the middle of a machine.  But yes, there's a problem.  Developers don't want to do packages because they want to develop.  Right.  So they don't want to do packages because they want to do  but they depend on the packages.  Because I don't want to system the machines which aren't packaged.  Right.  Thank you.  So somebody has some packages.  Aren't most developers in all distributions doing mostly packaging work?  Most of them, Devin at least I know.  We have it rather easy.  Packages are done for us.  You guys just steal from me.  Maybe you're right that you've grown too, sir.  Yes, a little bit of that.  Now, indeed, of course you can't see infrastructure, software, it gets confusing.  You've got things like your package manager for Devin as the package which we develop ourselves,  not we but some of us that develop ourselves.  And we do some other software that we need to do.  Most I think the developers for distribution will be packages.  I think Susie's open build service will present a very interesting challenge  because I think that it can build both devs and RPMs.  And if you have an automated system, you're taking away the human element of building packages.  And that's going to...  You have to have rules for the Devin packages, you have to have a backup for the RPMs  and you have to have a different backup for Susie RPMs in there.  You have to keep in mind the difference between distributions that are not just only packaging formats,  they are more than just different format for packaging.  Precisely, and when you have an automated tool that builds packages,  then you don't... you're hopping over a human and there's a difference in quality in the package.  So does it have to be a human?  Does it have the internal knowledge of difference between Devin and Red Hat, for example?  I don't know if open Susie has this, but I do know that there are distributions  that use open Susie's build system to build their distro.  What you still need to do is you still have to know, some human has to decide how do I configure this package,  do I enable this feature, do I disable it, do I build two packages that...  that's disabled and one that's enabled or do I just... or how does this build system...  does this package have a plug-in infrastructure, do I ship the plug-ins in the same package,  do I split them in separate packages? That's all... that's things that a human has to do.  And after that, I mean, you don't compile it yourself.  After that, after that site, you just run your compile tool and if a bot does it  or if you just type one command to compile, there's no huge difference from me.  So decisions still have to be made by a human who puts these specifications into the build system.  I mean, even if it's not a spec file, it's going to be a configure file or whatever...  It's any specification that's helping you build...  It's just a specification of how to build something and indeed, whether the bots or a human builds it,  if I type make or the bot types make, the result is going to be the same, hopefully.  Yeah, well, I do maintain a build system, so I do say hopefully.  Yeah, I mean... yeah, anyway.  I mean, for the open-source of build servers, it's of course the case that every package has a maintainer  and more specifically a maintainer group.  So we divided up the packages into projects where you can have groups of maintainers  and it's also true that we build for WNN and RPM-based distributions.  And I think that's a chance because we agreed, we just agreed that nobody really wants to do packaging work.  So we don't want to, this is a bad start for us, so we have to seek out specifically for packages.  Which I don't think that's the most effective thing to do in our world.  So maybe not the studio.  You still have to know what the, for example, naming convention is.  Yes, of course.  The packaging is extremely a difficult thing, but it's also true that most people come in as developers  or as users and then they have to be turned or turned and sell them to packages.  So you mean enabling users to build packages for, I mean, enabling people to collaborate on packaging  might be how to reduce the pain of packaging.  Would you put those packages officially in your distribution?  Would you take those packages by people you don't know and put them officially in your distribution?  Of course not.  No.  Is it okay for you if you create a service where users can put these packages into one centralized searchable repository?  That's OBS.  Or in ARCH?  We have some public and searchable repositories, but we always have to review process.  So we always have to maintain a group and these people are responsible to review this.  Yes, of course not include this in the official repository.  I think we just do, okay, sorry.  Yeah.  About the packaging being the hard work, do you think that part of this is that most packaging formats are hard for users to write?  Like spec files, e-builds, I don't say that one of them is harder than another one, it's not the other one.  But do you think that it depends on the clarity and simplicity of format to improve number of users who would like to write it?  Yeah, if you ask me personally, I would say yes, it's a tough and hard job to package, no matter if it's spec or dead end.  I would appreciate a more simple thing that covers all, for example.  But I know that it's completely often future because it's hard to do it.  I'm rather fascinated by all these statements because we actually have exactly the reverse problem.  So over 90% of people in ARCH just do packaging.  So people package and nobody wants to do the other stuff.  So there's people making packages and so on and then there's nobody taking care of that mirrors are in shape and then this task is given to someone and he gets lost in some work and disappears or anything.  And nobody wants to do development except the package manager, there's an active development but all other things that need development, well, they just get stuck and get lost and so on.  But packaging is actually the part that has enough people on it.  But the rest is just, well, it's just not improving as fast as it should, considering the number of, increasing number of users, increasing number of packages and so on.  So it's just a reverse problem of most people.  I don't think I said that.  What we have in Debian Mainpalm is not people wanting to develop because that does happen although there are some cases where we need some more people.  But I think the main issue we have in Debian is that because the effort for development to maintain is about four or five packages, people start complaining if you do more than 15 or something.  Yeah, except, I think I've got my name on a couple of hundred.  Yeah, exactly.  But if you do more than 15 or 30 packages, people start really complaining.  But the main problem we have with that is that people fail to see the big picture and I think that's one of our main issues for packaging.  I mean, it's very easy to get focused on those four or five or six packages that you're doing and not really care about what other packages have in track with yours and stuff like that.  So in Debian, packages, in average, have a small number of packages?  Small number of packages, yeah, absolutely.  There's one of the team maintainers.  Is this encouraged or is it just an effect?  That's a difficult question.  It used to be the case that every package only had one maintainer.  And every maintainer only had up to ten packages.  And it's still the case that there is one main maintainer who is the, who's the distant maintainer.  But more often than not these days for large things like KDE or Coke, no, the maintainer is a mailing list.  It's a group of people and then there's people who are listed as upload or co-maintening basically.  So we have team maintain packages and if you look at team maintained packages, then you do see that people are often member of a team where the team maintains like fifty hundred packages.  What I meant with the thirty packages is if you're listed as the sole maintainer of a package and you have more than thirty then people sign up.  But in teams the issue is not as big because then there's more people in the team who can actually look after the packaging care for it.  Drifting a little bit into the packaging.  Yeah, sorry.  It belongs to you too.  Yeah, it does.  It's kind of at the heart.  I'm sorry that the HR talk is directly besides this talk here.  We were talking about HR and HR is a new resource.  Yes, but we were talking about that all the time at the moment because probably people get into distribution and people get into teams and distributions.  That does have to do with the structure too, sure.  Is there any distribution here that pays for the infrastructure?  It's a kind of pay.  If I need to turn servers, I'll go somewhere and buy turned servers.  Basically 15% of our GEN2 machines are owned by the GEN2 Foundation.  We do not have a pay for any hosting.  In Debian, the case mostly that things like our main servers, if we need them, we just ask B.L.Garby who is Linux CTO at HP and he just hands us a bunch of servers.  So that's not the issue.  In the case of things like architecture specific hardware such as tool systems, that's harder because HP doesn't have all of them.  That's about it.  We need to be prepared to which happens with the other architecture quite often.  I guess we are the only ones who buys stuff that is a little specific because Mandriva is both a company and Mandriva Linux is a community project.  I would guess for Fedora the same would be true.  It's very datastic by an ACT하발ạt.  also has a lot of connected machines.  And I know there are at least a dozen hosting companies  at Dell, for example, and actually even  who have donated equipment which is hosted in  construction.  And I think also the interface stuff  have got to the cloud and Amazon hosts.  Oh, that one.  OK.  And they're there.  And it's more spondering ship between us.  That's the other reasons that we mix up all.  This is a very brief question.  Do you feel you have an over-concentration of machines  at any one single sponsor that could actually threaten things  if that sponsor or its own went away?  Good question.  I don't think we have that.  We have two or three core machines and with one sponsor,  that could be a problem.  You're a commercial organization.  CentOS is, though.  We're not commercial.  You're not.  We see it's not commercial.  We see it's community.  But your target is certainly business sense, as opposed  to, say, ordinary users.  Although we want people to say, well,  I need the stability of an enterprise distribution  but don't have the means to pay for that.  But I don't get the support.  And I know I don't get support for that.  So no, not really.  If there are business that want to use that, yes, sure.  Many hosts use it.  Not here or in Europe, probably, but in America.  Lots of hosts are using CentOS.  Here you get more, Debbie and Osuzi, I guess.  In Europe.  Or at least in Germany.  I think just to come to think about this,  I'd sort of like to reply to your question  about the single point of view.  I think the problem that we have within CentOS  is something which is completely on the extreme other end.  And that we would really like to have more than one machine  hosted at the same place.  I think we've only got two sites where you've got more  than one machine.  So now if you take the number of machines you've got  and think about the fact that you don't have more than two  machines in any one place, you'll see how many different  locations we spread over.  And every person has their own process for support.  Everybody has a new ticketing system that you have to improve.  We don't have that in Debbie, actually.  We've got a number of machines hosted at an HP data center.  And then some at Brain Food, I guess, though.  I'm not sure that's in Texas.  So we've got like some 10 locations.  And then of course the old small ones.  Houston and Foxconn, it's orange.  Sorry?  Houston and Foxconn.  Three places for folks.  Right, right, yeah.  Houston and Foxconn.  Right.  Yeah, but I was talking about Brain Food, which is Texas.  It's a small company.  They also have some.  They used to host masks in the open, some other things.  Right, we're generally, I'd say, 50% of our machines  are between four big sponsors.  They just have machines there.  Right.  Is it all problems we can't do failover?  Because machines aren't sitting beside each other.  Could be anything.  If those machines don't see each other,  that could be anything.  The data center's on fire.  We can't do failover for other reasons.  But yeah.  Yeah, OK.  There's actually a larger problem  than have these different locations.  Like the G2 forums, it's the most popular thing.  We keep having two increase the amount of hardware for that.  If you're doing database replication and stuff,  latency is a massive problem.  You have to put all in one location.  Right, yeah.  So at the moment, that's four machines at Hives NL,  the Dutch social network.  They said, you guys can have four old machines.  Oh, sorry.  Forums was at Hives.  And now, Bugs Miller is on those old machines.  And forums has moved to Gossamer, Trades, and Vancouver.  Because forums didn't fit on the old machines anymore.  Right, right.  So you had to ship a whole bunch of data.  That's why it sets more things, mainly in this.  Because I didn't come in the forums.  We have forums.  We have more forum subscribers than mainly subscribers.  But I think we have more active mainly users.  We have some more users.  We do forums.  We do forums.  They've been not met, but they've been not all.  So it's not official.  That's the distinction.  They've been not met and they've been not all.  Every they've been not all, they can set up a  they've been not all.  That's post name.  And they've been all.  Orc is just maybe something interesting.  So if you have them in the orc, that can only be modified  by the Degasus administration team.  I think in the last time, look at the Gen2 forums versus  the mailing list.  They're getting 5,000 forum posts today.  And the lists are only getting 3,000 or 4,000.  I think we're kind of the opposite.  The lists get a lot of posts, and the forums are kind of dead.  Well, that's not true.  The forum was, Yerun and Teess had set that up.  And then they really weren't in uniform thing.  But they set it up because people were.  Some people were.  And at some point they got compromised.  And that was very good.  And then they disabled it.  And then they came complaints.  And then somebody else volunteered to maintain it.  It was more intuitive.  I think we have more forum traffic than we have.  But I don't know how much forum.  But the Degasus mailing list is possibly huge.  Your mailing list is way busier than the Gen2.  You have 12,000 dev-in-developers on there.  Last but people who are interested.  Right.  We've got massive amounts of posts today.  And that's not even counting to panel.  For each 90% gets thrown away.  I think we have 100, 200 mailing lists  post a day on max days.  So that's not that much.  I get that much.  I'm merely, wait.  I don't want to think about it.  So here's another interesting statistic  to be in about location.  As of last Sunday, we were in 21 different countries.  But on Monday, we lost to Xi and Russia,  which we don't know if you want to get back again.  So we're down to 20.  But we are talking to people with two different data  centers in China.  So if that goes through the next week,  we may be back up to 21.  Yeah, or another thing that's, we have a machine in China.  In China.  Is that an amnesia?  Is that a great firewall?  Actually, it is an issue because both of these data centers  have multi-gigabit connectivity within China.  Right.  But one of them, and this is the truth,  this guy actually only has 128 kilobyte length  for the rest of the world.  How?  Because it is the maximum packet filtering cable  that the government of China is going to give them.  It's got the capacity, but it can't push more than 128k  at a time because of all the different.  Maybe throttle the traffic to that rate.  So the problem is he started syncing his main data  in the Google Lodge.  He hasn't finished yet.  Ha ha ha.  Wow.  We're at the same thing in Africa.  He supposedly has a, well, OK, only 10  and we're doing 22 Africa or into Egypt, of course.  But we couldn't see the machine.  It was just the initial r-sync wasn't done after four days,  after five days.  Just didn't finish.  So there's an open office update.  He's always syncing for three or five days.  He might have better called, he might have better  connectivity into Egypt or into neighboring countries.  But it was out.  But it was blocked.  You don't get the stuff there.  So, OK, another question I would be interested in.  Who actually has infrastructure in South America?  Besides Brazil, sorry.  I think, I don't know.  Do we have Brazil?  I think we have two Brazil machines.  Do we have one, please?  Let me check.  OK.  Or in Africa?  We had one in Chile.  Africa?  We had one in Chile.  We had one in Chile.  Anyone else having machines in Africa?  We're looking it up.  OK.  You can't see no more machines.  So I know we don't.  Slash.  What?  Egypt is the sole machine.  Asia is another point.  Well, yes, Taiwan probably, Singapore probably.  Yeah.  South Korea, I think there's one stuff in Asia.  I don't have anything like that.  Hard to say because it only says the name of the sponsor  on the location.  So we have to check all the, it's not that far.  Sorry.  Four minutes.  OK.  You have to decide who wins.  We actually have a break afterwards.  So we can run a little over if we want to.  Yeah.  But thanks.  Yeah, we're supposed to finish it in three minutes.  I show off hands, who wins.  What was that?  Who wins.  I win.  What do you win?  You get to maintain your server?  I don't know.  No one's got to win.  Let's do a great one.  The next stop starts in half an hour.  We're in Columbia.  Yeah, we are.  We're in Columbia.  We're in the same one.  Well, yeah, we, I think everybody has mirrors all over the  world.  That doesn't count.  So with the Chinese thing, we should finish that discussion.  What we're working on is that Hong Kong has a lot of  capacity.  So what we're working on is a couple of USB hard drives.  So we actually get it into Hong Kong and we ship using  post-school mail messages.  So it might be funny.  It actually sounds really funny.  But what happened is because we don't have any seats in China  and this machine that we run ourselves, there are actually  three or four external mirrors who mirror us in China.  We estimate that we've got six digits of users in China.  But these guys want to pick traffic.  But these guys want to hit update mirrors.  In other parts of the world, they generate a phenomenal amount  of traffic out of China.  And a lot of mirrors don't like it, whether it's because of  politics or whether it's because of policies.  We've had mirrors going around.  I think I set up a mirror for St.  Paul's in Germany.  I don't want, how much was it?  You got four pedabytes of traffic in China and one in  China?  The little traffic was because it didn't have a  model of I.O.  on there.  They're getting hit from some download software, there's  some download manager, which hits the mirrors in parallel.  And if you kill the IP address, it just comes back with a  nice IP address, just because most people can't get the  software in a different way.  So they try to spread as many mirrors as possible from as  many IP addresses as possible.  That seems to be some Chinese-based download  management software.  I've never heard of that before.  DDOS attack.  You guys have well-defined policies as to how often users  should hit mirrors before they get banned?  Because we don't control most mirrors.  But you guys are seeing mirrors.  OK.  With the Async Mirror site, there's actually two  separate tiers.  The core set in async.engin.org is strictly in  house.  We guarantee the content on those mirrors.  We validate that.  And they typically run a few minutes ahead of the  community mirrors, because the normal community mirrors are  well maintained and very fast.  All the actual core ones actually serve directly for  ramps.  It doesn't touch this.  Wow.  How large are your Async data?  That Async said, it's 280 bytes of data that you have to  watch out for.  If you don't watch out for 280 megabytes of data, I'm  mistaken.  If you don't watch out for high nodes, it loads up to  about 700 megs.  We at the moment have our internal sync machines open.  If you know which machine it is, you can sync from there,  which we reserve the right to publicize your machine there.  The Async panel says, well, only sync from here if you are  an official mirror.  It's not that bad.  It's on the house.  In Debbie, I think it's all based on IPX and lists.  So if you want to be a primary mirror, you talk to the  mirror admins, and then they give you an account.  And then you get it.  We have that for DVD at the moment.  So if you want to serve the DVD, find another mirror who  already has the DVD or RRs.  And then they work with SSH triggers.  So once we finish you, SSH can do there, and it styles the  process, and you can disconnect again.  So it's a really wonderful model.  Yeah, we do have restricted mirrors for the community  mirrors to access.  The rest of the gentubers are also in.country code.  Those are all the community-run ones.  They have a specific mirror that only they can access.  They have specific gates.  So does that mean that all of your R-sync mirrors have to  have a dedicated IP?  No.  Because if it's R-sync.country code, you can't actually be  who's R-sync.  Unless you just need to look for a part.  Yeah.  We do have a bunch of them that they regularly send us the  IP data every 60 minutes.  If you haven't seen it, we expire it out the list.  So kind of dynamic DNS.  Right.  Mocked up.  You're not going to slide.  OK.  I had some more slides, but we talked about that between  some of the last one.  In that case, there is a wrapper called PowerPay that  checks just the mirrors that you have on the mirror list.  And then download a piece of program from one mirror,  another piece from another mirror, another piece from  another mirror.  So you don't download all the software.  And it is on the bandwidth on a specific mirror.  And it is based on ARIA, too, for the one.  And it's really fast in the...  Well, I think it's a good idea.  Like you, though.  You should have been looking at the mirror brain talk just  before this one.  Yeah, exactly.  You should have been looking at it.  I should have been showing it.  Yeah.  Yeah, I had to.  Yeah, the talk right before this one was talking about a  program mirror brain, which actually does what you just  described me.  No.  I think you should have been talking to the city because  I had to with the metaline.  Yeah.  Yeah, that was coming in the previous talk.  Whatever.  There was a reason why I tried to block them together.  Yeah, there was a reason why I didn't want to do a mirror in  Ashkark here because there's a whole kind of different  problems, I guess.  Right, yeah, sure.  Sure.  I was going to say something.  You know, you were talking about bringing people in to the  infrastructure things.  Does anybody do interviews?  Or tasks or do you do some sort of a selection based on  something like that?  Definitely when new people join the infrastructure team,  there's a couple of things off the top of the easy items to do  on the to-do list that here are something that needs doing.  Yeah, I think if you start selecting people, I think you'll  find out most usually you've got too few volunteers as it is and  if you start selecting them, you end up with even less and then  you've got more age problems.  It's usually best to give them some low-hung fruit and see how  they do and then if they do well, then give them some more  responsibilities.  I think that's the best way to do it because otherwise you just  start to get people.  And if they do the low-hung fruit and either mess up or it  takes way too long or something else goes wrong, then you know  you don't want to give them anything else and you move on  and they move on and they've had a chance and you've seen it  didn't work out.  So I know I'm done at that point.  I think that's the best way to do it, basically.  So what, as an example, what would some of these tasks be?  One that comes up pretty frequently for Gen2 is actually  the initial setup of a machine just prior to firing the  automation.  In our case with Gen2, it's that you log into the machine,  you're going to end up, you have to be partitioning the  machine's menu.  That's not part of the automation just because the machine  tends to vary a fair bit.  So you have to be able to do that.  You have to partition the machine and set up the LVM.  And then you have to read enough of our internal CF engine  documentation and setup to add the new machine to CF engine  so that when you fire the automation, that the  automation is going to work.  And they get to read the logs and say, CF engine failed to  add this machine.  Why?  Because you forgot something in the configuration.  I'm not the last person to receive the information.  Yes.  I'm not going to move to puppet.  I don't know how good CF engine is with role  isolation where puppet and chef and VCFG are really bad at it.  So if you have to give somebody access to CF engine and  manifest behind it, that person could pretty much do  anything on any machine anywhere.  Yes, I know they can change a whole batch of stuff on  almost any machine.  And then you boot them out and you don't talk to them  anymore?  In our entire time with the Gintro infrastructure team,  we've had a single person that went abusive and he was  dealt with within six hours.  Exactly.  That's a question I have.  How do you throw people out?  Not that we have a luxury at the moment.  We've been able to throw people out.  We've had that problem within Debian and we've actually  booted a few people out mostly for not being abusive in  infrastructure kind of things but more socially abusive.  Yeah, I've been in Mexico City some years ago.  Yeah, that was one case.  That was a very well-publicized case and there was  actually a bit like on the spot and figured out how to get  built.  So I guess key distribution and possible distribution is not  automatically so you can revoke on key success.  Yeah, yeah.  What's behind my question?  That's what I said.  All right.  Sorry.  What we have is we've got to move on to the other actually.  So you just remove them from the other actually and your  account is gone within a few minutes actually because it  actually contains a local cash of the account.  So 15 minutes later your account is gone.  And for uploads, there's a central GPG key server which  contains your key if it goes out of there.  It sinks into the maintenance software and so a few minutes  or a couple of minutes.  Yeah, kicking on a regular developer is effective within a  couple of minutes.  Right.  The specific case was we're having to boot somebody out of  the infrastructure team.  Right.  And just the basic automation side of it is within half an  hour but all this stuff you have to go and check by hand  since the person has had root access to the machines.  Right.  It's 6,000 so of work.  Right.  Yeah, sure.  That takes a bit more.  What do you do with people that are in the team, whether it's  in the infrastructure team, whether that's half responsibilities,  they've worked well but certainly during their life, they  don't have a priority anymore to work on that.  They just don't hear anything.  How do you process getting those people out of the project  and getting people in because often they don't take  responsibility by saying that they don't have the time and  they just disappear?  I'll give a quick answer to this one but one of the other  junior two guys that's probably in the remover was actually  just going to be discussing it more fully.  We have a list, a mail that goes out twice a week to an  internal team that we have called the retirement team.  After 60 days with no commits, we'll send you an email and  we'll ask you what's up.  And if you don't respond to us within another 60 days, you're  gone.  You wait 60 days.  60 days to notification and you get 60 days to respond.  120 days.  120 days, four months.  Even if all your software is bug free.  Pardon?  If your software is bug free.  Well, you respond to your response.  Within Debian it's slightly less automated.  We've got a team which we don't call retirement.  We call it the MI-18 but it's basically the same thing.  What we do there is if somebody's bug pages is like  many bugs there and nothing's happening or stuff like that  then you get a mail from the MI-18 what's up and a few more  emails if nothing happens.  It does go a lot faster than 60 days from that first moment  but there's a few reminders because maybe the mail didn't  arrive or stand full to go in between stuff like that.  We'd actually send a few more emails.  Yeah, sure.  Anyway.  60 days and 60 days.  Right.  So, it needs eventually if we don't get any response or the  mail bounces which happens.  People forgot to update their email.  We actually give people local email accounts.  Right.  We do love you that you've changed it.  Let me know.  Mail is forwarded usually.  So, if you don't update that then the mail bounces then you're  gone.  In the case if people are not active and don't respond to the  MI-18 then eventually they get thrown out.  Yeah, particularly if they've been around for a long time  the mail starts bouncing we will make an attempt to contact you  via other means.  Yeah, so will the MI-18.  They will try but if you don't find them then it's gone.  But this is in case that the people are off the internet or  something like that.  But what in the case that they're still around, they're still  interested in helping or when they don't have the time,  answering email is easy.  Staying in the team is easy.  You know, because it gives you respect to stay in the team but  they're not really...  What you mean is they're still active, semi-active on the team  because as much as they should be...  They're active within the community but not necessarily the  team.  Right, some of that.  But you can't go...  Yeah, and we separately encourage that, we encourage the teams  to trim out their dead weight per se.  I don't think...  Most of them do.  I don't think we have a process for that in Bebion actually.  We don't.  And maybe we should because...  Yeah, the Perl team has probably 150 members.  But at most 30.  Yeah.  Oh, yeah.  Yeah.  Yeah.  Yeah.  I think the sort of thing is probably easier to quantify  this code or this activity because you can see it should be done  eventually.  Yeah, I would like, within the infrastructure...  I can see that you're making commits to the infrastructure  repository that controls those.  So should we sum up, maybe take one more question and finish  out so people can get something to eat or something?  That's a question.  From your experience, what would be the most important  lessons for a new distribution startup?  That's a good question.  Are we going to take five minutes?  I think first and start.  Because many distributions probably started by just one guy  or two guys, just started something and then it grew  and it grew and it grew and in the end you end up with a pile  of packages and a pile of infrastructure and some people  and nothing is organized and you sit there and make...  No, don't give in.  Don't give in to your processes.  Nothing is documented, nothing is organized and then you think,  now it doesn't work anymore, it's just stuck, it's just in  some kind of a deadlock phase.  So you just have to think, how do we do it, then do it and  then just say, well, let's start by choosing a package format,  let's start by choosing some basic packages that I've started.  Basically what you're saying is you need a plan.  You need a plan, yes, you need a concrete plan.  How do I organize my distribution?  Because certainly we had that problem.  There was one guy showing here that more people and more people  and more people and suddenly the founder of the project  and the main lead and the only one who had root access  on every server suddenly went MIA and the distribution was  basically dead so there was a package and they were new releases,  there was nothing really new for over a year and still,  until he came there and said, well, I don't have time for anything,  just someone else do it and then we started, well,  we need some processes, we need some things and I think it was  in 2005 after four years or maybe 2006 after five years  and maybe that's a bit late.  Another thing, think of automation early because it's easy to do  this administration and you have ten machines.  As soon as you have twenty machines, you're completely lost.  It's easy to set up automation when you have ten machines  and it's much harder when you have...  It's harder to get that automation when the machines are already running  because they might look completely different from what you're running at the moment.  They will look different from the one you get.  I want to be positive about that.  Well, the thing is, if you don't do automation in the beginning,  then your machines will diverge.  You will accumulate differences and when you implement automation,  when you will eventually implement automation,  you need to work away those differences and that makes an order.  Even with automation, I guarantee you,  they'll think you're going to miss it again, carry on diverging.  True, absolutely.  It's easier to repair because I repair it at a central point.  You have to notice it.  I guess documentation, in case nobody mentioned that documentation is...  Wait, did we say documentation?  Don't really use that.  Documenting is very hard sometimes.  Yes, indeed.  On that note, maybe we'll conclude the round table.  Thanks very much.
Infrastructure_round_table_gUm6OkcE6tQ_0_515.png Infrastructure_round_table_gUm6OkcE6tQ_516_587.png Infrastructure_round_table_gUm6OkcE6tQ_588_15191.png Infrastructure_round_table_gUm6OkcE6tQ_15192_31463.png Infrastructure_round_table_gUm6OkcE6tQ_31464_32171.png Infrastructure_round_table_gUm6OkcE6tQ_32172_71711.png Infrastructure_round_table_gUm6OkcE6tQ_71712_78755.png Infrastructure_round_table_gUm6OkcE6tQ_78756_81431.png Infrastructure_round_table_gUm6OkcE6tQ_81432_81491.png Infrastructure_round_table_gUm6OkcE6tQ_81492_86879.png Infrastructure_round_table_gUm6OkcE6tQ_86880_90539.png Infrastructure_round_table_gUm6OkcE6tQ_90540_91403.png Infrastructure_round_table_gUm6OkcE6tQ_91404_92483.png Infrastructure_round_table_gUm6OkcE6tQ_92484_94391.png Infrastructure_round_table_gUm6OkcE6tQ_94392_95027.png Infrastructure_round_table_gUm6OkcE6tQ_95028_95087.png Infrastructure_round_table_gUm6OkcE6tQ_95088_97043.png Infrastructure_round_table_gUm6OkcE6tQ_97044_100674.png