FreeBSD 8 to 10
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Wolff, Nick
This talk will discuss OARnet's (regional nonprofit ISP) ongoing transition of 100+ servers providing network services and management infrastructure from FreeBSD 8 to 10 and the work done to make this possible. The transition had two major goals: Integration of modern changes and updates with legacy modifications focused on minimizing custom work; and raising the already high standards for uptime and reliability of services our customers have come to expect. It was very important for this transition to find the right balance between progress and tried and tested designs.This transition had to find solutions for limited physical access limiting install/provisioning options, limited personnel(1 person), all software being compiled from source by hand for legacy reasons and more. All of these were overcome with careful engineering and integration of parts of FreeBSD, PC-SysInstall for installs with modifications for provisioning, Life-Preserver for backups, ZFS filesystems, and many more technologies. I will be discussing how we (OARnet a regional nonprofit ISP) made the transition from FreeBSD 8 to FreeBSD 10 which involved touching many systems including the install process(PC-SysInstall), system provisioning(custom), backups(Life-Preserver),and others. The transition FreeBSD 8 to FreeBSD 10 was accomplished by first identifying a mix of past decisions that had been made starting in the freebsd 4 era and making stops at 6 and 8. Then sorting the decisions made and carried forward into their underpinnings which mostly fell into 4 categories: inspired designs obsoleted technical issues personal preferences coin flips These decisions were combined with our use case for servers and restrictions from various sources especially time to come up with a FreeBSD 10 infrastructure that was easily maintainable and superbly stable. This is really in the end about my discovery of the lesson that many are already aware of which is "your problems are not just your own when it comes to system administration". Simply said most problems a sysadmin face have solutions that are already out there and that there are usually many interested parties if there isn't a shared solution . I believe this is one of the great strength of open source software. This talk is also my attempt to share some of my own/OARnet's solutions in return and as thanks for all the great ones I’ve gotten from the community.
 Hello, my name is Nick Wolff. I work with Ornette. That is the Higher Academic Research Network. I will talk more about that. I've been a free BSG user since around 2012. I lead what we term as our network management infrastructure at Ornette. That covers a really, really wide range of equipment. Mostly it's all free BSG-based infrastructure, but there's some VPNs and other servers from other companies and a mix of things in there that I may talk about a little bit later. I do some Python and some Shell to make my job a lot easier. I work with a lot more C than I'm comfortable with, but I would not call myself a C programmer. I can read. I can compile. I can patch. That's about it. I backfill routing engineer workload as needed. Technically my title is routing engineer, but as being the one guy that does system admin stuff, there's not enough time for that no matter anything else. My work is allowing me to be here graciously on work time today, but the views here are my own and may or may not represent other views where I work. I'm not on call since about 4.37 this morning once we got that straightened out. Oops. So what is Ornette? Ornette is Ohio's academic research network. We are part of the Ohio State University and the state government. It's a lot of fun. So we supply Internet-slash-network services to that long list of things up there. So everything from public safety to you getting Internet at your library to you getting Internet at your school. So these are big bandwidth connections, small bandwidth connections. All of this is done over 100 gigabits per second backbone, about 4,000 kilometers of DWDM lit fiber. We don't manage the fiber. We just light it and then if it breaks, we call other people. We've been a service provider since 1987. We're mostly Juniper based routing and switching. So right now we have about 1,000 devices. I guess two years ago we had 330. The workload needed to triple your devices out in the field, monitor those devices and all of that is significant. By the end of 2017, we're aiming at around 2,000 devices, so an additional doubling. This is because we've taken on a lot more of the state's network and doing a lot of point-to-point links for them and other things like that to kind of interconnect the whole state. And then here's what you guys all care about. We have a free BSD based network management infrastructure with around 100 devices in various states of production. As I replace older services, that number goes up. As I remove services on boxes, that number will go down. Most of these are Intel servers. Some of these are ALEX boxes and other small platforms. We tried to have as unified an infrastructure as possible because that allows the best use of our resources for some administration. This talk is actually not going to be as technical as I would love it to be just because to cover what I want to talk about, I do not have enough time. But I will be open for questions on whichever part of this you are most interested in at the end and I can kind of delve into that. But I do have some coverage on the technologies we're using and what we're using to do this. So some just quick design influences to kind of explain some of the choices I've made because there's a lot of the, I'm sure I will get a lot of why you're doing this, why aren't you doing this and here's it. So service uptime and stability is the top thing, period. It's above anything else, security. It's above that if, and when I say that, I mean it's above that in what we carry about most security-wise is keeping those services up and running. The, you know, it's what's the impact on the service going to be that matters. We don't carry any kind of confidential data. It's our customer's job to clean that up, to protect that, to do all that kind of stuff. Our job is to transmit packets across the network. Simple as that and, you know, secure our own infrastructure. Our services support and bootstrap a state's infrastructure. So if we're down, there is no network, there is no connectivity and I need to be able to get into our systems to fix this. I cannot rely on any kind of network-based service, any kind of real-time anything at the lowest levels. So for example, I would love to have distributed authentication and we're working on some methods for that, but we can't just throw LDAP out there and, you know, hope everything works because, you know, the systems may not be able to talk to each other. We can't use a lot of the more modern two-factor solutions for this reason. We're situated in Columbus, Ohio and we cover the entire state with a few branches outward of the state for various things. It is very nice that this is a three-hour drive to any point in the state where we may have equipment, but at the same point this is a three-hour drive. This is not easy to accomplish, so we need to plan around this as much as possible. Currently we have a 9600-bod serial out-of-band connection over an out-of-band layer-2 network across our equipment. We can go faster, but it is just generally easier to have the lower speed so that all the ports match and we're not fighting when something breaks and figuring out, well, is this just going the other end of Juniper, a server, you know, to be able to get into it. And we don't have any kind of out-of-band network. Our servers live directly on our network. And hopefully we have remotely switchable power so we can type on a console and we can flick the power on and off and that's our remote hands unless we need to send somebody out there and all of our infrastructure is designed around this. Also, we have a very high bandwidth, very configurable, very low latency network because we sit on our backbone. So basically as you go out steps instead of slowing down, it gets much, much faster. So we sit at one gig on the host which go to 10 or 100 gig per second routers with a very good connectivity. So I came into Ornette a few years ago, I guess, four at this point. And this is kind of, we've been running FreeBSD since 2.2 in some form or another and basically we have pulled forward configurations, settings, kernel configs, not many code patches luckily but a lot of other things over that time period. And even where there were points where we could get rid of those, it's much more expedient time-wise or at least it appears to be in the short term to just pull them forward. So what this means is that the majority of our systems a few years ago were H stable with some six and one FreeBSD 4 system left. Hopefully that FreeBSD 4 system is still running and is hopefully getting turned off on Tuesday, hopefully. No, I was until 3.47 this morning when my boss took over. All of this stuff is, all of our old stuff was 32-bit because we still had some hardware that had 32-bit processors and we wanted to be able to keep using it at the time. UFS of course, we had a custom installer called CF install because we had a variety of issues with sys install so we had to do something else and to reach our needs. I believe a custom backup solution called NB dump, we did manual source builds for all user land software and then basically R-synced user local out to the box so that we could keep everything in sync, not have issues with ports and stuff which has wonderfully, wonderfully, wonderfully improved since then and we're hoping to have solutions around that. A lot of little tweaks to the system, kernel configurations that were special, things like that just that had kind of piled up over the years of a situation happens, you need to fix that situation, you make some change. So quick talk about CF install, it was a sys install replacement combined with a deployment system if you want to even call it that much. It's designed to do a full reinstall with zero physical interruptions so basically how it works is we have a large pile of scripts. They take an image, they build MFS routes, they do what we call at the time branding which is put in IP addresses, put in configuration details based on what kind of hardware platform we had, it's set settings, made sure it had all the things that were needed and again it was a lot of incremental features and changes to fix problems. I found this out actually very specifically I think a few months ago when I was running one of the base six systems we had, so free BSE six systems and on the free BSE eight CF it has a flag and that flag stops you from installing twice so after the first install it wipes, it deletes itself and then if you boot into the CF install it doesn't allow you to do it, obviously. That apparently was added later as I found out as I tried to boot into the compact flash so I could run a file system check and I promptly deleted the whole system. While we had backups I was also lucky that this was a relatively unused system that had a lot of duplicates and was spread out for network based reasons so I was just able to move the NetFlow servers that it was running to other boxes and it wasn't that big of a problem but it's those little details that really get to you and cause some not so great things. NB dump is very simple, it uses UFS to dump if you guys are familiar. It locally encrypts it and compresses it and then there's a server running on another box that it can access via SSH. It basically, SSH keys only can run, that user can only run that one program and it splits it up over multiple rate arrays, it deals with full and incremental backups so basically even if we do lose a rate array there's a whole separate one, you know, it's our backups we don't trust anything. So with this migration I had a few goals that are probably more clear now than they were in the past when I started but I think it's very important to look at your situation, your environment and figure out what you really need to do here. So the integration of modern changes and upgrades with our legacy stuff is very obvious and that's kind of the point of an upgrade. To minimize all custom work I would very much like to be able to talk to the community when I'm having issues and be able to split workloads in sane ways. To maximize manageability and maintainability, again I am one guy maintaining around 100 previous DS servers and finally raising our already high standards for uptime and reliability of our services. We have public safety, we have, you know, schools, we have hospitals all on this network and underlying network services, even things as little as DNS, NTP, you know, having issues can have far-raising issues across the network either directly or indirectly when our engineers can't properly fix a problem that popped up somewhere else. You know, it's all about making sure that our engineers have the best access possible and have that access 24-7, 365. So decisions. One of the hardest parts about this project is that I was basically on my own digging over changes that had been made over a decade. So I kind of went through these decisions and split them into various categories. So we had some stuff like the CF install, the majority of it was very, it allowed us to function at the scale we do. It allowed us that every time we talked to somebody they asked us, you know, how do you do this much for this few people? Some things were just simply obsolete technical issues. We did hacks and configurations to get around issues with the IP stack or, you know, various other areas. Some of it was personal preferences, so I had to, you know, figure out where my boss's decisions had come into play. And some of them were, you know, simply coin flips, figuring out where we had to decide an option so we had chose something and it really probably didn't matter now what I did. So this is probably way too preachy, but I tried to, you know, sum up what I really learned from all this. And it's very simple that, you know, if you're a system admin, the problems you're running into are probably not just your own. You know, these are problems that a lot of people are facing. You may have your own individual ones, you're going to have your own peculiarities, but, you know, but being able to talk to the community and look at what everybody else is doing, we saw this a ton because a lot of our problems just were basically gone in FreeBSD 10 and things that had changed. And we'd talked to no one and it's just that other people had run into these and they'd gotten fixed. So with the work I'd done, the modern changes, I've so far have completed, we're now running ZFS instead of UFS. Makes my life so much easier. 64-bit, GPT partitioning, we're using a modern USB memstick install instead of CF install. I will talk about this later. We straightened out all of our, you know, configuration details, but I will admit I was over cautious if I didn't have a decision on something or I didn't really, you know, feel comfortable, I tended to move it forward. Things, this is actually coming useful as, you know, my boss, you know, asked me, oh, you know, have you turned on process accounting on something, me going, no, let me go do that. And then coming back to me going, oh, you know, it was already running because he had founded previously, you know, very important to have for troubleshooting. And luckily, we have gone from custom kernel or, you know, custom kernel options to being able to just do a standard make build world and everything works, don't need to touch anything anymore. So by default, we're doing a three-way mirror. And we just decided that with the cost of disks and how far stuff out with a three-way mirror, we could lose a disk and feel comfortable waiting a few days to send a guy out to replace that disk. And so, first, if there is a, if we only had a two-way mirror, we would have picked up our phone, talked to operations people and said, hey, you get to drive for two hours right now. So that's the current decision there. We're using a variety of other stuff in a few situations. I have some, for one of our log servers, we have two mirror vDevs that are together to get, you know, better read-write speeds. We have some RAID Z2 for our backups just to get the best density there. So, but for the majority, all, most of our systems are mirrors. Yes, we use disks from currently, I think, Seagate and Western Digital. And we also, not only do we use that, but we make sure that there are every other in the system. Even if we do a two-way mirror, we can't end up only using one set of disks. Luckily, I work with, we are currently working with some of the PC connection. We know them as GovConnection. It's just a manufacturer, or a non-manufacturer, distributor that currently does, takes our component list, builds it all together, sets all of our BIOS, updates our BIOS, and does a lot of stuff, which makes it a lot more manageable for me, because as I'll talk about a little bit later, once I have these images built, I can swap a USB stick into a server and it goes out there. Actually, to be more correct, I can hand a USB stick to one of my operation guys who is not a server guy. He can take the boxed-up server, drive it out there, put it in place, pop in the USB stick, pop in the network connection, do some physical labeling, and the server is up and running and secure. Oh, sorry. Also, in the future, I'm interested in EFI boot. I'm currently using Boot Zero, if anybody is familiar with that, to have some easy access to change where I'm booting from, which I'll talk about later, and automated provisioning, having all of our systems in a database so that whenever I need to go make a memstick, it pulls all the relevant details, pulls the right configs, and pumps me out a memstick. But just giving it a host name, or even maybe when I create a new system, it just has it waiting, the image file waiting for me somewhere. So here's a talk about my first tool that is just an in-house tool. I call it server install because I'm not that brilliant with names. It's really a tool set to personalize our memstick builds for our remote servers. It's a templating system that ensures we get identical systems, just all very easily written in shell. Basically with a single command, we get a memstick that has all the IP address information, has the host names. It's secure, it has our settings, has our users, has our groups, has whatever files we want, all of those things are there. So this is what it looks like. So 90% of the time, all I need to do is feed it our host name and feed it the IP address information of the box. I get a memstick out, that's ready to go. So in the future, we're migrating out to Ansible, and I have this applied in some scenarios. But A, one of the big things was to not move too fast with this migration so that everybody that's using our systems is aware of it. And B, we may get systems that only have a serial connection or that are off by themselves or we're continually rebuilding from the memory stick so we have perf servers we may send out and after that, at a customer site, we wipe them and we may not get a chance to have network access to redo the stuff again. We may have a serial connection over Cisco WebEx that we managed to do sitting on some customer's land that they won't let us touch reasonably directly. And this just kind of gives us that access. What? Yeah. I mean, again, there's lots of decisions here, and I can talk more about them once I get through all this. The things that we are looking at more work from here. I do in no way think this is completed, but this is where we are and this is where we've come from. So the main, the script you just saw is localize image. We call it, it's called localize image. It's fairly simple, injects the files and scripts into a standard memstick. The reason for the asterisk is that the free BSD stable build process right now builds a dangerously dedicated disk, which doesn't work with boot zero. Boot zero is an extremely minor amount of code inside of it that allows it to choose which disk you're booting from and hit F1, F6, move between the slices if anyone's familiar with this. We're using that. So we actually, our systems are set to boot against the USB disk and then they immediately get sent over to the hard drives after the first install so that we're not fighting with the BIOS at, you know, two in the morning when something's going wrong or we've just with zero consoles have run into issues in the past with getting stuck places you need control at delete that aren't there. It should mostly work, but this just gives us some extra ease. And I guess the other big thing is that we can reset the boot loader from inside the DOS so that on the next reboot it boots back into the memstick and, for example, reinstalls the system or gives us access or, you know, does something else without us having touched the BIOS. To deal with all the permissions of all the files we're just throwing out there, regenerate around custom M trees. That's just, there's a free BSD utility called M tree that you can feed at a big long list of data, users permissions and it goes and fixes all that or you can also use it to tell you when the stuff isn't matching what you want it to be. So as future expansions we want to wrap nano BSD builds inside of our tool. Just that's going to be very simplistic. All that's going to mean is that we have a single interface which however we're doing it to build them and then, you know, use nano BSD everywhere down. This is for some special perfect purpose boxes on most of the alexes and then also I want to migrate our bastion host to nano BSD to have an easier upgrade process and there will also be a more lockdown build than the rest of our systems. And then also I want to pull all of this info from the central database so all I need to do is feed it to host him. It figures out everything else. This also again talks in will tie into future ansible and other integrations that I would like to do. Random stuff we do. Luckily the serial access is only needed now if there's a different pod of course because free BSD has it on by default which they didn't when I started this process. That's all very generic things. We use that format for home directories that way we can NFS mount multiple boxes on top of each other and not have collisions if we need to just freeze your access to some stuff. So moving on from there, I was hit with the point of having to choose between BSD install and PC system install. I'm not sure how many of you are familiar with either tool. BSD install is the official BSD installer that I believe was brought in in 9 or 10. And then PC system install is what PC BSD wrote for system installer. I'm not, this is really was I guess a lot of it was a personal opinion to make this decision. But personally I chose PC system install for the following reasons. Both of them had positives. Really what it came down to was that PC system install has good hooks at multiple places in the config. It has a good config format. Every time I saw people talking about BSD install they were basically saying oh it's just shell scripts. Just rewrite whatever you need. Which is great. But I like how structured PC system install is. And it's also, I assume BSD install would be easy, but it is easily injectable into the builds that we do at post, everything else after the system is built. And it just works. It's easy to update, easy to maintain. So I'm just going to quickly run through what a config looks like for that in case you haven't seen it. There's really not that much important here. You let it know that it's free BSD instead of PC BSD. Give it the host name. Tell it what disk files you want. All of this should look familiar. Disset up if you want to do UFS, ZFS, jelly with CFS, jelly with your swap, jelly with your UFS. All of that stuff just works. And then network is all fairly simple. Yes, I know I have more name servers than will actually work. It's just what we have set up. And it's easier to have them there than to have people going, why do we only have to read in that file? And then almost all of our magic we inject through, and I'm using magic lightly, we inject through these run commands. So we just have shell scripts that do a little bit of stuff like moving around the mount point to the ZFS home directory, pushing our files inside of the CheyRoot for the system, and some other things like that. The next tool we use is Life Preserver. This comes from PC BSD. Sorry. Okay. Nope. Nope. Go ahead. That's what all of these are. This is the config file for nonintended. It has all of the details you need. That's why I brought it up because I've heard a lot of questions about the unintended. And then just so I can reiterate one more time. So our server and star tool basically goes in here and inserts the right host name, inserts the right IP address, and the other stuff for the individual machine. So that's the other thing that like you need to get the box up before you can get Ansible to do whatever. I need to have the home directory exist in the right place before I create the users because otherwise the password file looks all weird and I'd go back and clean that up and you know other things along those. I'm really grateful for the work PC BSD has done on this and the continual improvements. I complained to Chris that they were dumping the tar options out. So if you're running it on a 9600-bod serial console, it takes about two hours to install a system. If you're running it on a 115.2K mode, it still took like 20 minutes. So they like days later put a patch in for a quiet mode to kind of get rid of all that stuff to bring my install at times down to what they should be. And I was very grateful to take out my patch and put it in the official one. Life preserver, how many of you have heard of this tool? Anyone? This PC BSD's backup system mostly meant for ZFS. It has a lot of capabilities. It will deal with your scrubs on your ZFS systems, your cron tab for your snaps. So it basically covers everything you need to cover. If you want to send the ZFS send and receive over SSH, it will handle that. But we are personally using their iSCSI-based backup solution, and I realize that the next line looks ridiculous, but I kind of did it on purpose because I generally get weird looks when I explain what we're doing. So we're doing GFS send receive to a iSCSI back disk that's locally jelly encrypted. On the backup server, this is backed by Zvol, but it doesn't have to be. You could use anybody's iSCSI server, anything that's available. And then obviously, all of the iSCSI happens over S-tunnel to be secure, even though we also, anything being written there is actually jelly encrypted already. So there's lots of advantages that I found with this. Our biggest thing was finding, though, a solution that we don't need to trust the backup server, and that was push instead of pull. That's what our previous NB dump had done, and we wanted to keep those requirements. If we eventually use somebody else for backup, you know, we don't have to trust, you know, we just have to trust they're going to keep our data from getting corrupted, not that they have to keep it safe. And another reason I like it, this is fairly minor, but it's really easy to expand the search by just adding a new Z-pull, because you can just put those new Zvols for new systems onto that new Z-pull, and, you know, you don't have to, it's very, I guess, disconnected from the way we'd previously done a lot of stuff. It's not all expecting it all in one big RAID volume or something else. A lot of that's self-configurable. But for our applications, we had to make a few patches. So the ones we have completed internally and are using to make these systems work is, we call our alternating patch. It very simply alternates the days the backup happens, and randomly chooses a time to do the crown job between starting at midnight and starting at 4 a.m. What this allows us to do is we actually run this twice with different servers as the option, and it makes sure that they happen on different days, so that we have two different systems. And, you know, worst case, if one system dies, all we've got to do is go back to the other one, and we have a 48-hour-old backup worst case instead of a 24-hour backup. But it still is very, very useful. And by randomly setting the time to help our network capacity and help running into issues with the backup servers, but still leaves it tunable, so if we have one massive system, I can go in there after it's set up and, you know, slide back the window, have it start exactly at midnight so it has as much time as possible to run. Also metadata backup. You know, if you're locally encrypting, you need to get those encryption keys off the box, otherwise your backups are worthless. They currently have an implementation that uses Jelly, but you have to provide a password, so you have to have a user involved. Our implementation just uses public key encryption. We use SMIME at an open SSL to take the Jelly keys, the Jelly password, all the needed metadata, encrypted SMIME, you know, encrypted with SMIME, and shoot it off the email so we have access to it. There are much probably cleaner ways we could do this, but this was the one that we felt had the least tie to other infrastructures that we were worried about breaking down during a situation where we need to pull backups. We do not pull backups very often, and right now, ZFS basically fixes our only use case other than catastrophic failure, that use case being our user's deleting files on accident. With snapshots, we can go look back at that on the individual box, which is a great help. Still needed. Automated provisioning. Right now, I have to touch the backup server because we have keys on both sides. So that it can only talk to a single ISCSI session, and different servers can't see each other's sessions. If they can, they wouldn't be able to crack the data because it's encrypted using Jelly, but they would be able to do bad things like delete it, which we would like to avoid. And then also some other miscellaneous features. I want to add logging via syslog and generating sending reports. They already do a lot of this stuff via email, but we really want this coming in via syslog, where we can filter and then run, well, the generating and sending reports would be on the back end of our syslog. And then, I guess this one, basically, again, is the automated provisioning and doing verification of various information we already have out there to kind of say yes, no, on whether we can do that. So I guess to move on, I try, well, again, try not to get too preachy here. There are people with way, way more experience than me, but if I was going to talk to people doing this kind of stuff here are the lessons that I feel I've learned. Many people are having the same problems. This means talk to the community. Not only talk to them, but don't feel bad to complain about the problems you're having. And on the opposite side of that, make sure you listen to other people's complaints, even if they seem very simpler or whatever. Just listen. Last year I came here for my first time to BSDCan, and I did not have a backup solution that fit our infrastructure, our needs, and our requirements. And I was looking, do I need to talk to my boss about dropping some of those requirements? What do we need to do here to get this to work? And I believe I was trying to chris Moore that works for the PCBSD project, and he told me about Life Preserver. And I had it up and running at the first version within a week of getting back to the office after BSDCan last year, and I've been running it throughout the year, making changes and doing other work. Oh, sorry. I totally skipped. The reason I have these listed as done internally is I will be attempting to upstream these. They just need cleaned up in a sane way that other people will be interested in, and not be what I did to make them work in our systems. And I have a student working on that right now, so I should be sending stuff out to the PCBSD mailing list soon to get their input on what we want to do. Not only complain about the work, but make sure you praise the work that others are doing so they realize how important some of these subsystems are. The work PCBSD has done has massively made my life easier, but that's just the easiest thing right now to talk about. There are so many subsystems that are worked on continually for BSD that every day I hear about some patch, and I'm super excited because it is going to make my life easier in some way or another. Make our systems better, make things more secure, more user-friendly. The fact that 10 hours of planned work now can save me five hours of unplanned work some random night. This is not always true, and I'm not saying that you can always avoid future work. But what I'm saying is that if I'm given the option, I would prefer to plan ahead over-engineer solutions or engineer solutions, well, I guess over-engineer may be a bad word, and avoid that unplanned work that is going to totally throw off my schedule. When something pops up, I deal with it, I get it fixed, but then my schedule is just shot for a long time from there after, so that's kind of the view I try to have when I'm doing these things. Yeah, so yeah. Yeah, yeah, but yeah, so somebody was just saying that they think that I'm underestimating how important the prep is. Yeah, but just, anyway, do it, that makes a massive difference, and trying to cut out as many of those unneeded late nights as possible is a really big plus and really positive for your sanity. And then also that the free BSD build system is wonderful. I run a few commands, everything's done. I no longer looking at our old build instructions and where we were four or five years ago and looking at my current instructions for our current builds. I'm just so much happier that I get to deal with what we have now than what we used to have to deal with because of all of the work done, and because there are people, again, continually hitting these problems and trying to improve life for themselves and others. So thank you. Thank you, everyone, for coming here and listening to my talk. Thank all of the communities, Free BSD, Open BSD, Net BSD, and any others for all of the contributions they've made and all of the work they do. Special thanks to PCBSD. And yes, just thank you very much. So now I'm open to questions, and I can talk a lot more technically about anything I discussed or anything else we deal with if there is anything. I have a good chunk of time here if there's anything. Yep. So our Bastion host development servers, they run... Let me go through this list here. So our Bastion host, all of our authentication, we're running mostly TechExPlus for all of our switches and routers for our authentication. So all of that's running there. NTP and DNS, if those aren't working, really weird and bad things happen. Logging statistics visualization, we have a wonderful design team that works on live network visualization. So we pull in SNMP data and other data, combine it all, and it gets displayed on really pretty SVGs with the JavaScript backend that keep them live updating so that we can look at our network and know what's going on, know where link congestion is, know where a lot of other stuff is. That is a whole, you know, another talk to itself. Does that answer your question? Yeah. Okay. Okay. Yeah, okay. Yeah, so just to quickly reiterate, he was asking my experiences with life preserver and then my experience with jails. So life preserver, I really haven't run into that many issues with. Were you doing the SSH BAS, the ice-cozy? My major issue was I actually was using snapshots and I was only able to use them on all snapshots. Oh. I became self-savvy for the best of the substance and I spoke to the body's work. So I will say here that something that actually came from me doing snapshots before I had life preserver, I'm still using a different piece of software to do snapshots and that's actually one of my to-do fixes to get back to using life preservers. So yeah, so something else to snapshots, life preserver is just doing some scrubs and the actual backup of the data. Yeah. But other than that, life preserver's been fairly great. There's some things I need to look at and I wanted to get unmapped working, which will work over on ice-cozy so that as stuff grows and shrinks, we're not using too much space on the ice-cozy. Here's all sparsely done. There seems to be a bug right now though where Jelly's not passing that through and it's something that you kind of want to have the option of whether or not you want that to be passed through. If you're us, you do want it passed through. It makes so much more sense. You want good reliable or good people not knowing how much data you have on that disk. You don't want it to be passed through. I'm just going to understand how less is the reward of the data that you've got. The data that's cool on the hook and it's going to be about 10. Okay, so I'll go through that one more time because as I said that... One second. So the basis of how the Life Preserver works in our setup, you're getting the ice-cozy section of the replication, we have a box that serves ice-cozy. We have S-Tunnel running in that box that we can connect to from our other servers. Our servers that are doing the backup, so the clients in this case, launch an ice-cozy session that's preconfigured so that they have a raw disk available that's coming from the ice-cozy server. We then put... Jelly then gets put on top of that, so it's encrypted locally. And then ZFS is put on top of that and ZFS send receive goes from the local ZFS pool to the ice-cozy-based ZFS pool. Some other nice things about this since there's not currently ZFS send receive compression, decompresses, recompresses, instead of using something else in the middle, all this actually happens local to the box. And it also means that this scales very well because the compression doesn't at all... The ice-cozy server in the back end doesn't at all care. I've heard other people talk about doing this over very crappy internet connections and the ice-cozy still works fine. I obviously do it over a very, very fast connection. I have a 10-gig server, so multiple servers can be kind of maxing out their sends. Does that answer your question? Is that okay anymore in that area that you're interested in? Yeah, I think so. Nope, it's going to a separate pool, which is very nice because it just uses all the ZFS infrastructure. The other big thing Life Preserver does that I should not forget to mention is it takes all user attributes and puts them under a prefix. So your mount locations become Life Preserver colon mount point, which stops them from getting mounted or other weird things from happening. If you've ever played with Send Receive, you dash R, it doesn't get mounted the first time. You reboot the system and all of a sudden it auto-mounts them and you're very not happy. It deals with all this stuff. That happens whether you use the SSH-based, which basically deals with it on top and for the most part is very similar to normal ZFS send receive, deals with controlling when it happens, things like that. Or if you're doing over I SCSI, it's very similar. Questions? So I guess back to the question about jails. I'm currently only using them in a few situations, mostly to split out some production and testing places so I can have them running on the same boxes. And I'm also using jails to support my Edge web server. It's actually an IMA web server, it's a reverse proxy so that all public traffic comes to the reverse proxy and back to us. Luckily, I only deal with our engineering side, so I can restrict that down to only our customer address space and a few additional things. But one of these things is just we try to be as paranoid as possible, which in our situation we have the ability to do. I don't use any tools for that actually. I just take whatever image is on the memstick, install it into a skeleton, what I call the skeleton jail, and then I ZFS clone that to every new jail so that I'm not duplicating, you know, I'm not using extra space that I don't need to. And it's a handful of jails in a few places, it's not a big thing that needs orchestration. Does that answer your question? I'll try to make it helpful. Anything else? Okay, then I think that's all I have for today. Thank you very much. Thank you. If you want to get ahold of me, I'm on Twitter, my personal email, my work email. If you have any other questions, feel free to grab me. I'll be right back at the conference. Thank you very much.
 Hello, my name is Nick Wolff.  I work with Ornette.  That is the Higher Academic Research Network.  I will talk more about that.  I've been a free BSG user since around 2012.  I lead what we term as our network management infrastructure at Ornette.  That covers a really, really wide range of equipment.  Mostly it's all free BSG-based infrastructure, but there's some VPNs and other servers from  other companies and a mix of things in there that I may talk about a little bit later.  I do some Python and some Shell to make my job a lot easier.  I work with a lot more C than I'm comfortable with, but I would not call myself a C programmer.  I can read.  I can compile.  I can patch.  That's about it.  I backfill routing engineer workload as needed.  Technically my title is routing engineer, but as being the one guy that does system admin  stuff, there's not enough time for that no matter anything else.  My work is allowing me to be here graciously on work time today, but the views here are  my own and may or may not represent other views where I work.  I'm not on call since about 4.37 this morning once we got that straightened out.  Oops.  So what is Ornette?  Ornette is Ohio's academic research network.  We are part of the Ohio State University and the state government.  It's a lot of fun.  So we supply Internet-slash-network services to that long list of things up there.  So everything from public safety to you getting Internet at your library to you getting Internet  at your school.  So these are big bandwidth connections, small bandwidth connections.  All of this is done over 100 gigabits per second backbone, about 4,000 kilometers of  DWDM lit fiber.  We don't manage the fiber.  We just light it and then if it breaks, we call other people.  We've been a service provider since 1987.  We're mostly Juniper based routing and switching.  So right now we have about 1,000 devices.  I guess two years ago we had 330.  The workload needed to triple your devices out in the field, monitor those devices and  all of that is significant.  By the end of 2017, we're aiming at around 2,000 devices, so an additional doubling.  This is because we've taken on a lot more of the state's network and doing a lot of  point-to-point links for them and other things like that to kind of interconnect the whole  state.  And then here's what you guys all care about.  We have a free BSD based network management infrastructure with around 100 devices in  various states of production.  As I replace older services, that number goes up.  As I remove services on boxes, that number will go down.  Most of these are Intel servers.  Some of these are ALEX boxes and other small platforms.  We tried to have as unified an infrastructure as possible because that allows the best use  of our resources for some administration.  This talk is actually not going to be as technical as I would love it to be just because to cover  what I want to talk about, I do not have enough time.  But I will be open for questions on whichever part of this you are most interested in at  the end and I can kind of delve into that.  But I do have some coverage on the technologies we're using and what we're using to do this.  So some just quick design influences to kind of explain some of the choices I've made because  there's a lot of the, I'm sure I will get a lot of why you're doing this, why aren't  you doing this and here's it.  So service uptime and stability is the top thing, period.  It's above anything else, security.  It's above that if, and when I say that, I mean it's above that in what we carry about  most security-wise is keeping those services up and running.  The, you know, it's what's the impact on the service going to be that matters.  We don't carry any kind of confidential data.  It's our customer's job to clean that up, to protect that, to do all that kind of stuff.  Our job is to transmit packets across the network.  Simple as that and, you know, secure our own infrastructure.  Our services support and bootstrap a state's infrastructure.  So if we're down, there is no network, there is no connectivity and I need to be able to  get into our systems to fix this.  I cannot rely on any kind of network-based service, any kind of real-time anything at  the lowest levels.  So for example, I would love to have distributed authentication and we're working on some methods  for that, but we can't just throw LDAP out there and, you know, hope everything works  because, you know, the systems may not be able to talk to each other.  We can't use a lot of the more modern two-factor solutions for this reason.  We're situated in Columbus, Ohio and we cover the entire state with a few branches outward  of the state for various things.  It is very nice that this is a three-hour drive to any point in the state where we may  have equipment, but at the same point this is a three-hour drive.  This is not easy to accomplish, so we need to plan around this as much as possible.  Currently we have a 9600-bod serial out-of-band connection over an out-of-band layer-2 network  across our equipment.  We can go faster, but it is just generally easier to have the lower speed so that all  the ports match and we're not fighting when something breaks and figuring out, well, is  this just going the other end of Juniper, a server, you know, to be able to get into  it.  And we don't have any kind of out-of-band network.  Our servers live directly on our network.  And hopefully we have remotely switchable power so we can type on a console and we can  flick the power on and off and that's our remote hands unless we need to send somebody  out there and all of our infrastructure is designed around this.  Also, we have a very high bandwidth, very configurable, very low latency network because  we sit on our backbone.  So basically as you go out steps instead of slowing down, it gets much, much faster.  So we sit at one gig on the host which go to 10 or 100 gig per second routers with a  very good connectivity.  So I came into Ornette a few years ago, I guess, four at this point.  And this is kind of, we've been running FreeBSD since 2.2 in some form or another and basically  we have pulled forward configurations, settings, kernel configs, not many code patches luckily  but a lot of other things over that time period.  And even where there were points where we could get rid of those, it's much more expedient  time-wise or at least it appears to be in the short term to just pull them forward.  So what this means is that the majority of our systems a few years ago were H stable with  some six and one FreeBSD 4 system left.  Hopefully that FreeBSD 4 system is still running and is hopefully getting turned off on Tuesday,  hopefully.  No, I was until 3.47 this morning when my boss took over.  All of this stuff is, all of our old stuff was 32-bit because we still had some hardware  that had 32-bit processors and we wanted to be able to keep using it at the time.  UFS of course, we had a custom installer called CF install because we had a variety of issues  with sys install so we had to do something else and to reach our needs.  I believe a custom backup solution called NB dump, we did manual source builds for all  user land software and then basically R-synced user local out to the box so that we could  keep everything in sync, not have issues with ports and stuff which has wonderfully, wonderfully,  wonderfully improved since then and we're hoping to have solutions around that.  A lot of little tweaks to the system, kernel configurations that were special, things like  that just that had kind of piled up over the years of a situation happens, you need to  fix that situation, you make some change.  So quick talk about CF install, it was a sys install replacement combined with a deployment  system if you want to even call it that much.  It's designed to do a full reinstall with zero physical interruptions so basically how it  works is we have a large pile of scripts.  They take an image, they build MFS routes, they do what we call at the time branding  which is put in IP addresses, put in configuration details based on what kind of hardware platform  we had, it's set settings, made sure it had all the things that were needed and again  it was a lot of incremental features and changes to fix problems.  I found this out actually very specifically I think a few months ago when I was running  one of the base six systems we had, so free BSE six systems and on the free BSE eight CF  it has a flag and that flag stops you from installing twice so after the first install  it wipes, it deletes itself and then if you boot into the CF install it doesn't allow  you to do it, obviously.  That apparently was added later as I found out as I tried to boot into the compact flash  so I could run a file system check and I promptly deleted the whole system.  While we had backups I was also lucky that this was a relatively unused system that  had a lot of duplicates and was spread out for network based reasons so I was just able  to move the NetFlow servers that it was running to other boxes and it wasn't that big of a  problem but it's those little details that really get to you and cause some not so great  things.  NB dump is very simple, it uses UFS to dump if you guys are familiar.  It locally encrypts it and compresses it and then there's a server running on another box  that it can access via SSH.  It basically, SSH keys only can run, that user can only run that one program and it  splits it up over multiple rate arrays, it deals with full and incremental backups so  basically even if we do lose a rate array there's a whole separate one, you know, it's  our backups we don't trust anything.  So with this migration I had a few goals that are probably more clear now than they were  in the past when I started but I think it's very important to look at your situation,  your environment and figure out what you really need to do here.  So the integration of modern changes and upgrades with our legacy stuff is very obvious and  that's kind of the point of an upgrade.  To minimize all custom work I would very much like to be able to talk to the community  when I'm having issues and be able to split workloads in sane ways.  To maximize manageability and maintainability, again I am one guy maintaining around 100  previous DS servers and finally raising our already high standards for uptime and reliability  of our services.  We have public safety, we have, you know, schools, we have hospitals all on this network  and underlying network services, even things as little as DNS, NTP, you know, having issues  can have far-raising issues across the network either directly or indirectly when our engineers  can't properly fix a problem that popped up somewhere else.  You know, it's all about making sure that our engineers have the best access possible  and have that access 24-7, 365.  So decisions.  One of the hardest parts about this project is that I was basically on my own digging  over changes that had been made over a decade.  So I kind of went through these decisions and split them into various categories.  So we had some stuff like the CF install, the majority of it was very, it allowed us  to function at the scale we do.  It allowed us that every time we talked to somebody they asked us, you know, how do  you do this much for this few people?  Some things were just simply obsolete technical issues.  We did hacks and configurations to get around issues with the IP stack or, you know, various  other areas.  Some of it was personal preferences, so I had to, you know, figure out where my boss's  decisions had come into play.  And some of them were, you know, simply coin flips, figuring out where we had to decide  an option so we had chose something and it really probably didn't matter now what I did.  So this is probably way too preachy, but I tried to, you know, sum up what I really  learned from all this.  And it's very simple that, you know, if you're a system admin, the problems you're running  into are probably not just your own.  You know, these are problems that a lot of people are facing.  You may have your own individual ones, you're going to have your own peculiarities, but,  you know, but being able to talk to the community and look at what everybody else is doing,  we saw this a ton because a lot of our problems just were basically gone in FreeBSD 10 and  things that had changed.  And we'd talked to no one and it's just that other people had run into these and they'd  gotten fixed.  So with the work I'd done, the modern changes, I've so far have completed, we're now running  ZFS instead of UFS.  Makes my life so much easier.  64-bit, GPT partitioning, we're using a modern USB memstick install instead of CF install.  I will talk about this later.  We straightened out all of our, you know, configuration details, but I will admit I was over cautious  if I didn't have a decision on something or I didn't really, you know, feel comfortable,  I tended to move it forward.  Things, this is actually coming useful as, you know, my boss, you know, asked me, oh,  you know, have you turned on process accounting on something, me going, no, let me go do that.  And then coming back to me going, oh, you know, it was already running because he had  founded previously, you know, very important to have for troubleshooting.  And luckily, we have gone from custom kernel or, you know, custom kernel options to being  able to just do a standard make build world and everything works, don't need to touch  anything anymore.  So by default, we're doing a three-way mirror.  And we just decided that with the cost of disks and how far stuff out with a three-way  mirror, we could lose a disk and feel comfortable waiting a few days to send a guy out to replace  that disk.  And so, first, if there is a, if we only had a two-way mirror, we would have picked up  our phone, talked to operations people and said, hey, you get to drive for two hours  right now.  So that's the current decision there.  We're using a variety of other stuff in a few situations.  I have some, for one of our log servers, we have two mirror vDevs that are together to  get, you know, better read-write speeds.  We have some RAID Z2 for our backups just to get the best density there.  So, but for the majority, all, most of our systems are mirrors.  Yes, we use disks from currently, I think, Seagate and Western Digital.  And we also, not only do we use that, but we make sure that there are every other in  the system.  Even if we do a two-way mirror, we can't end up only using one set of disks.  Luckily, I work with, we are currently working with some of the PC connection.  We know them as GovConnection.  It's just a manufacturer, or a non-manufacturer, distributor that currently does, takes our  component list, builds it all together, sets all of our BIOS, updates our BIOS, and does  a lot of stuff, which makes it a lot more manageable for me, because as I'll talk about  a little bit later, once I have these images built, I can swap a USB stick into a server  and it goes out there.  Actually, to be more correct, I can hand a USB stick to one of my operation guys who  is not a server guy.  He can take the boxed-up server, drive it out there, put it in place, pop in the USB stick,  pop in the network connection, do some physical labeling, and the server is up and running  and secure.  Oh, sorry.  Also, in the future, I'm interested in EFI boot.  I'm currently using Boot Zero, if anybody is familiar with that, to have some easy access  to change where I'm booting from, which I'll talk about later, and automated provisioning,  having all of our systems in a database so that whenever I need to go make a memstick,  it pulls all the relevant details, pulls the right configs, and pumps me out a memstick.  But just giving it a host name, or even maybe when I create a new system, it just has it  waiting, the image file waiting for me somewhere.  So here's a talk about my first tool that is just an in-house tool.  I call it server install because I'm not that brilliant with names.  It's really a tool set to personalize our memstick builds for our remote servers.  It's a templating system that ensures we get identical systems, just all very easily written  in shell.  Basically with a single command, we get a memstick that has all the IP address information, has  the host names.  It's secure, it has our settings, has our users, has our groups, has whatever files we  want, all of those things are there.  So this is what it looks like.  So 90% of the time, all I need to do is feed it our host name and feed it the IP address  information of the box.  I get a memstick out, that's ready to go.  So in the future, we're migrating out to Ansible, and I have this applied in some scenarios.  But A, one of the big things was to not move too fast with this migration so that everybody  that's using our systems is aware of it.  And B, we may get systems that only have a serial connection or that are off by themselves  or we're continually rebuilding from the memory stick so we have perf servers we may send  out and after that, at a customer site, we wipe them and we may not get a chance to have  network access to redo the stuff again.  We may have a serial connection over Cisco WebEx that we managed to do sitting on some  customer's land that they won't let us touch reasonably directly.  And this just kind of gives us that access.  What?  Yeah.  I mean, again, there's lots of decisions here, and I can talk more about them once  I get through all this.  The things that we are looking at more work from here.  I do in no way think this is completed, but this is where we are and this is where we've  come from.  So the main, the script you just saw is localize image.  We call it, it's called localize image.  It's fairly simple, injects the files and scripts into a standard memstick.  The reason for the asterisk is that the free BSD stable build process right now builds a  dangerously dedicated disk, which doesn't work with boot zero.  Boot zero is an extremely minor amount of code inside of it that allows it to choose  which disk you're booting from and hit F1, F6, move between the slices if anyone's familiar  with this.  We're using that.  So we actually, our systems are set to boot against the USB disk and then they immediately  get sent over to the hard drives after the first install so that we're not fighting with  the BIOS at, you know, two in the morning when something's going wrong or we've just  with zero consoles have run into issues in the past with getting stuck places you need  control at delete that aren't there.  It should mostly work, but this just gives us some extra ease.  And I guess the other big thing is that we can reset the boot loader from inside the  DOS so that on the next reboot it boots back into the memstick and, for example, reinstalls  the system or gives us access or, you know, does something else without us having touched  the BIOS.  To deal with all the permissions of all the files we're just throwing out there, regenerate  around custom M trees.  That's just, there's a free BSD utility called M tree that you can feed at a big long list  of data, users permissions and it goes and fixes all that or you can also use it to tell  you when the stuff isn't matching what you want it to be.  So as future expansions we want to wrap nano BSD builds inside of our tool.  Just that's going to be very simplistic.  All that's going to mean is that we have a single interface which however we're doing  it to build them and then, you know, use nano BSD everywhere down.  This is for some special perfect purpose boxes on most of the alexes and then also I want  to migrate our bastion host to nano BSD to have an easier upgrade process and there will  also be a more lockdown build than the rest of our systems.  And then also I want to pull all of this info from the central database so all I need to  do is feed it to host him.  It figures out everything else.  This also again talks in will tie into future ansible and other integrations that I would  like to do.  Random stuff we do.  Luckily the serial access is only needed now if there's a different pod of course because  free BSD has it on by default which they didn't when I started this process.  That's all very generic things.  We use that format for home directories that way we can NFS mount multiple boxes on top  of each other and not have collisions if we need to just freeze your access to some stuff.  So moving on from there, I was hit with the point of having to choose between BSD install  and PC system install.  I'm not sure how many of you are familiar with either tool.  BSD install is the official BSD installer that I believe was brought in in 9 or 10.  And then PC system install is what PC BSD wrote for system installer.  I'm not, this is really was I guess a lot of it was a personal opinion to make this  decision.  But personally I chose PC system install for the following reasons.  Both of them had positives.  Really what it came down to was that PC system install has good hooks at multiple places  in the config.  It has a good config format.  Every time I saw people talking about BSD install they were basically saying oh it's  just shell scripts.  Just rewrite whatever you need.  Which is great.  But I like how structured PC system install is.  And it's also, I assume BSD install would be easy, but it is easily injectable into  the builds that we do at post, everything else after the system is built.  And it just works.  It's easy to update, easy to maintain.  So I'm just going to quickly run through what a config looks like for that in case you haven't  seen it.  There's really not that much important here.  You let it know that it's free BSD instead of PC BSD.  Give it the host name.  Tell it what disk files you want.  All of this should look familiar.  Disset up if you want to do UFS, ZFS, jelly with CFS, jelly with your swap, jelly with  your UFS.  All of that stuff just works.  And then network is all fairly simple.  Yes, I know I have more name servers than will actually work.  It's just what we have set up.  And it's easier to have them there than to have people going, why do we only have to  read in that file?  And then almost all of our magic we inject through, and I'm using magic lightly, we inject  through these run commands.  So we just have shell scripts that do a little bit of stuff like moving around the mount  point to the ZFS home directory, pushing our files inside of the CheyRoot for the system,  and some other things like that.  The next tool we use is Life Preserver.  This comes from PC BSD.  Sorry.  Okay.  Nope.  Nope.  Go ahead.  That's what all of these are.  This is the config file for nonintended.  It has all of the details you need.  That's why I brought it up because I've heard a lot of questions about the unintended.  And then just so I can reiterate one more time.  So our server and star tool basically goes in here and inserts the right host name, inserts  the right IP address, and the other stuff for the individual machine.  So that's the other thing that like you need to get the box up before you can get Ansible  to do whatever.  I need to have the home directory exist in the right place before I create the users  because otherwise the password file looks all weird and I'd go back and clean that up  and you know other things along those.  I'm really grateful for the work PC BSD has done on this and the continual improvements.  I complained to Chris that they were dumping the tar options out.  So if you're running it on a 9600-bod serial console, it takes about two hours to install  a system.  If you're running it on a 115.2K mode, it still took like 20 minutes.  So they like days later put a patch in for a quiet mode to kind of get rid of all that  stuff to bring my install at times down to what they should be.  And I was very grateful to take out my patch and put it in the official one.  Life preserver, how many of you have heard of this tool?  Anyone?  This PC BSD's backup system mostly meant for ZFS.  It has a lot of capabilities.  It will deal with your scrubs on your ZFS systems, your cron tab for your snaps.  So it basically covers everything you need to cover.  If you want to send the ZFS send and receive over SSH, it will handle that.  But we are personally using their iSCSI-based backup solution, and I realize that the next  line looks ridiculous, but I kind of did it on purpose because I generally get weird looks  when I explain what we're doing.  So we're doing GFS send receive to a iSCSI back disk that's locally jelly encrypted.  On the backup server, this is backed by Zvol, but it doesn't have to be.  You could use anybody's iSCSI server, anything that's available.  And then obviously, all of the iSCSI happens over S-tunnel to be secure, even though we  also, anything being written there is actually jelly encrypted already.  So there's lots of advantages that I found with this.  Our biggest thing was finding, though, a solution that we don't need to trust the backup server,  and that was push instead of pull.  That's what our previous NB dump had done, and we wanted to keep those requirements.  If we eventually use somebody else for backup, you know, we don't have to trust, you know,  we just have to trust they're going to keep our data from getting corrupted, not that  they have to keep it safe.  And another reason I like it, this is fairly minor, but it's really easy to expand the  search by just adding a new Z-pull, because you can just put those new Zvols for new systems  onto that new Z-pull, and, you know, you don't have to, it's very, I guess, disconnected  from the way we'd previously done a lot of stuff.  It's not all expecting it all in one big RAID volume or something else.  A lot of that's self-configurable.  But for our applications, we had to make a few patches.  So the ones we have completed internally and are using to make these systems work is, we  call our alternating patch.  It very simply alternates the days the backup happens, and randomly chooses a time to do  the crown job between starting at midnight and starting at 4 a.m.  What this allows us to do is we actually run this twice with different servers as the option,  and it makes sure that they happen on different days, so that we have two different systems.  And, you know, worst case, if one system dies, all we've got to do is go back to the other  one, and we have a 48-hour-old backup worst case instead of a 24-hour backup.  But it still is very, very useful.  And by randomly setting the time to help our network capacity and help running into issues  with the backup servers, but still leaves it tunable, so if we have one massive system,  I can go in there after it's set up and, you know, slide back the window, have it start  exactly at midnight so it has as much time as possible to run.  Also metadata backup.  You know, if you're locally encrypting, you need to get those encryption keys off the  box, otherwise your backups are worthless.  They currently have an implementation that uses Jelly, but you have to provide a password,  so you have to have a user involved.  Our implementation just uses public key encryption.  We use SMIME at an open SSL to take the Jelly keys, the Jelly password, all the needed metadata,  encrypted SMIME, you know, encrypted with SMIME, and shoot it off the email so we have access  to it.  There are much probably cleaner ways we could do this, but this was the one that we felt  had the least tie to other infrastructures that we were worried about breaking down during  a situation where we need to pull backups.  We do not pull backups very often, and right now, ZFS basically fixes our only use case  other than catastrophic failure, that use case being our user's deleting files on accident.  With snapshots, we can go look back at that on the individual box, which is a great help.  Still needed.  Automated provisioning.  Right now, I have to touch the backup server because we have keys on both sides.  So that it can only talk to a single ISCSI session, and different servers can't see each  other's sessions.  If they can, they wouldn't be able to crack the data because it's encrypted using Jelly,  but they would be able to do bad things like delete it, which we would like to avoid.  And then also some other miscellaneous features.  I want to add logging via syslog and generating sending reports.  They already do a lot of this stuff via email, but we really want this coming in via syslog,  where we can filter and then run, well, the generating and sending reports would be on  the back end of our syslog.  And then, I guess this one, basically, again, is the automated provisioning and doing verification  of various information we already have out there to kind of say yes, no, on whether we  can do that.  So I guess to move on, I try, well, again, try not to get too preachy here.  There are people with way, way more experience than me, but if I was going to talk to people  doing this kind of stuff here are the lessons that I feel I've learned.  Many people are having the same problems.  This means talk to the community.  Not only talk to them, but don't feel bad to complain about the problems you're having.  And on the opposite side of that, make sure you listen to other people's complaints, even  if they seem very simpler or whatever.  Just listen.  Last year I came here for my first time to BSDCan, and I did not have a backup solution  that fit our infrastructure, our needs, and our requirements.  And I was looking, do I need to talk to my boss about dropping some of those requirements?  What do we need to do here to get this to work?  And I believe I was trying to chris Moore that works for the PCBSD project, and he told  me about Life Preserver.  And I had it up and running at the first version within a week of getting back to the office  after BSDCan last year, and I've been running it throughout the year, making changes and  doing other work.  Oh, sorry.  I totally skipped.  The reason I have these listed as done internally is I will be attempting to upstream these.  They just need cleaned up in a sane way that other people will be interested in, and not  be what I did to make them work in our systems.  And I have a student working on that right now, so I should be sending stuff out to the  PCBSD mailing list soon to get their input on what we want to do.  Not only complain about the work, but make sure you praise the work that others are doing  so they realize how important some of these subsystems are.  The work PCBSD has done has massively made my life easier, but that's just the easiest  thing right now to talk about.  There are so many subsystems that are worked on continually for BSD that every day I hear  about some patch, and I'm super excited because it is going to make my life easier in some  way or another.  Make our systems better, make things more secure, more user-friendly.  The fact that 10 hours of planned work now can save me five hours of unplanned work some  random night.  This is not always true, and I'm not saying that you can always avoid future work.  But what I'm saying is that if I'm given the option, I would prefer to plan ahead over-engineer  solutions or engineer solutions, well, I guess over-engineer may be a bad word, and avoid  that unplanned work that is going to totally throw off my schedule.  When something pops up, I deal with it, I get it fixed, but then my schedule is just  shot for a long time from there after, so that's kind of the view I try to have when  I'm doing these things.  Yeah, so yeah.  Yeah, yeah, but yeah, so somebody was just saying that they think that I'm underestimating  how important the prep is.  Yeah, but just, anyway, do it, that makes a massive difference, and trying to cut out  as many of those unneeded late nights as possible is a really big plus and really positive for  your sanity.  And then also that the free BSD build system is wonderful.  I run a few commands, everything's done.  I no longer looking at our old build instructions and where we were four or five years ago and  looking at my current instructions for our current builds.  I'm just so much happier that I get to deal with what we have now than what we used to  have to deal with because of all of the work done, and because there are people, again,  continually hitting these problems and trying to improve life for themselves and others.  So thank you.  Thank you, everyone, for coming here and listening to my talk.  Thank all of the communities, Free BSD, Open BSD, Net BSD, and any others for all of the  contributions they've made and all of the work they do.  Special thanks to PCBSD.  And yes, just thank you very much.  So now I'm open to questions, and I can talk a lot more technically about anything I discussed  or anything else we deal with if there is anything.  I have a good chunk of time here if there's anything.  Yep.  So our Bastion host development servers, they run...  Let me go through this list here.  So our Bastion host, all of our authentication, we're running mostly TechExPlus for all of  our switches and routers for our authentication.  So all of that's running there.  NTP and DNS, if those aren't working, really weird and bad things happen.  Logging statistics visualization, we have a wonderful design team that works on live  network visualization.  So we pull in SNMP data and other data, combine it all, and it gets displayed on really pretty  SVGs with the JavaScript backend that keep them live updating so that we can look at  our network and know what's going on, know where link congestion is, know where a lot  of other stuff is.  That is a whole, you know, another talk to itself.  Does that answer your question?  Yeah.  Okay.  Okay.  Yeah, okay.  Yeah, so just to quickly reiterate, he was asking my experiences with life preserver  and then my experience with jails.  So life preserver, I really haven't run into that many issues with.  Were you doing the SSH BAS, the ice-cozy?  My major issue was I actually was using snapshots and I was only able to use them on all snapshots.  Oh.  I became self-savvy for the best of the substance and I spoke to the body's work.  So I will say here that something that actually came from me doing snapshots before I had life preserver,  I'm still using a different piece of software to do snapshots and that's actually one of my to-do fixes  to get back to using life preservers.  So yeah, so something else to snapshots, life preserver is just doing some scrubs and the actual backup of the data.  Yeah.  But other than that, life preserver's been fairly great.  There's some things I need to look at and I wanted to get unmapped working,  which will work over on ice-cozy so that as stuff grows and shrinks,  we're not using too much space on the ice-cozy.  Here's all sparsely done.  There seems to be a bug right now though where Jelly's not passing that through and it's something that you kind of want to have the option of whether or not you want that to be passed through.  If you're us, you do want it passed through.  It makes so much more sense.  You want good reliable or good people not knowing how much data you have on that disk.  You don't want it to be passed through.  I'm just going to understand how less is the reward of the data that you've got.  The data that's cool on the hook and it's going to be about 10.  Okay, so I'll go through that one more time because as I said that...  One second.  So the basis of how the Life Preserver works in our setup,  you're getting the ice-cozy section of the replication,  we have a box that serves ice-cozy.  We have S-Tunnel running in that box that we can connect to from our other servers.  Our servers that are doing the backup, so the clients in this case,  launch an ice-cozy session that's preconfigured so that they have a raw disk available  that's coming from the ice-cozy server.  We then put... Jelly then gets put on top of that, so it's encrypted locally.  And then ZFS is put on top of that and ZFS send receive goes from the local ZFS pool  to the ice-cozy-based ZFS pool.  Some other nice things about this since there's not currently ZFS send receive compression,  decompresses, recompresses, instead of using something else in the middle,  all this actually happens local to the box.  And it also means that this scales very well because the compression doesn't at all...  The ice-cozy server in the back end doesn't at all care.  I've heard other people talk about doing this over very crappy internet connections  and the ice-cozy still works fine.  I obviously do it over a very, very fast connection.  I have a 10-gig server, so multiple servers can be kind of maxing out their sends.  Does that answer your question?  Is that okay anymore in that area that you're interested in?  Yeah, I think so.  Nope, it's going to a separate pool, which is very nice because it just uses all the ZFS infrastructure.  The other big thing Life Preserver does that I should not forget to mention  is it takes all user attributes and puts them under a prefix.  So your mount locations become Life Preserver colon mount point,  which stops them from getting mounted or other weird things from happening.  If you've ever played with Send Receive, you dash R, it doesn't get mounted the first time.  You reboot the system and all of a sudden it auto-mounts them and you're very not happy.  It deals with all this stuff.  That happens whether you use the SSH-based, which basically deals with it on top  and for the most part is very similar to normal ZFS send receive,  deals with controlling when it happens, things like that.  Or if you're doing over I SCSI, it's very similar.  Questions?  So I guess back to the question about jails.  I'm currently only using them in a few situations,  mostly to split out some production and testing places  so I can have them running on the same boxes.  And I'm also using jails to support my Edge web server.  It's actually an IMA web server, it's a reverse proxy  so that all public traffic comes to the reverse proxy and back to us.  Luckily, I only deal with our engineering side,  so I can restrict that down to only our customer address space  and a few additional things.  But one of these things is just we try to be as paranoid as possible,  which in our situation we have the ability to do.  I don't use any tools for that actually.  I just take whatever image is on the memstick,  install it into a skeleton, what I call the skeleton jail,  and then I ZFS clone that to every new jail  so that I'm not duplicating, you know,  I'm not using extra space that I don't need to.  And it's a handful of jails in a few places,  it's not a big thing that needs orchestration.  Does that answer your question?  I'll try to make it helpful.  Anything else?  Okay, then I think that's all I have for today.  Thank you very much.  Thank you.  If you want to get ahold of me, I'm on Twitter,  my personal email, my work email.  If you have any other questions, feel free to grab me.  I'll be right back at the conference. Thank you very much.
freebsd_8_to_10_T8sIibxw6aM_0_2375.png freebsd_8_to_10_T8sIibxw6aM_2376_6107.png freebsd_8_to_10_T8sIibxw6aM_6108_11027.png freebsd_8_to_10_T8sIibxw6aM_11028_14423.png freebsd_8_to_10_T8sIibxw6aM_14424_17135.png freebsd_8_to_10_T8sIibxw6aM_17136_20315.png freebsd_8_to_10_T8sIibxw6aM_20316_21923.png freebsd_8_to_10_T8sIibxw6aM_21924_23015.png freebsd_8_to_10_T8sIibxw6aM_23016_28211.png freebsd_8_to_10_T8sIibxw6aM_28212_28259.png freebsd_8_to_10_T8sIibxw6aM_28260_29195.png freebsd_8_to_10_T8sIibxw6aM_29196_30227.png freebsd_8_to_10_T8sIibxw6aM_30228_32867.png freebsd_8_to_10_T8sIibxw6aM_32868_33251.png freebsd_8_to_10_T8sIibxw6aM_33252_33347.png freebsd_8_to_10_T8sIibxw6aM_33348_35999.png freebsd_8_to_10_T8sIibxw6aM_36000_36959.png freebsd_8_to_10_T8sIibxw6aM_36960_38723.png freebsd_8_to_10_T8sIibxw6aM_38724_38771.png freebsd_8_to_10_T8sIibxw6aM_38772_38867.png freebsd_8_to_10_T8sIibxw6aM_38868_38963.png freebsd_8_to_10_T8sIibxw6aM_38964_39827.png freebsd_8_to_10_T8sIibxw6aM_39828_40643.png freebsd_8_to_10_T8sIibxw6aM_40644_41663.png freebsd_8_to_10_T8sIibxw6aM_41664_41879.png freebsd_8_to_10_T8sIibxw6aM_41880_42275.png freebsd_8_to_10_T8sIibxw6aM_42276_42563.png freebsd_8_to_10_T8sIibxw6aM_42564_42623.png freebsd_8_to_10_T8sIibxw6aM_42624_42707.png freebsd_8_to_10_T8sIibxw6aM_42708_42755.png freebsd_8_to_10_T8sIibxw6aM_42756_42815.png freebsd_8_to_10_T8sIibxw6aM_42816_44447.png freebsd_8_to_10_T8sIibxw6aM_44448_47807.png freebsd_8_to_10_T8sIibxw6aM_47808_48659.png freebsd_8_to_10_T8sIibxw6aM_48660_48707.png freebsd_8_to_10_T8sIibxw6aM_48708_48755.png freebsd_8_to_10_T8sIibxw6aM_48756_48803.png freebsd_8_to_10_T8sIibxw6aM_48804_52823.png freebsd_8_to_10_T8sIibxw6aM_52824_52871.png freebsd_8_to_10_T8sIibxw6aM_52872_54803.png freebsd_8_to_10_T8sIibxw6aM_54804_55439.png freebsd_8_to_10_T8sIibxw6aM_55440_59207.png freebsd_8_to_10_T8sIibxw6aM_59208_59723.png freebsd_8_to_10_T8sIibxw6aM_59724_60431.png freebsd_8_to_10_T8sIibxw6aM_60432_60479.png freebsd_8_to_10_T8sIibxw6aM_60480_60527.png freebsd_8_to_10_T8sIibxw6aM_60528_60575.png freebsd_8_to_10_T8sIibxw6aM_60576_60623.png freebsd_8_to_10_T8sIibxw6aM_60624_60731.png freebsd_8_to_10_T8sIibxw6aM_60732_66707.png freebsd_8_to_10_T8sIibxw6aM_66708_66755.png freebsd_8_to_10_T8sIibxw6aM_66756_66923.png freebsd_8_to_10_T8sIibxw6aM_66924_73559.png freebsd_8_to_10_T8sIibxw6aM_73560_74075.png freebsd_8_to_10_T8sIibxw6aM_74076_74089.png