Master & Minions or the Dream of Python Automation
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Hendryx-Parker, Calvin
In this day and age, using hand-crafted servers is quickly becoming a losing proposition. Debugging a Python application that runs on custom made servers gets extremely tricky, and since apps no longer run on a single server, releases become increasingly resource intensive. It is therefore vital to have a process to control multiple FreeBSD servers simultaneously in the cloud from a single command. This talk goes through real life and practical examples to roll out orchestration and configuration management projects using Salt on FreeBSD. After going over the pains of manual deployments, we will introduce the open source tools Saltstack and Cloud Custodian to automate the management of FreeBSD servers in a public cloud. We will then walk attendees through a real case study to give a practical understanding of what rolling out orchestration and configuration management means. We will end with hard learned tips to guard against common pitfalls. Saltstack is a event-driven orchestration and configuration management platform written in Python. It lets you run a command across thousands of servers at once through its remote execution function. Its master can listen to events such as user logins or CPU usage thresholds and automatically have minions take corrective actions. Cloud Custodian is an event-driven compliance platform that was originally developed in Python by Capital One. It allows for an organization to ensure compliance in its cloud infrastructure via policies. Cloud Custodian can make sure the correct encryption keys and ciphers are used. It can ensure new applications are always built on the latest machine images. Cloud Custodian can dramatically reduce costs by automatically removing unused cloud resources. With a mandate to migrate all of their applications to the AWS public cloud, the client had to find a way to move their highly custom website hosted on FreeBSD. We used Salt to build the new cloud infrastructure using the APIs provided by AWS. It allowed Notre Dame to deploy new environments for testing or, in an emergency, spin up instances in another region that match identically the current production environment. We also used Salt to power the release process so that the web team can release to the multiple servers that make up the environment with one single command orchestrated from the Salt master. They do not have to have deep knowledge of the servers and command line to release changes to the site. Upfront choices about the operating system or distribution are critical to making this process of deploying simpler. The natural choice for deploying onto AWS might be to use Amazon Linux, but it turns out it has an older, more limited package manager. If you are expecting to use newer features of certain software such as HAProxy, you will need to ensure your distribution supports it out of the box. Managing extra builds and compiling from source is a short term solution as it will add the overhead of maintaining the environment.
 So I want to welcome you all. Thank you for joining me today. In my session, I know I'm up against some specific competition in the other rooms. I think there's some good talks going on. So I'm glad you guys chose to come on here and hang out with me. My name's Calvin Hempick-Sparker, and I am CTO and co-founder of 6FDef, which is a Python and Cloud Consulting Company based in Indiana, and some of you have already looked up. But my history with BSD and mostly previous BD goes way back. I was mentioning earlier, I started using previous BD back in version 216, which was 1996, for those that were counting. And then I founded a company in 1999, back in the previous BD three eight-a's, if you all remember that kind of history. So we've been doing 20 years of previous BD work professionally. We have many previous, a couple hundred, few hundred previous BD boxes deployed in our infrastructure. And we managed most all of our, actually, 100% of them using Solstack. So today I'm going to talk about how we basically, I call this a dream of technical automation, but it's also my dream of BSD automation, because it actually worked really well for us and previous team. I say I'm kind of a quietly opinionated BSD user, but no more. I found that I probably should express my opinions a little more vocally about BSD and why I think it's actually great. And you'll see why I think it was tough, because it kind of burnt us. I'll discuss some stories about what we went through. But I've always tried to kind of suppress my opinions for fear of like, can you know, pushing my agenda onto others? But I think I will start pushing my agenda onto others a little more, because previous BD and BSD in general is definitely worth doing. So quick overview of what we're going to talk about today. We are going to go through a couple stories about some automation, taking some things from say a server closet into cloud. So we've got to talk about automation around, in our case, AWS cloud. But this is all pretty much valid for if you're doing automation in private cloud or any other public cloud, even though I'm talking about AWS, a lot of the tooling works across the platform, which is nice, which is why I chose some of these tools. Also, Pythonista, I have been using Pythons since the year 2000. So if you guys have Python questions, I love those as well. And then we'll talk mostly about BSD and automation here in this talk. Now there are some things in life that you would like to be handcrafted. For example, fine woodworking should be handcrafted, very artisanal type things. Jewelry should be handcrafted. And my personal favorite is that beverages should be handcrafted. So fine cocktails, the more handcrafted, the better that they are. But your infrastructure probably does not fall into this category of things that should be handcrafted. The age of handcrafted servers should be long over. The term bespoke should be reserved for fine clothing and not our servers or how we deploy our applications to our servers. So to contribute. The chances of a snowflake being exactly like another is one in one million trillion. So that's one all by 18 zeros. So beautiful snowflakes are not reproducible, generally speaking. And the same thing happens with your servers. If you actually handcraft, how many people would handcraft a server and never been able to get the same application around exactly the same way on another server? I've never had that fail or I've gone into a room. And there have definitely been cases in our life. Our company's like where, well, the thing magically worked and we set it up five years ago and the customer's like, well, we need to upgrade, move data center, change something around and we can now not get the same application to run again like it did. We basically have to figure out a way of lifting and shifting this ancient thing from one place to another to keep this one specific application running. We want to eliminate that from your lives and I want to eliminate it from my life because it really turns into a major headache, right? Things you can't maintain and you can't keep up to date. You also can't patch. We were talking about security earlier, the fact that they released a Windows XP patch says a lot about security. Now we love our pets, right? We give our pets cuddly names because we'll heard this like pets versus cattle analogy for servers. So it's actually coined by Randy Byas after hearing a talk from Bill Baker on scaling SQL servers. We want to not name your servers with cute names because then they matter to you and you care about them when they get sick. If they happen to pass, you'll be very, very sad and there'll be lots of sad users obviously that these things happen to these very cuddly servers. We want to actually have our servers be more like cattle. If part of the herd gets sick, you can push them off to the side and basically spin up new cattle in their place. Sorry if anybody's vegan in the room, but it does make for a nice metaphor when talking about server clusters or our infrastructure. We're down in this world of DevOps and many non-cloud-native applications actually have the ability to be deployed with cloud-native tools. So you can actually take what were these beautifully hand-catchered things that kind of sat in a single server and actually take advantage of various cloud technologies, AWS, and AWS building blocks to build up a more cloud-native version of our application, which actually makes it easier for us to repeat. So how do we take these beautiful hand-catched bare-knit servers, running our application, and translate that into the cloud? That's what I want to go over today with you guys. So real quick, I hope you guys all seem quite low. You know it's like 20 years old now? It's something I've seen before. No, not at all. So a little bit of credit to some of the half-news posts. My slides will have some links to where I kind of got a good start on these things. So the first rule of Fight Club, I always say you don't talk about Fight Club, I kind of change these around to be a little more DevOps-based for us. So the first rule of DevOps Club is you do not log into your servers. Second rule of DevOps Club, you do not log into your servers. And why do we do this, right? Any chance you have to kind of turn that thing into a beautiful snowflake, most system-invents pie will take advantage of that and do it. And they shouldn't. And that temptation to log in and just quickly fix that one little thing really needs to be dealt with at the top level, maybe like your DevOps or operational management leadership, is that we don't want to encourage people to log into servers because that does cause the proliferation of these unique snowflakes. Our third rule of DevOps Club is that the deployment fails rollback. So if you cannot accomplish this rule, you need to start putting in place processes that will allow you then to be able to rollback. Making sure you've got a virtual control system, some kind of continuous integration in a building or that you're building, pristine images of your application or you have a way of getting back to a previous state. You always need to make sure you have a way back because that allows you to move forward faster. If you have a comfort zone or a comfort with being able to move quickly forward, knowing that you can get back easily, that gives confidence level to your deployments that you don't have from before. If you know that it's the only way forward and there's no way back, that's kind of like burning all the ships, right, when you come to the new world. That can be really rough if something goes wrong. But if you know you got a way back, it makes new DevOps and automation a heck of a lot easier because you feel a lot more comfortable with it. Fourth rule, all artifacts are stored in source control. So this helps with the third rule. Make sure that all artifacts are stored in source control, not only for the rollback, but also for auditing. If you want to do code review, we're moving into a more of a world where management of our infrastructure looks like development. We can do the same things that developers do. We can do code review. We have full source control. We expect good commit messages. We want to understand what has happened to our infrastructure over time. And the only way we can do that is if we have source control in place. Our fifth rule, only one deployment at a time. We are still humans and our brains can typically handle kind of one task at a time, even though we think we can multitask. We're pretty terrible at it. It just does as a species. So if we can maintain one deployment at a time, that keeps us focused on making less human errors. Let's all be honest, right? The computers don't make errors. The computers just do what we tell them to do. So if we're making the errors, the computers will help and actually DevOps will help propagate it even faster. So errors can move a lot faster in a DevOps world. Six rule, actually one of my favorite ones. And we'll talk about specific ones for you to see. No one-offs and no special cases. If you start making those special edge cases, the complexity of the whole system to manage as a whole becomes much more difficult to deal with. It's just kind of common sense, but I think a lot of people see the easy way forward and oh, if I just happen to add in this third party package repo from some random site I found on Google, it'll make it all better right now. It will not make it all better later when that site becomes bit rotted and no one's maintaining it or if it becomes compromised and now people are putting code into those packages that you just randomly are trusting. So with the case of no-off and special cases, we try to find ways to take that natural path forward with our infrastructure and use the tools that are available to us in the way they're intended to be used. And as soon as we start bending that too much one way or another, we've got to find either another way to do it or find a way that works better with that natural path. The seventh rule, the plan was to go on as long as they have to. So much like the fights will go on as long as they have to, our deployments will go on as long as they have to so we can get things done and get things moving forward. So if the deployment is cut short and not in an unknown state, we better be able to exercise rule number three, which is to be able to roll back. And the eighth rule and final rule of DevOps Club, if this is your first metadev of DevOps Club, you have to push the product. There are a lot of organizations who are taking this mentality, especially on the development side, that we want to get frequent and often changes to production. And a lot of times they'll bring in new developers and because they do have a comfort level and a confidence level in the processes in place and we know we can roll back, they encourage new developers to make changes on day one of their employment that get pushed abroad. And I think the same thing should be held true for infrastructure people. If we're doing DevOps correctly, those should be low risk. They should be caught quickly either by continuous integration and unit testing and we should have the ability to roll back. But I think it gives the new person on the job a confidence level that they can build and make things happen and even if they break something, there's an easy way back. Please ask me questions all along the way. This can be as interactive as you want. Have you been influenced by Agile? It doesn't need to be exactly Agile, but I see a lot of... Yes. I think we have, so for our company 6VDV, we don't specifically use from the specific Agile methodologies, but Waterfall doesn't fit this model well at all. We really have this iterative cycle of keeping things moving forward as not as fast as possible, but as swiftly as possible. I think accelerating the velocity and changes into the infrastructure gives our end users more features faster. It also means sure that the changes are smaller, so if the change set is very, very small, it's much easier to roll back. If you have these giant monolithic change sets to either infrastructure or the software, that rollback process becomes almost exponentially as complex to be able to roll back out if you happen to have a failed release. So a lot of the way these things happen is by keeping changes small and keeping them in the same frequency and keeping them rolling forward. What do you think how far is the time when that will also want to be able to change that old energy? I think that's now for us. Our goal is that these are building blocks. I'll show you. We've taken these non-cognitive applications. So it's like a lot of people may run legacy apps where it tended to have a lot of manipulation of the systems that get these apps running. I think there is ways, for example, tools like Solstack, Pubg, Ansible, that allow you to do all that configuration from a, like, I want to configure it locally on my own machine. I want to be able to test that configuration. So actually having continuous integration where I can run an automated test process, and then the release process is fully hands-off. Like, we want to push button as possible. And new releases of the software will follow along that same line as well. And so I think even for a small company, we're like 20 people, and we're able to do this. I understand maybe the enterprise feels like it's that giant aircraft carrier that turns really, really slowly. I think that because these changes can be small and you can start implementing these kinds of methodologies in a small fashion, you can sneak them in and start doing it today. And that's my opinion on this. There's a suggestion that they won't hear the question that you should repeat. I'll share a few questions. The question we were just answering was, how long until you think we'll get into a point where we won't SSA-tion to servers and we'll deploy without any interaction? I think we're there today. I think a lot of organizations better be thinking about moving that direction. I mean, if you guys are coming up to Amazon Web Services Cloud, which probably most of us are, you know that they have how many three-letter acronyms, TLA's, services available. Those are all basically virtual machines running on some hardware someplace, but they've configured them to be quickly deployable as a service. I think our EC2 instances of our application should be thought of in the same process. Like it should be nearly point-and-click to deploy our web application into an EC2 instance or to scale out a process, load balancer, leveraging those other backend pieces that Amazon gives you for the serialization storage of the data layer. Our things can be replaceable. The application code runs basically in almost a stateless container. Actually, there's another take on this welcoming DevOps Club. You guys know the movie Back to Back Club. There's actually a welcoming take on DevOps Club from a blog which is linked into my site as well. Those will be up whenever Dan releases a next version of the Viet-Viet site. He'll be on the public-use site. We're talking about DevOps Club. When we're in this world, we need to talk about tools. For us, we use Python. Luckily, the Amazon Web Services platform is like Python-first, almost. They've developed and actually released the Bado Free Library for dealing with the APIs. The other part we use is Salt Stack. I mentioned wearing an OG, like old school Salt Stack shirt today in honor of the talk. That's also written in Python. It fits well with our Python and Cloud resulting mindset. I think it's easy to use for a lot of people who even touch Python or have never touched Python because most of the time, they're editing YAML, which is a very human-friendly, readable data format that you can type out and plain text to set the states and things. I'll show some examples of what that looks like as we go along through the slides. The use case I want to talk about first is taking that single, this was literally a server at a closet. We went from the closet to the cloud, we'll discuss for you. It is the university of Notre Dame had their engineering website. I think it was literally within some closet, in some building on campus, which is probably not uncommon for these kinds of groups. But they had a mandate to move everything into the cloud. Given that mandate, we thought this was an excellent opportunity to automate the release process of this so that it could require less technical expertise to actually be able to release your software. With this current stack of software, there's obviously the web properties like engine X and varnish, and HAProxy. In this case, it's a content managed like this public-fixing website. If you go to engineering.indy.edu, you'll land on the site, not running here anymore, but in the new infrastructure. This is running a content managed like Gulf loan, which has a lot of interesting things at a single backend storage layer where all the data actually lived. We wanted to be able to deploy this in a cloud-native way so that if one server went down, we didn't care. We could actually spin up in number of servers into a load balancer and have it take care of the load, or if they, for example, had a dignitary visiting campus, we wanted to build a handle of additional capacity and have that kind of elasticness in the promise of abandoning the cloud services. The promise of abandoning the cloud services is there. You can do it, but you have to tell it to do it, and it'll just do it for you. There's nothing automatic about being in the cloud. It still is, literally, just other people's servers. Luckily, they give you nice APIs to be able to interact and perform these things in an automated fashion if you use the right tools. We want to take this server in a cloud, or server in a closet, up into the cloud. Again, going to the closet of the cloud. This is, again, an extra opportunity to take this aging simple server and improve their performance and resilience. Give them some high availability aspects that we are going to be in a public cloud and have access to resources easily. We don't have to wait a month for a server to be in the prison and deployed into their closet. We can actually grab those resources as needed. How do we make it repeatable? For example, you could go into the Amazon Web Services console. You can click around to your heart's content. This is nearly as evil as logging into your servers over at the stage and configuring things by hand. You'll never get it the same way twice. In our case, we wanted to leverage those cloud-made technologies, for example, Cloud Front for CDN. Now, instead of having a single, like a varnish server, we have cash from a single point on the planet, we can now take advantage of the 145 points of presence that the Cloud Front CDN gives us. We want to be able to use a load balancer that can allow us to dynamically add new instances in and out of that load balancer, so using the elastic load balancer service. The AD1 and AD2 boxes here are two EC2 instances, one in each availability zone. So inside of Amazon regions, you have availability zones, which are different data centers separated by some minimum amount of distance. But that gives us that high availability in case one data center goes offline for some reason. Someone takes a whole, cuts a line, whatever case they need. Those EC2 instances, those are the only virtual machines on here that you could SSH into. All the other services here are managed by the Amazon Web Services Cloud. But those two servers that are on here contain no data we care about. None of the data for the site or the storage of the files happens in that layer. This allows us then to make them more like cattle. They can be taken offline, other servers can come up and replace them inside this infrastructure. We do put the data into some other Amazon components. Like EFAS, which is their hosted database service, and then RDS, which is their hosted database service. So they give us the ability to store our data into places where we can bring our servers on and offline, quickly build a brand of image, deploy those images out of EC2s for this, and have them connected to our existing data storage and they're online in, could be seconds. And really they can spend a very, very fast. And then everything is backed up into S3 for the durability factor of having a resilient backup. You could point and click your way through this, but it would be tricky, especially if you start getting into the VPCs and networking and the private subnets and the public subnets and all the other pieces that are involved here. So into the automation. Obviously we need to be able to have a way to point and click and have a repeatable way for us to put those pieces into place. So we'll kind of start at the network with it. That's probably one of the trickiest bits of Amazon to figure out. They give you a default VPC when you log into your first Amazon account. It's generally useful to kind of try some things out, kind of prove a concept type stuff, but it's not really recommended for production. You're going to want to set up different VPCs for different purposes. In our case, we want to be able to have this thing be push-button that it actually did. The automations will deploy out to the full network effect for any environment we want. So if we have a production environment, we want to build a test environment that matches it at a moment's notice, or we want to build another test environment inside it so we can do performance testing and not impact maybe a test around what's happening in a specific other environment. So we want them all to match. There can't be differences here because they're going to run into those strange edge cases, something will fail in one case and not in another. But we need a way to control this. So we set up basically a control VPC where we put our salt master and VPN, so that is the administration point for all the tools. I want to go so far as to make sure that even that control VPC is set in an automated fashion. I don't want to manually configure anything through the console or shell into a box to get this all actually bootstrapped. So this is where we start using SaltStack and Vado3. And we recently use Vado3 as we have to have a way to bootstrap that salt master. Now I'll talk a little bit more about the master and manual process of SaltStack because that's actually how it runs, uses a control and then it's a kind of agent on your EC2 services or to talk to the Amazon Cloud. But we'll use Vado3 to actually get these bootstrapped. So we do have a bit of a chicken and egg problem. How do we get the salt master in place so we can build the other VPCs without manually going in and clicking and creating so that's. Luckily the Vado3 is very easy to use. So if you're familiar with Python, it's very, you know, the world of Python called this Pythonic. It kind of follows the tenets of the language of Python. So actually another recommendation for you all, if you guys have a Python prompt on your computer, is to open up Python and type import this and you'll get a nice Zen of Python, which kind of may explain why Python needs to think the way they think. So that's the least we're going to try out. But in, you know, Python and using the Vado3 library, we can create a new VPC with code. In our case, to create a cold VPC, we pass in the insider block of IP addresses for that VPC. We can add a tag in there for the cold VPC and we can add some DNS attributes for that VPC. And that's basically all the code is needed to get started with that controlled VPC. So we wrote this into a single script about less than 100 lines long that boosts drafts that VPC for us. What's also nice about SaltStack is it does have public cloud and private cloud support. So we can talk to AWS at your digital ocean or if you're in VMware, internal infrastructure, you can talk to VMware and do similar things. This is not limited to only AWS work. You can actually use it in your own infrastructure as well. And so to do all this, you have to have credentials. You have to have your APIs, your secret key, and your secret bits of information you need to use. And as a service to you all, I want to make sure that you all don't end up as a next headline on hacker news and place for exposing your API keys. Make sure you have some mechanism for storing your keys securely. On my laptop, I have no keys written onto the file system that are not encrypted in Subnaynor. And typically, I don't actually keep them on my laptop anyway. I use it as a tool. I present, we use LastPass for password management. We were using LastPass literally this week. We were using one password. What's nice about those tools is they do have command line integrations typically. So LastPass has a LastPass command line tool that has someone who wrote a wrapper script for it called LpassM that allows me to basically grab the environment variables out of my LastPass store and put them into my environment just for that session. As soon as I closed that shell, they were only over in my memory. They are no longer, they're not written anywhere on my computer. One password of similar feature is an OP command line tool. You can get a specific item out of your laptop, your one password vault, grab, for example, that one uses properties. So I basically set properties for the TD and secret. And then I export them directly into my environment whenever I start running these tools and start working on that project. If you want a little more sophisticated version of these kinds of things, you may want to look into tools like Hashtag or Vault where you can start sort of having an infrastructure for storing your secrets. But please, please, please don't do what the documentation says. Where the adress, CLI, where they tell you to put it into like a.file on your home directory. Yeah, I think it's sure it's easy to get going. It's also very easy to get hacked. Anyone who has access to your computer could possibly think of those credentials and start using your APIs. And spending a lot of money very quickly if they really knew what they were doing. Or if they didn't know what they were doing. In this case, yeah, this environment grabbed in the access key. And the secret key, the secret access key, and the TID was also nice with the environment variables with the BOTO3 library. You can specify other kinds of configurations. So there are typically other configurations like the region. You can also specify specific profiles. There are many other kind of settings you can set via the environment variables. If you store those all in the vault right next to the keys, importing them is trivial because I basically have a one-liner that gets my environment activated and ready to use with whatever project I'm working on. So from here, we initialize an orc. I don't expect you to be able to read all that code. We initialize the initial VPC and we will spin up an EC2 instance into the VPC, our control VPC for our soltmexter, and in our case, we have a bootstrap sh that we will read in and put into our user data. So each EC2 instance, you can pass in a shell script or user in that script to basically install the initial code onto our digital package on that server. So we do everything, including pulling out our Git repository of the configurations on that machine on one script. So no humans touch the box, it starts up, installs solt, grabs the latest set of states for our solt integration and ready to go without any human interaction and then write a script. So we basically start up an instance. You pass in the user data, so that's our bootstrap sh script that installs private keys, sets up the repositories, it makes everything ready to go for us to now deploy the production environment or deploy the testing environment. One thing to watch out for, a little tip here when you're doing these kind of things, the cloud infrastructure is not instantaneous. You may run some of these commands and they may return instantly, but the infrastructure may not be there for you to use yet. So in our case, if you want to do something like add a tag to that instance, that may work, but if you want to actually interact with the instance, it may not be there just because of the asynchronous nature of the spinning up resources in the cloud. A specific example of this is if you want to associate a last IP directly to the EC2 instance, you cannot do that until the instance is actually in a running state. Now EC2's API on Bado does have some processes for doing this, or you can just put in some sleeves in your code that's kind of the format way to do it. But if you want to do the more fully asynchronous, there is a waiter method of the Bado 3 code where you can say, wait until the instance has started and then continue my code. So it basically does like a polling of the API to see when your actual instance is ready to go before you move on to the next steps. Yeah. Question one. Let's see, when you keep numbering your team, who is that server right now? How are you getting these keys as a key or not as a page or whatever identification? So in this case for this specific problem, we will do these per project and we'll have a deploy key in the salt method that is shared with the team, or you could deploy in the SSH copy the keys into the control servers. So that those control servers are distributing or we are logging the control server to check for that source? They would log in the control server. You would log in the control server to run the salt command to build the other infrastructure. What we typically have is an SSH key shared along with the team for that project that would have in there on their own computer to log in. And let's say you have internet money. Do you have an internet money process to check internet money and start it is not money for G-Goran alert? So that would be separate from our animations. We will put in place, we use Zadx for a lot of our monitoring. We are actually going to be moving to use Informetius for that. So we will hook Informetius to Grafana or Elastic stack for those kinds of monitoring. You could use CloudWatch, depending on what cloud you are in, using it on the cloud, you could use CloudWatch for those kinds of notifications. So that is kind of a lot more, there are a lot more choices there on how you want to handle that. So you go in our environment, if you see like I directly found that the SSH and then we check what is up. So you can do that. Basically everything happens from the master. So you will log in the master. You can do a service status from the master, like cross like your state of complete of like 10 internet servers in the front end. You can see the status of all 10 of them simultaneously because you can use what I will give right to that in a moment. Alright, because that is one of the features I like about Sol compared to some of the other tools, where it gives us the ability to do remote commands and not just configuration management. Which is what I think would be a reaction. So what is SolSTAC? This is a perfect segue. I did not plan it, I swear. So SolSTAC is an event driven orchestration run that hang on. It is more than just configuration management. So some of the things that I like about Sol that make it different than say public chef or Ansible is in fact you do have remote execution. So I can log into the Sol Mastery and interactively query my whole environment or send commands to the whole environment or to a subset of my environment based on lots of different criteria. So you can query and select machines based on values and certain variables or network sub paths or the laws of the name or if you can set up different kinds of ways. And you can actually do compound query methods against your environment. So you can select all the web servers or all the internet nodes and say server status. And you can do server start for example across that same level of nodes and start all 10 of them simultaneously if you wanted to. A lot of the tools don't have the level of interaction with your machines. With the remote execution you can also populate say for example databases of your machine information. So if you wanted to grab, you know, with the CPU processor, network cards, serial numbers, you know, all the identification information or any kind of metadata from the machine, you can do that as well with that remote execution. And then output it into, you know, my SQL database, Postgres database, LDAP and wherever you want to land that kind of data, you know, whatever, however your infrastructure is managed, you can do that with Solstack very, very quickly. The Solstack also has support for, like I said, the various public clouds. It also has built-in modules for things like curl. So you can get, you can access the AWS metadata server and actually assign metadata into the machines based on metadata that's been set at kind of a higher Amazon level, like tags, or you can get data about coming from the metadata service you wouldn't normally have access to just on a machine itself. So that's the remote execution. The next really nice feature about Solstack is the event driven orchestration. Since you are using typically Solstack in Master and Minion, you have a server with agents running on all your various other servers in your infrastructure. You can use that to, you know, not only query in your remote execution, but listen for events coming across the event bus. So Solstack uses 0 and Q for a message transmission across all the servers. So it's very, very fast. You can actually execute things in parallel across a thousand servers and with very low overhead. Because of this Master Minion architecture, the UDP or UDP is for UDP and TCP. So 0 and Q, I'm not 100% certain on which one I'm always using, but I think you can use both. I know you for the firewall chip, like 4506 on UDP and TCP or in the talk. So the Minions will register in with the Master. At that point they're on the message bus with the rest of the machines in the architecture. You can now listen for events coming across. So there's going to be salt specific events. For example, the new Minion comes online. There'll be an event saying it's new Minions available. If a Minion goes offline, there'll be events likewise from the Master saying I've lost track of a Minion. And then you can interact with it based on that. So you can now take an action based on a Minion disappearing. You can also set a beacon on the machines. You can watch for activity and send specials, your own custom events across that same bus. So if you, for example, are watching for someone has a Sage engine server and they shouldn't be, you can actually interact with it. So if someone has a Sage engine that can send a custom event across the bus, you can have what's called a reactor on the salt side to then take action. You know, either sending a wall message to them or some custom interaction you want to have happen. You can use it for monitoring. Some people have used, for example, a graphite across the, this is a transport mechanism for your monitoring. So you could send from any other kind of data you wanted across that bus back to the Master event to deal with that as events. So the event reactor may be to put it into the Master database of monitoring data. So you can have an agent list, but the common way you run this is with Masters and Minions. It does support an SSH transport like the Ansible, the FIght is from there, the Ansible. You can use a similar transport Ansible where it does send it to all the machines. Obviously you sacrifice or give up performance in that case versus being able to have this agent list operation. It can support cloud provision. So there are, the cloud library is built into it for specifically spending on cloud infrastructure. In our case, we bootstrapped an Amazon EC2 instance using the bottom three library to get us started. But from here on out, we're going to use Solstack and its native cloud libraries to talk to Amazon and provision infrastructure for our projects. And then I like the fact that the speed and scalability of using Xero and Q, it gives us that scalability across say thousands of servers. You can also set up like sub-Masters. If you had, for example, wanted to have a master in each data center, they would all talk to that centralized master. They all act like a master, but they talk locally to their own sets of machines. So if you have, you know, managed your directly with distributed machines, you can actually coordinate between the master and what are called syntax to go to talk to the machines that are closer to physically the fleets of machines you'd want to actually manage and maintain. So this is similar, like I said, to some of the tools, but I think gives you this additional benefits over what you would get with a Chef puppet. For example, we can use fabric for orchestration, but it doesn't handle a lot of configuration management. You can use Terraform for cloud provision, but it doesn't handle the orchestration pieces. This brings a lot of that into one nice tool. Now, I will say we are using Terraform more and more for some of the infrastructure and then maintaining Solstack for our configuration management and orchestration pieces, that more cloud orchestration like a real-time piece. And Terraform seems to be better about keeping up with all the latest APIs. Amazon is a bit of a move that kind of sands of the desert type thing that it moves very quickly underneath you. And so whatever tools you use, you have to make sure they keep up with the APIs underneath as quickly. So, again, I want to reiterate, it's always best when millions have their master. You know, millions are very, very sad if they don't have a master. They can work just not as effectively or efficiently. So, let's make sure we actually use a master with our millions. I have a silly question. Yes. How long was your discussion about the replacing master's sleep terminology? Luckily, I didn't have to be a part of that. That's been there. I've been using Solstack since version 0.10. It's been about seven years now, I think, or so. And it was a rough at the beginning. I would say that things are very much more stable as far as the APIs of Solstack itself. One of the things I liked about Solstack early on was they were moving very fast. And they still move very fast as far as development speed. But they're not doing the same kind of breaking changes that they were doing early on. I think they saw one and had a bad idea and they addressed it quickly. Unfortunately for some of us users, we just kind of had to bear it. But it was definitely rough, but obviously I didn't run away. I'm still here, I'm still happy with Solstack. It's very stable now. So they have a new numbering scheme and the user tracker is very active. As far as an open source project goes, I find them to be a very good model of a corporation or company running an open source project. It is not an open core type project. You guys are going to be running an open source licensing and have some people license out their tools. Their Solstack server is the exact same Solstack you use if you have their enterprise product. The only thing the enterprise product adds on top of that is going to be a UI that kind of has ACLs and automation pieces that are more point-click for the enterprise. But the underlying Solstack Solstack server you run is exactly the same one as part of what's included with the enterprise version. You're not running a version behind or beta version or some of that open core software that does this to get you to buy everything. Solstack is going to do that. They're going to play really nicely. The issue tracker people will respond very quickly. So if you do have issues, I highly recommend going to GitHub, putting in issues. They take pull requests. I contributed that pull request for previous D support in the bootstrap script and it got merged in 15 minutes. I don't think they do it that fast anymore because it was back in the old days when things were moving fast and quick. But I would say they do watch that very, very closely. Now, when we orchestrate our servers, for example, if we want to run a deployment to a specific environment and build out those VPCs, every aspect of our infrastructure is going to be defined by what Solstack calls pillars and states. So you think of the states as like classes or kind of the skeleton structure of how a service would run. And pillars are the data which filter and fill in the variables. So you can reuse those states across multiple projects or multiple clients. You may refactor it into what's called a formula. For example, if you always want to deploy Postgres and you have a standard way that if your company always deploys Postgres, you would use that across multiple projects, but the pillar data would be what's different, like database names, the HBA, like authentication bits, the passwords, users, that would all be in pillar data. And pillar data is secured by the gloss. So for example, if I have only the database servers in production, I have access to the production pillar data. The staging database servers do not, or the dev database servers do not access that data. So they cannot, your developers, for example, would not have authority or access to the production database passwords if you're using the pillar data correctly, which is how we do it. In our case, we actually want to be able to use that same set of orchestrations across multiple variants. So in the first instance here, we're running our orchestration to build the VPCs, the private subnets, all the NAT gateways, Internet gateways, elastic IPs, all the stuff that basically gets pushed into Amazon VPC is done in this deployed environment, and we pass it in which environment we're going to build out. So if we want to build five more test environments, we would build pillar data for those other environments, because you may have different IP, like Cydre blocks, they're really different, so you don't overlap across two VPCs with the same IP blocks. You can just basically set that up in your pillar data for that environment, run this exact same command and it gets that different data and then you're in the fresh environment that's exactly matching the previous environment is deployed for. So no more unique snowflakes, and we actually have a repeatable way of deploying multiple environments. So this is running with our matching dominions. They're very happy. So now without humans, we're able to deploy all of this. The control infrastructure is up. That orchestration we just looked at, deploying these two VPCs with the private and public subnets, and then inside of each of those subnets, we get the various infrastructure pieces now built out. So we have them defined inside of our Solstack infrastructure. What size, ECT instances, what images to deploy, it even goes so far to also provision the EFS, which is the NFS storage, the RDS, which sets up the database backend, it's the ANCIs database user that sets the passwords. The code gets deployed via Solstack as well. So as part of this full orchestration, the Solstack will pull out the code and get however, or whoever, get back and we're using. It will use the same pillar data. So the pillar data is that canonical source of authentication information or custom variables for that environment. It will actually set into the code the database passwords for us. So again, the practice is not to store our database passwords anywhere that is not easily accessible. So none of our code repositories for the developers contain any passwords for any environments. And that's all coming from Sol. So when you get deployed in an environment, Sol handles that for us automatically. They're putting the passwords in the code for the right places for the connection strings for us. So note that all of this is deployed in one go for one deploy environment command. That may take a few minutes on the first run, but then subsequent runs, so for example, if you wanted to add in additional servers, maybe you wanted to expand this load balancer to include two more servers, you would run the exact same command. It would ensure that those two extra servers are brought online, put into the load balancer, hooked up with the right password, deployed the right code, all behind the scenes without having to redo all this. It would basically ensure that the environment is in a state you could define for it. So now I talked a little bit about code deployment. When you go to deploy code, you typically don't want to deploy the whole infrastructure every time. So this is where Sol comes into play with its orchestration. So we've used it for configuration management, and we used it to orchestrate building the infrastructure. We can also now use it to orchestrate releasing code to the servers. One of our customers, the Central New Mexico Community College, and they run a VMware cloud internally, and we use SolStack in their infrastructure to do this exact type of task. And maybe we do this because as we start deploying across more and more servers, things can get tricky, right? It can get more complicated to deploy. Developers and the people who are doing the code don't have time to understand a lot of that infrastructure. So people who deploy and use kind of this point click infrastructure or point click deploy of code, and we've got a big one for them. So in our case, we can schedule or make, excuse me, we can schedule or allow the developers to do like zero downtime releases by orchestrating every piece that released for them. So in this case, we loop over every app server. So the orchestrations are just Genja templates of YAML. So if you understand Genja or use Genja Templating, you've got all the loops and helpers and everything available to you. So we dynamically determine what the app servers are and then loop over each intern deploying code. Now what we do is we'll, for example, stop the varnish server which takes it out of the load balancer. We actually switched this up now to use AHA proxy and then API to top AHA proxy to take the machines out of the load balancer, perform the release, make sure the instances are back live again before we put them back into the load balancer. So we can actually do this, you know, basically step through the whole environment, making sure that servers are back online before we put them back in load balancer. And then for the developer, they just run one command and then kind of wait for the output. If there's any errors along the way, salt stack will stop, enough receipt to thrash the rest of your environment. It'll basically stop you at a point where the machine's no longer in load balancer, but something may be broken. The site's still alive because the other instances that are part of the cluster are still handling the site. So in this case, you know, we run the control, the instances stop them. We may roll out a rollout specific code. In our case, we may have the dev branch, the default code, if you release a production, we can specify that as part of the orchestration. This also allows them to roll back. They can pass in a variable for which, like, get tag or get revision they want, and it'll release that specific code out to the servers for them. They've never had to log into any application server, so doing this all from the salt master. And then release tasks. So if the release is more than just put code on a place and reserve a server, you may have to do things like run or gulp, like to do some kind of pre-completion of CSS and JavaScript. Those kinds of tasks all now happen as part of the salt orchestration. This is all part of one big orchestration file. This is the gulp that's here, so they actually put in gulp and then install the new components that came in as part of the theme or the front end of the website. Those all get handled automatically because humans will typically forget to do this on release and then you end up with a broken site and no one's happy. And that's not just releases. We actually use the same technology orchestration to do things like bring production data from production back to the dev environment. Because it involves typically not only dumping the databases, but they want dev, dropping them, and then grounding, for example, the blog database or blog storage. The site, the technology we use splits the large data files into a hash directory structure. Then we can do a little synchronized back to the dev environment in sync with the database at the same time. And humans are not good at this thing. Computers are very good at it. And now the developers who are non-technical about the mainline can basically run one command and get their data on the dev environment refresh from production. We also use the first scheduling database backups so we can finally have it run during a maintenance window, switching and knowing which machine is currently the slave. So we run a Postgres cluster with a master and a slave. And we only ever want to run our database backups against the slave since it's in read-only mode. So we'll have you know which one is the slave. So I should give us the ability to query the machines with their mode execution, determine which machine is actually running a slave, and then only run the database backups on the slave machine. So it doesn't matter if they had failed over some time during the day, it doesn't matter which one's master or slave. The scripts, the sol orchestration is figured out for us. Kind of running out of time. So I'll give you a little bit of a preview as you go on. But there are some more stuff in my slides about using Cloud Presedient for management of hours, for example running dev VMs only during business hours, and have them turn on and off automatically using a tool called Cloud Presedient. It's also written in Python and also uses a bottom 3 API on the back end. It's also nice for doing compliance issues. So if people are writing S3 buckets with keys that are unencrypted, you can actually self-heal that type of activity because it runs on AWS Lambda, listening for Cloud Trail events such as like new keys or EC2 instances being spun up, maybe with old AMIs. You can then have multi-step work to learn the user group, spun it up and then shut it down if they don't react with a certain number of days. And they're getting grabby charts and grabby production data, they can't get over that. Sound shooting, that was not definitely as easy as they'd make it all sound. The special cases thing, this is where Pre-VT comes into play. Instead of Python specifying there should be one and preferably one way to do something, but it may not be obvious and let you dutch. This is where the non-obvious part came into play. We did not want to have special cases for installing certain kinds of packages on the machines for the dependencies of the application itself. So for Nipple, WV or HAProxy, we needed specific new versions of those tools. So like what great, we'll use AMNL Linux because that's like the native Linux for run AWS. Practical repositories are old, sad. We will practice in top seven because it will be newer, not new enough. And for both, for us, PreviousD package system is awesome and actually allowed us to run with just the standard package repositories and get the versions of the tools we needed to run our play. It was painful and we spent probably two weeks wrestling with this, but I really stuck the line of not having special cases. We did not want to install third party repositories onto these Linux distributions, only to have them get run or go out of maintenance for us. Recommendations for you guys, if you do start using this, there are testing tools. You can integrate this with continuous integration. So there's one called PictionSol that allows you to test your salt states. You can actually write states. It'll spin up in multiple environments across multiple OSs and actually run your states against each of them to see if they would fail in any way. So running that continuous integration every time you make a change, that gets you to that confidence level to make sure that rollbacks are possible. And then I've got some examples using Bottle3 in my GitHub, or our company GitHub, we're doing pretty wrong log tools using Bottle3 for tracing through logs for requests. Are we living the dream? I would say absolutely living the dream because I got to use PreviousD in the cloud natively and saved my life because I was able to use BSD. And I'm very happy to have you. I do want to thank you all for sticking with me. I won't even play eight minutes over, so sorry. Forgive me, but if you have any questions. I'm not sure if they're actually early. Am I? I'm 17 or I'm at 12.15. It does. Okay, well then I got time for quite a couple of questions because I love you. How did you guys come off that stupid start? I think the whole thing started kind of looking back at those rules of like DevOps Hub. How can I make sure that we don't log in servers, that things are repeatable, we can roll back, and that I can get a consistent environment every time I do a release? Obviously, we have our own people. So we, I mean, people who are working on log-in deployments from an infrastructure standpoint, it's really only about two people, one or two people who are handling the infrastructure pieces for these various projects. So I like the fact that it doesn't require a lot of overhead and there's not a lot of maintenance to it. I mean, to kind of say forget it, because if something goes wrong, we can just tear down a server and replace it in a moment's notice with a new server running on the same code. Typically nothing goes wrong because no one's logging in the servers to make up in a unique snowflakes. You know, knock on wood, right? But I think it's just thinking about how to make things repeatable. We run into this problem a lot as developers. How do I get a new developer to just join the team to spawn up quickly to be productive? And so it's a lot about making things repeatable even at the desktop level. Like we'll use soft stacker vagrant to even get like VMs locally. A lot of our infrastructure pieces here, we have local vagrant files that will spin up with mini cluster on your laptop so I can test things out before I push into the dev environment or production environment. How do you deal when you have to update the orchestrating part? When you have to do big change to the infrastructure at the higher level? How do you deal to it? Because it seems to me if you want to change maybe the connection, how it's wired up. Oh, the infrastructure part? Because I see how it's easy to orchestrate the deployment. But for the infrastructure, maybe you have like snowflakes or infrastructure because you have the test. So that is probably the reason why we're switching to Terraform for a lot of the infrastructure pieces. If you're in a framework with Terraform, you produce a model of how you want the end result to look like and then calculate the differences from where you get that now. And so with Solstack, you have to more declaratively say I want to change this infrastructure from that infrastructure. Terraform will figure that part out for you and can make that change. Obviously you want to improve the apply before you look at what the difference is going to be because then you may destroy things along the way inadvertently. But Terraform is probably more powerful when it comes to doing that than that. And how does it integrate? Are there really something that you're switching completely? No, not necessarily. Solstack has a Terraform module built into it. So we can actually execute Terraform models against your environment so that you can actually do that integration. Another piece we're probably using, the same company who does Terraform, who does background, probably sensing a theme here that I like to have to show folks, they also make a tool called Packer for creating images. And then Packer supports using Solstack to build the images. So our same Solstack to build these infrastructure kind of live can be the same Solstack to produce images that we can deploy instantly. As you probably know, Solstack is a developer that we know that we have an exciting situation with my iLotify. With which one? With iLotify because of deep insight, etc. So are there any plans in Solstack that we need to install something that we can just support? Thank you. Probably? That I don't know. Okay, I don't have a good answer on that. Okay, okay. Does the Solstack master have a... You said you can get it to a stable pass. Yes. Are you just running Solstack as a prong job? No, so Solstack has its own internal scheduler. So for example, with the database backups to having it run at certain times, you can schedule kind of like app jobs, like one off, do this thing at a certain time, or you can say schedule on a certain schedule, either periodically at a specific time or every September, you know, hours, minutes, run at a certain internal schedule. So I like doing that over prong jobs because now everything is kind of centralized around Solstack. I'm not having Solstack install prong jobs on the road, but I'll have Solstack actually automate those tasks and run them on the various minions for me. Because at that point I get feedback from a centralized spot. But I actually wanted to take the output of every one of those prong jobs or tasks and dump them into Elastic. I can now quicker across them or get notifications or get graphs of the frequency of the runs and also frequency of errors. So instead of relying on, you know, cron emailing in my inbox, then saying that I went wrong, I can actually have a fully automated feedback loop that monitors the tasks that you're running. And I'll be back. So the feedback that you get from this long version is in the sun, do you use this work to Elastic search? You can. So there's play support inside of Solstack for many different outputters, like where you want to send the output of the commands, like whether it goes into database. Natively you just get like a JSON, like a dump or a plain type of output. You can get it a JSON or a CSV or if you have an application you want to stuff it into like Elastic, or if you want to put it on Kibana, because you think, I don't know Kibana, but I don't know Kafka. Like if you've got a messaging service, you know, Enterprise-Wide, and other things you're listening for, that type of information and consuming it, you can actually have it done on your com. Cool. Thank you all. Thank you. You're welcome. Thank you. You're welcome. Thank you. Thank you.
 So I want to welcome you all.  Thank you for joining me today.  In my session, I know I'm up against some specific competition in the other rooms.  I think there's some good talks going on.  So I'm glad you guys chose to come on here and hang out with me.  My name's Calvin Hempick-Sparker, and I am CTO and co-founder of 6FDef, which is a Python  and Cloud Consulting Company based in Indiana, and some of you have already looked up.  But my history with BSD and mostly previous BD goes way back.  I was mentioning earlier, I started using previous BD back in version 216, which was  1996, for those that were counting.  And then I founded a company in 1999, back in the previous BD three eight-a's, if you  all remember that kind of history.  So we've been doing 20 years of previous BD work professionally.  We have many previous, a couple hundred, few hundred previous BD boxes deployed in our  infrastructure.  And we managed most all of our, actually, 100% of them using Solstack.  So today I'm going to talk about how we basically, I call this a dream of technical automation,  but it's also my dream of BSD automation, because it actually worked really well for  us and previous team.  I say I'm kind of a quietly opinionated BSD user, but no more.  I found that I probably should express my opinions a little more vocally about BSD and  why I think it's actually great.  And you'll see why I think it was tough, because it kind of burnt us.  I'll discuss some stories about what we went through.  But I've always tried to kind of suppress my opinions for fear of like, can you know,  pushing my agenda onto others?  But I think I will start pushing my agenda onto others a little more, because previous  BD and BSD in general is definitely worth doing.  So quick overview of what we're going to talk about today.  We are going to go through a couple stories about some automation, taking some things  from say a server closet into cloud.  So we've got to talk about automation around, in our case, AWS cloud.  But this is all pretty much valid for if you're doing automation in private cloud or any other  public cloud, even though I'm talking about AWS, a lot of the tooling works across the  platform, which is nice, which is why I chose some of these tools.  Also, Pythonista, I have been using Pythons since the year 2000.  So if you guys have Python questions, I love those as well.  And then we'll talk mostly about BSD and automation here in this talk.  Now there are some things in life that you would like to be handcrafted.  For example, fine woodworking should be handcrafted, very artisanal type things.  Jewelry should be handcrafted.  And my personal favorite is that beverages should be handcrafted.  So fine cocktails, the more handcrafted, the better that they are.  But your infrastructure probably does not fall into this category of things that should  be handcrafted.  The age of handcrafted servers should be long over.  The term bespoke should be reserved for fine clothing and not our servers or how we deploy  our applications to our servers.  So to contribute.  The chances of a snowflake being exactly like another is one in one million trillion.  So that's one all by 18 zeros.  So beautiful snowflakes are not reproducible, generally speaking.  And the same thing happens with your servers.  If you actually handcraft, how many people would handcraft a server and never been able  to get the same application around exactly the same way on another server?  I've never had that fail or I've gone into a room.  And there have definitely been cases in our life.  Our company's like where, well, the thing magically worked and we set it up five years  ago and the customer's like, well, we need to upgrade, move data center, change something  around and we can now not get the same application to run again like it did.  We basically have to figure out a way of lifting and shifting this ancient thing from one place  to another to keep this one specific application running.  We want to eliminate that from your lives and I want to eliminate it from my life because  it really turns into a major headache, right?  Things you can't maintain and you can't keep up to date.  You also can't patch.  We were talking about security earlier, the fact that they released a Windows XP patch  says a lot about security.  Now we love our pets, right?  We give our pets cuddly names because we'll heard this like pets versus cattle analogy  for servers.  So it's actually coined by Randy Byas after hearing a talk from Bill Baker on scaling  SQL servers.  We want to not name your servers with cute names because then they matter to you and  you care about them when they get sick.  If they happen to pass, you'll be very, very sad and there'll be lots of sad users obviously  that these things happen to these very cuddly servers.  We want to actually have our servers be more like cattle.  If part of the herd gets sick, you can push them off to the side and basically spin up  new cattle in their place.  Sorry if anybody's vegan in the room, but it does make for a nice metaphor when talking  about server clusters or our infrastructure.  We're down in this world of DevOps and many non-cloud-native applications actually have  the ability to be deployed with cloud-native tools.  So you can actually take what were these beautifully hand-catchered things that kind  of sat in a single server and actually take advantage of various cloud technologies, AWS,  and AWS building blocks to build up a more cloud-native version of our application, which  actually makes it easier for us to repeat.  So how do we take these beautiful hand-catched bare-knit servers, running our application,  and translate that into the cloud?  That's what I want to go over today with you guys.  So real quick, I hope you guys all seem quite low.  You know it's like 20 years old now?  It's something I've seen before.  No, not at all.  So a little bit of credit to some of the half-news posts.  My slides will have some links to where I kind of got a good start on these things.  So the first rule of Fight Club, I always say you don't talk about Fight Club, I kind  of change these around to be a little more DevOps-based for us.  So the first rule of DevOps Club is you do not log into your servers.  Second rule of DevOps Club, you do not log into your servers.  And why do we do this, right?  Any chance you have to kind of turn that thing into a beautiful snowflake, most system-invents  pie will take advantage of that and do it.  And they shouldn't.  And that temptation to log in and just quickly fix that one little thing really needs to  be dealt with at the top level, maybe like your DevOps or operational management leadership,  is that we don't want to encourage people to log into servers because that does cause  the proliferation of these unique snowflakes.  Our third rule of DevOps Club is that the deployment fails rollback.  So if you cannot accomplish this rule, you need to start putting in place processes that  will allow you then to be able to rollback.  Making sure you've got a virtual control system, some kind of continuous integration in a building  or that you're building, pristine images of your application or you have a way of getting  back to a previous state.  You always need to make sure you have a way back because that allows you to move forward  faster.  If you have a comfort zone or a comfort with being able to move quickly forward, knowing  that you can get back easily, that gives confidence level to your deployments that you don't have  from before.  If you know that it's the only way forward and there's no way back, that's kind of like  burning all the ships, right, when you come to the new world.  That can be really rough if something goes wrong.  But if you know you got a way back, it makes new DevOps and automation a heck of a lot easier  because you feel a lot more comfortable with it.  Fourth rule, all artifacts are stored in source control.  So this helps with the third rule.  Make sure that all artifacts are stored in source control, not only for the rollback,  but also for auditing.  If you want to do code review, we're moving into a more of a world where management of  our infrastructure looks like development.  We can do the same things that developers do.  We can do code review.  We have full source control.  We expect good commit messages.  We want to understand what has happened to our infrastructure over time.  And the only way we can do that is if we have source control in place.  Our fifth rule, only one deployment at a time.  We are still humans and our brains can typically handle kind of one task at a time, even though  we think we can multitask.  We're pretty terrible at it.  It just does as a species.  So if we can maintain one deployment at a time, that keeps us focused on making less  human errors.  Let's all be honest, right?  The computers don't make errors.  The computers just do what we tell them to do.  So if we're making the errors, the computers will help and actually DevOps will help propagate  it even faster.  So errors can move a lot faster in a DevOps world.  Six rule, actually one of my favorite ones.  And we'll talk about specific ones for you to see.  No one-offs and no special cases.  If you start making those special edge cases, the complexity of the whole system to manage  as a whole becomes much more difficult to deal with.  It's just kind of common sense, but I think a lot of people see the easy way forward and  oh, if I just happen to add in this third party package repo from some random site I  found on Google, it'll make it all better right now.  It will not make it all better later when that site becomes bit rotted and no one's  maintaining it or if it becomes compromised and now people are putting code into those  packages that you just randomly are trusting.  So with the case of no-off and special cases, we try to find ways to take that natural path  forward with our infrastructure and use the tools that are available to us in the way  they're intended to be used.  And as soon as we start bending that too much one way or another, we've got to find either  another way to do it or find a way that works better with that natural path.  The seventh rule, the plan was to go on as long as they have to.  So much like the fights will go on as long as they have to, our deployments will go on  as long as they have to so we can get things done and get things moving forward.  So if the deployment is cut short and not in an unknown state, we better be able to  exercise rule number three, which is to be able to roll back.  And the eighth rule and final rule of DevOps Club, if this is your first metadev of DevOps  Club, you have to push the product.  There are a lot of organizations who are taking this mentality, especially on the development  side, that we want to get frequent and often changes to production.  And a lot of times they'll bring in new developers and because they do have a comfort level and  a confidence level in the processes in place and we know we can roll back, they encourage  new developers to make changes on day one of their employment that get pushed abroad.  And I think the same thing should be held true for infrastructure people.  If we're doing DevOps correctly, those should be low risk.  They should be caught quickly either by continuous integration and unit testing and we should  have the ability to roll back.  But I think it gives the new person on the job a confidence level that they can build  and make things happen and even if they break something, there's an easy way back.  Please ask me questions all along the way.  This can be as interactive as you want.  Have you been influenced by Agile?  It doesn't need to be exactly Agile, but I see a lot of...  Yes.  I think we have, so for our company 6VDV, we don't specifically use from the specific  Agile methodologies, but Waterfall doesn't fit this model well at all.  We really have this iterative cycle of keeping things moving forward as not as fast as possible,  but as swiftly as possible.  I think accelerating the velocity and changes into the infrastructure gives our end users  more features faster.  It also means sure that the changes are smaller, so if the change set is very, very small,  it's much easier to roll back.  If you have these giant monolithic change sets to either infrastructure or the software,  that rollback process becomes almost exponentially as complex to be able to roll back out if  you happen to have a failed release.  So a lot of the way these things happen is by keeping changes small and keeping them  in the same frequency and keeping them rolling forward.  What do you think how far is the time when that will also want to be able to change that  old energy?  I think that's now for us.  Our goal is that these are building blocks.  I'll show you.  We've taken these non-cognitive applications.  So it's like a lot of people may run legacy apps where it tended to have a lot of manipulation  of the systems that get these apps running.  I think there is ways, for example, tools like Solstack, Pubg, Ansible, that allow you  to do all that configuration from a, like, I want to configure it locally on my own machine.  I want to be able to test that configuration.  So actually having continuous integration where I can run an automated test process,  and then the release process is fully hands-off.  Like, we want to push button as possible.  And new releases of the software will follow along that same line as well.  And so I think even for a small company, we're like 20 people, and we're able to do this.  I understand maybe the enterprise feels like it's that giant aircraft carrier that turns  really, really slowly.  I think that because these changes can be small and you can start implementing these  kinds of methodologies in a small fashion, you can sneak them in and start doing it today.  And that's my opinion on this.  There's a suggestion that they won't hear the question that you should repeat.  I'll share a few questions.  The question we were just answering was, how long until you think we'll get into a point  where we won't SSA-tion to servers and we'll deploy without any interaction?  I think we're there today.  I think a lot of organizations better be thinking about moving that direction.  I mean, if you guys are coming up to Amazon Web Services Cloud, which probably most of us  are, you know that they have how many three-letter acronyms, TLA's, services available.  Those are all basically virtual machines running on some hardware someplace, but they've configured  them to be quickly deployable as a service.  I think our EC2 instances of our application should be thought of in the same process.  Like it should be nearly point-and-click to deploy our web application into an EC2 instance  or to scale out a process, load balancer, leveraging those other backend pieces that  Amazon gives you for the serialization storage of the data layer.  Our things can be replaceable.  The application code runs basically in almost a stateless container.  Actually, there's another take on this welcoming DevOps Club.  You guys know the movie Back to Back Club.  There's actually a welcoming take on DevOps Club from a blog which is linked into my site  as well.  Those will be up whenever Dan releases a next version of the Viet-Viet site.  He'll be on the public-use site.  We're talking about DevOps Club.  When we're in this world, we need to talk about tools.  For us, we use Python.  Luckily, the Amazon Web Services platform is like Python-first, almost.  They've developed and actually released the Bado Free Library for dealing with the APIs.  The other part we use is Salt Stack.  I mentioned wearing an OG, like old school Salt Stack shirt today in honor of the talk.  That's also written in Python.  It fits well with our Python and Cloud resulting mindset.  I think it's easy to use for a lot of people who even touch Python or have never touched  Python because most of the time, they're editing YAML, which is a very human-friendly, readable  data format that you can type out and plain text to set the states and things.  I'll show some examples of what that looks like as we go along through the slides.  The use case I want to talk about first is taking that single, this was literally a server  at a closet.  We went from the closet to the cloud, we'll discuss for you.  It is the university of Notre Dame had their engineering website.  I think it was literally within some closet, in some building on campus, which is probably  not uncommon for these kinds of groups.  But they had a mandate to move everything into the cloud.  Given that mandate, we thought this was an excellent opportunity to automate the release  process of this so that it could require less technical expertise to actually be able to  release your software.  With this current stack of software, there's obviously the web properties like engine X  and varnish, and HAProxy.  In this case, it's a content managed like this public-fixing website.  If you go to engineering.indy.edu, you'll land on the site, not running here anymore,  but in the new infrastructure.  This is running a content managed like Gulf loan, which has a lot of interesting things  at a single backend storage layer where all the data actually lived.  We wanted to be able to deploy this in a cloud-native way so that if one server went down, we didn't  care.  We could actually spin up in number of servers into a load balancer and have it take care  of the load, or if they, for example, had a dignitary visiting campus, we wanted to build  a handle of additional capacity and have that kind of elasticness in the promise of abandoning  the cloud services.  The promise of abandoning the cloud services is there.  You can do it, but you have to tell it to do it, and it'll just do it for you.  There's nothing automatic about being in the cloud.  It still is, literally, just other people's servers.  Luckily, they give you nice APIs to be able to interact and perform these things in an  automated fashion if you use the right tools.  We want to take this server in a cloud, or server in a closet, up into the cloud.  Again, going to the closet of the cloud.  This is, again, an extra opportunity to take this aging simple server and improve their  performance and resilience.  Give them some high availability aspects that we are going to be in a public cloud and have  access to resources easily.  We don't have to wait a month for a server to be in the prison and deployed into their  closet.  We can actually grab those resources as needed.  How do we make it repeatable?  For example, you could go into the Amazon Web Services console.  You can click around to your heart's content.  This is nearly as evil as logging into your servers over at the stage and configuring things  by hand.  You'll never get it the same way twice.  In our case, we wanted to leverage those cloud-made technologies, for example, Cloud  Front for CDN.  Now, instead of having a single, like a varnish server, we have cash from a single point on  the planet, we can now take advantage of the 145 points of presence that the Cloud Front  CDN gives us.  We want to be able to use a load balancer that can allow us to dynamically add new instances  in and out of that load balancer, so using the elastic load balancer service.  The AD1 and AD2 boxes here are two EC2 instances, one in each availability zone.  So inside of Amazon regions, you have availability zones, which are different data centers separated  by some minimum amount of distance.  But that gives us that high availability in case one data center goes offline for some  reason.  Someone takes a whole, cuts a line, whatever case they need.  Those EC2 instances, those are the only virtual machines on here that you could SSH into.  All the other services here are managed by the Amazon Web Services Cloud.  But those two servers that are on here contain no data we care about.  None of the data for the site or the storage of the files happens in that layer.  This allows us then to make them more like cattle.  They can be taken offline, other servers can come up and replace them inside this infrastructure.  We do put the data into some other Amazon components.  Like EFAS, which is their hosted database service, and then RDS, which is their hosted  database service.  So they give us the ability to store our data into places where we can bring our servers  on and offline, quickly build a brand of image, deploy those images out of EC2s for this,  and have them connected to our existing data storage and they're online in, could be seconds.  And really they can spend a very, very fast.  And then everything is backed up into S3 for the durability factor of having a resilient  backup.  You could point and click your way through this, but it would be tricky, especially if  you start getting into the VPCs and networking and the private subnets and the public subnets  and all the other pieces that are involved here.  So into the automation.  Obviously we need to be able to have a way to point and click and have a repeatable way  for us to put those pieces into place.  So we'll kind of start at the network with it.  That's probably one of the trickiest bits of Amazon to figure out.  They give you a default VPC when you log into your first Amazon account.  It's generally useful to kind of try some things out, kind of prove a concept type  stuff, but it's not really recommended for production.  You're going to want to set up different VPCs for different purposes.  In our case, we want to be able to have this thing be push-button that it actually did.  The automations will deploy out to the full network effect for any environment we want.  So if we have a production environment, we want to build a test environment that matches  it at a moment's notice, or we want to build another test environment inside it so we can  do performance testing and not impact maybe a test around what's happening in a specific  other environment.  So we want them all to match.  There can't be differences here because they're going to run into those strange edge cases,  something will fail in one case and not in another.  But we need a way to control this.  So we set up basically a control VPC where we put our salt master and VPN, so that is  the administration point for all the tools.  I want to go so far as to make sure that even that control VPC is set in an automated fashion.  I don't want to manually configure anything through the console or shell into a box to  get this all actually bootstrapped.  So this is where we start using SaltStack and Vado3.  And we recently use Vado3 as we have to have a way to bootstrap that salt master.  Now I'll talk a little bit more about the master and manual process of SaltStack because  that's actually how it runs, uses a control and then it's a kind of agent on your EC2  services or to talk to the Amazon Cloud.  But we'll use Vado3 to actually get these bootstrapped.  So we do have a bit of a chicken and egg problem.  How do we get the salt master in place so we can build the other VPCs without manually  going in and clicking and creating so that's.  Luckily the Vado3 is very easy to use.  So if you're familiar with Python, it's very, you know, the world of Python called this  Pythonic.  It kind of follows the tenets of the language of Python.  So actually another recommendation for you all, if you guys have a Python prompt on your  computer, is to open up Python and type import this and you'll get a nice Zen of Python,  which kind of may explain why Python needs to think the way they think.  So that's the least we're going to try out.  But in, you know, Python and using the Vado3 library, we can create a new VPC with code.  In our case, to create a cold VPC, we pass in the insider block of IP addresses for that  VPC.  We can add a tag in there for the cold VPC and we can add some DNS attributes for that  VPC.  And that's basically all the code is needed to get started with that controlled VPC.  So we wrote this into a single script about less than 100 lines long that boosts drafts  that VPC for us.  What's also nice about SaltStack is it does have public cloud and private cloud support.  So we can talk to AWS at your digital ocean or if you're in VMware, internal infrastructure,  you can talk to VMware and do similar things.  This is not limited to only AWS work.  You can actually use it in your own infrastructure as well.  And so to do all this, you have to have credentials.  You have to have your APIs, your secret key, and your secret bits of information you need  to use.  And as a service to you all, I want to make sure that you all don't end up as a next headline  on hacker news and place for exposing your API keys.  Make sure you have some mechanism for storing your keys securely.  On my laptop, I have no keys written onto the file system that are not encrypted in  Subnaynor.  And typically, I don't actually keep them on my laptop anyway.  I use it as a tool.  I present, we use LastPass for password management.  We were using LastPass literally this week.  We were using one password.  What's nice about those tools is they do have command line integrations typically.  So LastPass has a LastPass command line tool that has someone who wrote a wrapper script  for it called LpassM that allows me to basically grab the environment variables out of my LastPass  store and put them into my environment just for that session.  As soon as I closed that shell, they were only over in my memory.  They are no longer, they're not written anywhere on my computer.  One password of similar feature is an OP command line tool.  You can get a specific item out of your laptop, your one password vault, grab, for example,  that one uses properties.  So I basically set properties for the TD and secret.  And then I export them directly into my environment whenever I start running these tools and start  working on that project.  If you want a little more sophisticated version of these kinds of things, you may want to  look into tools like Hashtag or Vault where you can start sort of having an infrastructure  for storing your secrets.  But please, please, please don't do what the documentation says.  Where the adress, CLI, where they tell you to put it into like a.file on your home directory.  Yeah, I think it's sure it's easy to get going.  It's also very easy to get hacked.  Anyone who has access to your computer could possibly think of those credentials and start  using your APIs.  And spending a lot of money very quickly if they really knew what they were doing.  Or if they didn't know what they were doing.  In this case, yeah, this environment grabbed in the access key.  And the secret key, the secret access key, and the TID was also nice with the environment  variables with the BOTO3 library.  You can specify other kinds of configurations.  So there are typically other configurations like the region.  You can also specify specific profiles.  There are many other kind of settings you can set via the environment variables.  If you store those all in the vault right next to the keys, importing them is trivial  because I basically have a one-liner that gets my environment activated and ready to  use with whatever project I'm working on.  So from here, we initialize an orc.  I don't expect you to be able to read all that code.  We initialize the initial VPC and we will spin up an EC2 instance into the VPC, our  control VPC for our soltmexter, and in our case, we have a bootstrap sh that we will  read in and put into our user data.  So each EC2 instance, you can pass in a shell script or user in that script to basically  install the initial code onto our digital package on that server.  So we do everything, including pulling out our Git repository of the configurations on  that machine on one script.  So no humans touch the box, it starts up, installs solt, grabs the latest set of states  for our solt integration and ready to go without any human interaction and then write a script.  So we basically start up an instance.  You pass in the user data, so that's our bootstrap sh script that installs private keys, sets  up the repositories, it makes everything ready to go for us to now deploy the production  environment or deploy the testing environment.  One thing to watch out for, a little tip here when you're doing these kind of things, the  cloud infrastructure is not instantaneous.  You may run some of these commands and they may return instantly, but the infrastructure  may not be there for you to use yet.  So in our case, if you want to do something like add a tag to that instance, that may work,  but if you want to actually interact with the instance, it may not be there just because  of the asynchronous nature of the spinning up resources in the cloud.  A specific example of this is if you want to associate a last IP directly to the EC2  instance, you cannot do that until the instance is actually in a running state.  Now EC2's API on Bado does have some processes for doing this, or you can just put in some  sleeves in your code that's kind of the format way to do it.  But if you want to do the more fully asynchronous, there is a waiter method of the Bado 3 code  where you can say, wait until the instance has started and then continue my code.  So it basically does like a polling of the API to see when your actual instance is ready  to go before you move on to the next steps.  Yeah.  Question one.  Let's see, when you keep numbering your team, who is that server right now?  How are you getting these keys as a key or not as a page or whatever identification?  So in this case for this specific problem, we will do these per project and we'll have  a deploy key in the salt method that is shared with the team, or you could deploy in the  SSH copy the keys into the control servers.  So that those control servers are distributing or we are logging the control server to check  for that source?  They would log in the control server.  You would log in the control server to run the salt command to build the other infrastructure.  What we typically have is an SSH key shared along with the team for that project that  would have in there on their own computer to log in.  And let's say you have internet money.  Do you have an internet money process to check internet money and start it is not money for  G-Goran alert?  So that would be separate from our animations.  We will put in place, we use Zadx for a lot of our monitoring.  We are actually going to be moving to use Informetius for that.  So we will hook Informetius to Grafana or Elastic stack for those kinds of monitoring.  You could use CloudWatch, depending on what cloud you are in, using it on the cloud, you  could use CloudWatch for those kinds of notifications.  So that is kind of a lot more, there are a lot more choices there on how you want to handle  that.  So you go in our environment, if you see like I directly found that the SSH and then we  check what is up.  So you can do that.  Basically everything happens from the master.  So you will log in the master.  You can do a service status from the master, like cross like your state of complete of  like 10 internet servers in the front end.  You can see the status of all 10 of them simultaneously because you can use what I will give right  to that in a moment.  Alright, because that is one of the features I like about Sol compared to some of the other  tools, where it gives us the ability to do remote commands and not just configuration  management.  Which is what I think would be a reaction.  So what is SolSTAC?  This is a perfect segue.  I did not plan it, I swear.  So SolSTAC is an event driven orchestration run that hang on.  It is more than just configuration management.  So some of the things that I like about Sol that make it different than say public chef  or Ansible is in fact you do have remote execution.  So I can log into the Sol Mastery and interactively query my whole environment or send commands  to the whole environment or to a subset of my environment based on lots of different criteria.  So you can query and select machines based on values and certain variables or network  sub paths or the laws of the name or if you can set up different kinds of ways.  And you can actually do compound query methods against your environment.  So you can select all the web servers or all the internet nodes and say server status.  And you can do server start for example across that same level of nodes and start all 10  of them simultaneously if you wanted to.  A lot of the tools don't have the level of interaction with your machines.  With the remote execution you can also populate say for example databases of your machine information.  So if you wanted to grab, you know, with the CPU processor, network cards, serial numbers,  you know, all the identification information or any kind of metadata from the machine,  you can do that as well with that remote execution.  And then output it into, you know, my SQL database, Postgres database, LDAP and wherever you want to land that kind of data,  you know, whatever, however your infrastructure is managed, you can do that with Solstack very, very quickly.  The Solstack also has support for, like I said, the various public clouds.  It also has built-in modules for things like curl.  So you can get, you can access the AWS metadata server and actually assign metadata into the  machines based on metadata that's been set at kind of a higher Amazon level, like tags,  or you can get data about coming from the metadata service you wouldn't normally have access to just on a machine itself.  So that's the remote execution.  The next really nice feature about Solstack is the event driven orchestration.  Since you are using typically Solstack in Master and Minion, you have a server with agents running  on all your various other servers in your infrastructure.  You can use that to, you know, not only query in your remote execution, but listen for events coming across the event bus.  So Solstack uses 0 and Q for a message transmission across all the servers.  So it's very, very fast.  You can actually execute things in parallel across a thousand servers and with very low overhead.  Because of this Master Minion architecture, the UDP or UDP is for UDP and TCP.  So 0 and Q, I'm not 100% certain on which one I'm always using, but I think you can use both.  I know you for the firewall chip, like 4506 on UDP and TCP or in the talk.  So the Minions will register in with the Master.  At that point they're on the message bus with the rest of the machines in the architecture.  You can now listen for events coming across.  So there's going to be salt specific events.  For example, the new Minion comes online.  There'll be an event saying it's new Minions available.  If a Minion goes offline, there'll be events likewise from the Master saying I've lost track of a Minion.  And then you can interact with it based on that.  So you can now take an action based on a Minion disappearing.  You can also set a beacon on the machines.  You can watch for activity and send specials, your own custom events across that same bus.  So if you, for example, are watching for someone has a Sage engine server and they shouldn't be,  you can actually interact with it.  So if someone has a Sage engine that can send a custom event across the bus,  you can have what's called a reactor on the salt side to then take action.  You know, either sending a wall message to them or some custom interaction you want to have happen.  You can use it for monitoring.  Some people have used, for example, a graphite across the, this is a transport mechanism for your monitoring.  So you could send from any other kind of data you wanted across that bus back to the Master event  to deal with that as events.  So the event reactor may be to put it into the Master database of monitoring data.  So you can have an agent list, but the common way you run this is with Masters and Minions.  It does support an SSH transport like the Ansible, the FIght is from there, the Ansible.  You can use a similar transport Ansible where it does send it to all the machines.  Obviously you sacrifice or give up performance in that case versus being able to have this agent list operation.  It can support cloud provision.  So there are, the cloud library is built into it for specifically spending on cloud infrastructure.  In our case, we bootstrapped an Amazon EC2 instance using the bottom three library to get us started.  But from here on out, we're going to use Solstack and its native cloud libraries to talk to Amazon and provision infrastructure for our projects.  And then I like the fact that the speed and scalability of using Xero and Q, it gives us that scalability across say thousands of servers.  You can also set up like sub-Masters.  If you had, for example, wanted to have a master in each data center, they would all talk to that centralized master.  They all act like a master, but they talk locally to their own sets of machines.  So if you have, you know, managed your directly with distributed machines, you can actually coordinate between the master and what are called syntax to go to talk to the machines that are closer to physically the fleets of machines you'd want to actually manage and maintain.  So this is similar, like I said, to some of the tools, but I think gives you this additional benefits over what you would get with a Chef puppet.  For example, we can use fabric for orchestration, but it doesn't handle a lot of configuration management.  You can use Terraform for cloud provision, but it doesn't handle the orchestration pieces.  This brings a lot of that into one nice tool.  Now, I will say we are using Terraform more and more for some of the infrastructure and then maintaining Solstack for our configuration management and orchestration pieces,  that more cloud orchestration like a real-time piece.  And Terraform seems to be better about keeping up with all the latest APIs.  Amazon is a bit of a move that kind of sands of the desert type thing that it moves very quickly underneath you.  And so whatever tools you use, you have to make sure they keep up with the APIs underneath as quickly.  So, again, I want to reiterate, it's always best when millions have their master.  You know, millions are very, very sad if they don't have a master.  They can work just not as effectively or efficiently.  So, let's make sure we actually use a master with our millions.  I have a silly question.  Yes.  How long was your discussion about the replacing master's sleep terminology?  Luckily, I didn't have to be a part of that.  That's been there.  I've been using Solstack since version 0.10.  It's been about seven years now, I think, or so.  And it was a rough at the beginning.  I would say that things are very much more stable as far as the APIs of Solstack itself.  One of the things I liked about Solstack early on was they were moving very fast.  And they still move very fast as far as development speed.  But they're not doing the same kind of breaking changes that they were doing early on.  I think they saw one and had a bad idea and they addressed it quickly.  Unfortunately for some of us users, we just kind of had to bear it.  But it was definitely rough, but obviously I didn't run away.  I'm still here, I'm still happy with Solstack.  It's very stable now.  So they have a new numbering scheme and the user tracker is very active.  As far as an open source project goes, I find them to be a very good model of a corporation or company running an open source project.  It is not an open core type project.  You guys are going to be running an open source licensing and have some people license out their tools.  Their Solstack server is the exact same Solstack you use if you have their enterprise product.  The only thing the enterprise product adds on top of that is going to be a UI that kind of has ACLs and automation pieces that are more point-click for the enterprise.  But the underlying Solstack Solstack server you run is exactly the same one as part of what's included with the enterprise version.  You're not running a version behind or beta version or some of that open core software that does this to get you to buy everything.  Solstack is going to do that.  They're going to play really nicely.  The issue tracker people will respond very quickly.  So if you do have issues, I highly recommend going to GitHub, putting in issues.  They take pull requests.  I contributed that pull request for previous D support in the bootstrap script and it got merged in 15 minutes.  I don't think they do it that fast anymore because it was back in the old days when things were moving fast and quick.  But I would say they do watch that very, very closely.  Now, when we orchestrate our servers, for example, if we want to run a deployment to a specific environment and build out those VPCs,  every aspect of our infrastructure is going to be defined by what Solstack calls pillars and states.  So you think of the states as like classes or kind of the skeleton structure of how a service would run.  And pillars are the data which filter and fill in the variables.  So you can reuse those states across multiple projects or multiple clients.  You may refactor it into what's called a formula.  For example, if you always want to deploy Postgres and you have a standard way that if your company always deploys Postgres,  you would use that across multiple projects, but the pillar data would be what's different, like database names, the HBA,  like authentication bits, the passwords, users, that would all be in pillar data.  And pillar data is secured by the gloss.  So for example, if I have only the database servers in production, I have access to the production pillar data.  The staging database servers do not, or the dev database servers do not access that data.  So they cannot, your developers, for example, would not have authority or access to the production database passwords  if you're using the pillar data correctly, which is how we do it.  In our case, we actually want to be able to use that same set of orchestrations across multiple variants.  So in the first instance here, we're running our orchestration to build the VPCs, the private subnets, all the NAT gateways, Internet gateways,  elastic IPs, all the stuff that basically gets pushed into Amazon VPC is done in this deployed environment,  and we pass it in which environment we're going to build out.  So if we want to build five more test environments, we would build pillar data for those other environments,  because you may have different IP, like Cydre blocks, they're really different, so you don't overlap across two VPCs with the same IP blocks.  You can just basically set that up in your pillar data for that environment, run this exact same command and it gets that different data  and then you're in the fresh environment that's exactly matching the previous environment is deployed for.  So no more unique snowflakes, and we actually have a repeatable way of deploying multiple environments.  So this is running with our matching dominions. They're very happy.  So now without humans, we're able to deploy all of this. The control infrastructure is up.  That orchestration we just looked at, deploying these two VPCs with the private and public subnets,  and then inside of each of those subnets, we get the various infrastructure pieces now built out.  So we have them defined inside of our Solstack infrastructure.  What size, ECT instances, what images to deploy, it even goes so far to also provision the EFS, which is the NFS storage,  the RDS, which sets up the database backend, it's the ANCIs database user that sets the passwords.  The code gets deployed via Solstack as well. So as part of this full orchestration,  the Solstack will pull out the code and get however, or whoever, get back and we're using.  It will use the same pillar data. So the pillar data is that canonical source of authentication information  or custom variables for that environment. It will actually set into the code the database passwords for us.  So again, the practice is not to store our database passwords anywhere that is not easily accessible.  So none of our code repositories for the developers contain any passwords for any environments.  And that's all coming from Sol. So when you get deployed in an environment, Sol handles that for us automatically.  They're putting the passwords in the code for the right places for the connection strings for us.  So note that all of this is deployed in one go for one deploy environment command.  That may take a few minutes on the first run, but then subsequent runs, so for example,  if you wanted to add in additional servers, maybe you wanted to expand this load balancer to include two more servers,  you would run the exact same command. It would ensure that those two extra servers are brought online,  put into the load balancer, hooked up with the right password, deployed the right code,  all behind the scenes without having to redo all this.  It would basically ensure that the environment is in a state you could define for it.  So now I talked a little bit about code deployment.  When you go to deploy code, you typically don't want to deploy the whole infrastructure every time.  So this is where Sol comes into play with its orchestration.  So we've used it for configuration management, and we used it to orchestrate building the infrastructure.  We can also now use it to orchestrate releasing code to the servers.  One of our customers, the Central New Mexico Community College, and they run a VMware cloud internally,  and we use SolStack in their infrastructure to do this exact type of task.  And maybe we do this because as we start deploying across more and more servers, things can get tricky, right?  It can get more complicated to deploy.  Developers and the people who are doing the code don't have time to understand a lot of that infrastructure.  So people who deploy and use kind of this point click infrastructure or point click deploy of code,  and we've got a big one for them.  So in our case, we can schedule or make, excuse me, we can schedule or allow the developers to do like zero downtime releases  by orchestrating every piece that released for them.  So in this case, we loop over every app server.  So the orchestrations are just Genja templates of YAML.  So if you understand Genja or use Genja Templating, you've got all the loops and helpers and everything available to you.  So we dynamically determine what the app servers are and then loop over each intern deploying code.  Now what we do is we'll, for example, stop the varnish server which takes it out of the load balancer.  We actually switched this up now to use AHA proxy and then API to top AHA proxy to take the machines out of the load balancer,  perform the release, make sure the instances are back live again before we put them back into the load balancer.  So we can actually do this, you know, basically step through the whole environment, making sure that servers are back online before we put them back in load balancer.  And then for the developer, they just run one command and then kind of wait for the output.  If there's any errors along the way, salt stack will stop, enough receipt to thrash the rest of your environment.  It'll basically stop you at a point where the machine's no longer in load balancer, but something may be broken.  The site's still alive because the other instances that are part of the cluster are still handling the site.  So in this case, you know, we run the control, the instances stop them.  We may roll out a rollout specific code.  In our case, we may have the dev branch, the default code, if you release a production, we can specify that as part of the orchestration.  This also allows them to roll back.  They can pass in a variable for which, like, get tag or get revision they want, and it'll release that specific code out to the servers for them.  They've never had to log into any application server, so doing this all from the salt master.  And then release tasks.  So if the release is more than just put code on a place and reserve a server, you may have to do things like run or gulp, like to do some kind of pre-completion of CSS and JavaScript.  Those kinds of tasks all now happen as part of the salt orchestration.  This is all part of one big orchestration file.  This is the gulp that's here, so they actually put in gulp and then install the new components that came in as part of the theme or the front end of the website.  Those all get handled automatically because humans will typically forget to do this on release and then you end up with a broken site and no one's happy.  And that's not just releases.  We actually use the same technology orchestration to do things like bring production data from production back to the dev environment.  Because it involves typically not only dumping the databases, but they want dev, dropping them, and then grounding, for example, the blog database or blog storage.  The site, the technology we use splits the large data files into a hash directory structure.  Then we can do a little synchronized back to the dev environment in sync with the database at the same time.  And humans are not good at this thing.  Computers are very good at it.  And now the developers who are non-technical about the mainline can basically run one command and get their data on the dev environment refresh from production.  We also use the first scheduling database backups so we can finally have it run during a maintenance window,  switching and knowing which machine is currently the slave.  So we run a Postgres cluster with a master and a slave.  And we only ever want to run our database backups against the slave since it's in read-only mode.  So we'll have you know which one is the slave.  So I should give us the ability to query the machines with their mode execution,  determine which machine is actually running a slave, and then only run the database backups on the slave machine.  So it doesn't matter if they had failed over some time during the day,  it doesn't matter which one's master or slave.  The scripts, the sol orchestration is figured out for us.  Kind of running out of time.  So I'll give you a little bit of a preview as you go on.  But there are some more stuff in my slides about using Cloud Presedient for management of hours,  for example running dev VMs only during business hours,  and have them turn on and off automatically using a tool called Cloud Presedient.  It's also written in Python and also uses a bottom 3 API on the back end.  It's also nice for doing compliance issues.  So if people are writing S3 buckets with keys that are unencrypted,  you can actually self-heal that type of activity because it runs on AWS Lambda,  listening for Cloud Trail events such as like new keys or EC2 instances being spun up,  maybe with old AMIs.  You can then have multi-step work to learn the user group,  spun it up and then shut it down if they don't react with a certain number of days.  And they're getting grabby charts and grabby production data, they can't get over that.  Sound shooting, that was not definitely as easy as they'd make it all sound.  The special cases thing, this is where Pre-VT comes into play.  Instead of Python specifying there should be one and preferably one way to do something,  but it may not be obvious and let you dutch.  This is where the non-obvious part came into play.  We did not want to have special cases for installing certain kinds of packages on the machines  for the dependencies of the application itself.  So for Nipple, WV or HAProxy, we needed specific new versions of those tools.  So like what great, we'll use AMNL Linux because that's like the native Linux for run AWS.  Practical repositories are old, sad.  We will practice in top seven because it will be newer, not new enough.  And for both, for us, PreviousD package system is awesome and actually allowed us to run  with just the standard package repositories and get the versions of the tools we needed to run our play.  It was painful and we spent probably two weeks wrestling with this,  but I really stuck the line of not having special cases.  We did not want to install third party repositories onto these Linux distributions,  only to have them get run or go out of maintenance for us.  Recommendations for you guys, if you do start using this, there are testing tools.  You can integrate this with continuous integration.  So there's one called PictionSol that allows you to test your salt states.  You can actually write states.  It'll spin up in multiple environments across multiple OSs  and actually run your states against each of them to see if they would fail in any way.  So running that continuous integration every time you make a change,  that gets you to that confidence level to make sure that rollbacks are possible.  And then I've got some examples using Bottle3 in my GitHub,  or our company GitHub, we're doing pretty wrong log tools using Bottle3  for tracing through logs for requests.  Are we living the dream?  I would say absolutely living the dream because I got to use PreviousD in the cloud natively  and saved my life because I was able to use BSD.  And I'm very happy to have you.  I do want to thank you all for sticking with me.  I won't even play eight minutes over, so sorry.  Forgive me, but if you have any questions.  I'm not sure if they're actually early.  Am I?  I'm 17 or I'm at 12.15.  It does.  Okay, well then I got time for quite a couple of questions because I love you.  How did you guys come off that stupid start?  I think the whole thing started kind of looking back at those rules of like DevOps Hub.  How can I make sure that we don't log in servers,  that things are repeatable, we can roll back,  and that I can get a consistent environment every time I do a release?  Obviously, we have our own people.  So we, I mean, people who are working on log-in deployments from an infrastructure standpoint,  it's really only about two people, one or two people who are handling the infrastructure pieces for these various projects.  So I like the fact that it doesn't require a lot of overhead and there's not a lot of maintenance to it.  I mean, to kind of say forget it, because if something goes wrong,  we can just tear down a server and replace it in a moment's notice with a new server running on the same code.  Typically nothing goes wrong because no one's logging in the servers to make up in a unique snowflakes.  You know, knock on wood, right?  But I think it's just thinking about how to make things repeatable.  We run into this problem a lot as developers.  How do I get a new developer to just join the team to spawn up quickly to be productive?  And so it's a lot about making things repeatable even at the desktop level.  Like we'll use soft stacker vagrant to even get like VMs locally.  A lot of our infrastructure pieces here, we have local vagrant files that will spin up with mini cluster on your laptop  so I can test things out before I push into the dev environment or production environment.  How do you deal when you have to update the orchestrating part?  When you have to do big change to the infrastructure at the higher level?  How do you deal to it?  Because it seems to me if you want to change maybe the connection, how it's wired up.  Oh, the infrastructure part?  Because I see how it's easy to orchestrate the deployment.  But for the infrastructure, maybe you have like snowflakes or infrastructure because you have the test.  So that is probably the reason why we're switching to Terraform for a lot of the infrastructure pieces.  If you're in a framework with Terraform, you produce a model of how you want the end result to look like  and then calculate the differences from where you get that now.  And so with Solstack, you have to more declaratively say I want to change this infrastructure from that infrastructure.  Terraform will figure that part out for you and can make that change.  Obviously you want to improve the apply before you look at what the difference is going to be  because then you may destroy things along the way inadvertently.  But Terraform is probably more powerful when it comes to doing that than that.  And how does it integrate? Are there really something that you're switching completely?  No, not necessarily. Solstack has a Terraform module built into it.  So we can actually execute Terraform models against your environment so that you can actually do that integration.  Another piece we're probably using, the same company who does Terraform, who does background,  probably sensing a theme here that I like to have to show folks, they also make a tool called Packer for creating images.  And then Packer supports using Solstack to build the images.  So our same Solstack to build these infrastructure kind of live can be the same Solstack to produce images that we can deploy instantly.  As you probably know, Solstack is a developer that we know that we have an exciting situation with my iLotify.  With which one?  With iLotify because of deep insight, etc.  So are there any plans in Solstack that we need to install something that we can just support?  Thank you.  Probably?  That I don't know.  Okay, I don't have a good answer on that.  Okay, okay.  Does the Solstack master have a...  You said you can get it to a stable pass.  Yes.  Are you just running Solstack as a prong job?  No, so Solstack has its own internal scheduler.  So for example, with the database backups to having it run at certain times, you can schedule kind of like app jobs,  like one off, do this thing at a certain time, or you can say schedule on a certain schedule,  either periodically at a specific time or every September, you know, hours, minutes, run at a certain internal schedule.  So I like doing that over prong jobs because now everything is kind of centralized around Solstack.  I'm not having Solstack install prong jobs on the road,  but I'll have Solstack actually automate those tasks and run them on the various minions for me.  Because at that point I get feedback from a centralized spot.  But I actually wanted to take the output of every one of those prong jobs or tasks and dump them into Elastic.  I can now quicker across them or get notifications or get graphs of the frequency of the runs and also frequency of errors.  So instead of relying on, you know, cron emailing in my inbox, then saying that I went wrong,  I can actually have a fully automated feedback loop that monitors the tasks that you're running.  And I'll be back.  So the feedback that you get from this long version is in the sun, do you use this work to Elastic search?  You can.  So there's play support inside of Solstack for many different outputters,  like where you want to send the output of the commands, like whether it goes into database.  Natively you just get like a JSON, like a dump or a plain type of output.  You can get it a JSON or a CSV or if you have an application you want to stuff it into like Elastic,  or if you want to put it on Kibana, because you think, I don't know Kibana, but I don't know Kafka.  Like if you've got a messaging service, you know, Enterprise-Wide,  and other things you're listening for, that type of information and consuming it,  you can actually have it done on your com.  Cool.  Thank you all.  Thank you.  You're welcome.  Thank you.  You're welcome.  Thank you.  Thank you.
Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_0_107.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_108_203.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_204_299.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_300_2903.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_2904_3851.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_3852_4367.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_4368_4883.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_4884_6467.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_6468_7535.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_7536_8927.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_8928_10799.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_10800_10955.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_10956_11003.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_11004_11423.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_11424_11843.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_11844_12263.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_12264_12467.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_12468_12671.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_12672_13091.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_13092_13295.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_13296_13715.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_13716_14135.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_14136_14339.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_14340_14759.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_14760_15383.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_15384_15587.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_15588_15755.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_15756_16427.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_16428_16631.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_16632_16847.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_16848_17675.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_17676_17879.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_17880_18095.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_18096_18299.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_18300_18347.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_18348_18503.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_18504_18719.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_18720_19343.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_19344_19547.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_19548_19763.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_19764_19847.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_19848_20171.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_20172_20975.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_20976_21023.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_21024_21215.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_21216_21419.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_21420_21635.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_21636_22463.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_22464_22859.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_22860_24131.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_24132_24911.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_24912_26819.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_26820_27347.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_27348_30575.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_30576_31031.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_31032_31307.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_31308_33395.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_33396_33959.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_33960_36191.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_36192_36263.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_36264_37367.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_37368_40235.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_40236_46835.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_46836_47171.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_47172_56627.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_56628_56987.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_56988_57107.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_57108_63851.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_63852_64019.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_64020_67055.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_67056_68291.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_68292_68483.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_68484_69827.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_69828_70451.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_70452_71063.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_71064_71567.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_71568_71951.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_71952_73967.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_73968_75155.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_75156_75227.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_75228_75383.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_75384_75647.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_75648_77063.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_77064_78035.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_78036_78347.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_78348_83135.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_83136_87695.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_87696_87743.png Master___Minions_or_the_Dream_of_BSD_Automation_Using_Salt_and_Python_to_manage_the_a_BSD_87744_87784.png