It's Time for Us to Move: The Story and Migrating Hosted Chef to AWS
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Harrison, Mark
Hosted Chef is one of the biggest Chef installations there is, with tens of thousands of organizations managing hundreds of thousands of Chef clients. By 2015, Hosted Chef had been growing exponentially for several years, and it was quickly outgrowing its home. It was time for a change, and so last October we migrated Hosted Chef from its original data center into AWS. As if the migration of a large production service wasn't enough, we were using an aging code base with practices and procedures that were years old, with references to CouchDB and workarounds from Chef 0.9! It was time to modernize all of our cookbooks, start using modern features, and generally rewrite everything at the same time. This talk is the story of that migration, the decisions we made, the challenges we faced, and the spectacular results. I'll cover what worked and what didn't go so well, and along the way I'll share some critical insights that will be useful to anyone running a large Chef installation in a cloud environment such as AWS.
 First up is going to be Mark Harrison. He's a senior sysadmin at Chef. He is the sysadmin in a company full of sysadmins. And he's going to talk about migrating hosted Chef to AWS. Thank you. Can everyone hear me okay? Yep. Okay. Quick show of hands. How many people here use hosted Chef? Okay. Keep your hand up and keep it up if you knew we migrated last year before this talk. If you knew we migrated hosted Chef to AWS last year. Okay. So I'm just curious how many people actually knew. Okay. For those who don't know, hosted Chef is Chef software service product for Chef server. And it's pretty big. It's got about 100,000 organizations. It's got about 400,000 nodes. And unlike most web services or websites, it's got a very, very spiky load profile. And one thing about this is, here's a normal website. So normal website, over the course of a day, you've got like the lull in the morning and then your customers wake up. It starts to grow slowly. And then you hit the peak of the day, which some people it's like midday for Netflix. It's probably 8pm. And then people go to bed and it starts to go back down again. This is the hosted Chef, request second graph. And this is over 12 hours. Now, you can see there, but there's spikes at the top of every hour. And then smaller spikes 30 minutes past the hour. Anyone could guess why that might be? Anyone not from Chef? Go ahead. Precisely. People are running Chef Client via Chrome. If you run Chef Client as a demon, then it will actually pick a random time. But if you run it via Chrome, it can start immediately. And sometimes there's a little bit of display and you can configure that. But if you don't configure that, then it just runs immediately at the top of the hour. What that means is that we have to cope with this huge increase in load at the top of every hour. And at the bottom. Load at 15 minutes and 45. So yesterday, you all went to the keynote and you may have noticed Seth mentioned hosted in 2010. And he mentioned this command 9vc to server create. And yet I'm talking here about moving Chef to AWS and not from AWS. So just want to explain a little bit there. In 2010, hosted Chef actually used to be in AWS. And then round about 2011, we actually moved to Rackspace by metal because of performance issues. So it's actually somewhat ironic that performance issues are what led us to migrate back to AWS last year. So why did we actually move? Well, we were outgrowing our current infrastructure. We were hitting scaling issues. But the reasons for moving to AWS are the reasons that all of you want to move to the cloud. Cost. Flexibility. Basically the ability to get what you want when you want it for as long as you want it. And only pay for what you need. And the ability to scale up as needed. So no lead time. We don't have to wait two weeks for a new service to arrive. Well, there's one other reason. Developer time. So the developers were spending lots and lots of time supposing hosted. And the reason for this is hosted looked nothing like an on-prem Chef install that you use. So also, we were using the same configuration, like Chef cookbook, Chef recipes that we've been using for hosted since the time it was in AWS previously. So we still had configuration from Chef 0.9 and old ways of doing things. And it was time for us to modernize things. So here is a normal on-prem Chef install. This is what any of you will install if you want to scale your Chef infrastructure or Chef server. You've got load balancer at the beginning. Then you've got some front-ends, one or more, more if you need to scale up. And then you've got a couple of back-ends in a HAPer. This is the architecture diagram for old hosted. Now, you might not be able to see it very well there, but we've got two different load balancers, engine X, we've got three old long services, every different microservice that hosted Chef runs, all the Chef server runs, that had its own machine or its own set of machines. So that's another big reason that we wanted to move. So the devs were basically spending way too much time supporting that big architecture. They were doing deployments, helping with troubleshooting. And we wanted to be able to make hosted Chef look a lot more like the Chef service that all of you install. And if we could do that, then we would get lots and lots of benefits. And we could run the hosted Chef infrastructure without having to rely on devs providing custom code parts because they're providing the code for everybody else. And this is how we did. This is the architecture diagram for new hosted. And it's got a few services split out, but overall, it's pretty close. It's a lot better than that. So we've decided we wanted to move. We decided we're going to Edge of the West. And we decided roughly how we wanted to look. Now we have to actually do the migration. So we split the migration up into about three different tasks or projects. Getting things running in Edge of the West, actually migrating the data, and then monitoring everything once we're done. Now, monitoring was two weeks ago. So I'm going to focus on the first two. So first task was to set things up in Edge of the West. So you run deep package, install Chef server code, Chef server CTL reconfigure, Chef server CTL start, and you're done. Thank you, everybody. It's not actually that easy. Otherwise, I won't be here talking to you today. So one of the first things that we had to do was split out some of the backend services. So I mentioned earlier that you had these just two backend services in a HAPer. The problem with that is you have your stateful services. So you've got all the s that that's those all of your Chef data. And you've also got search. Problem with that is to scale up. You basically need to buy a bigger instance. So if anyone's seen those like new X1 giant Godzilla instances, we didn't want to have to get one of them because the cost is like $1,000 an hour, something ridiculous like that. So we split out our data and search into separate services. So for the Chef database, we decided to use all the s. And thankfully, it's got the Amazon's got the magic service. You push a button and you get this database hand delivered by magical Amazon elves. So this best self this best problem self, right? Actually, all the s has worked out really, really well for us so far. For the Chef database, at least, all the s has been working fairly swimmingly for the reporting database, which I'll get to later. That's another story. Next component in the backends search. And that has three main components. The solar service, RabbitMQ, which solar holds all your search information when you run knife search. RabbitMQ takes all the information when you make a change in Chef, like you save your node, it posts it onto the queue. And then there's a service called ChefExpander, which takes the changes off the queue and then posts the changes to solar. And if you're actually using a brand new version of Chef server, using the new Chef head chair that's been released in the past two weeks, all of this is actually just elastic search. What we decided to do was just keep all these services together on the back end machine. And there's one other component to that diagram that's part of the back end that I've not mentioned yet, and that is Redis. Redis is used by the front end machines basically for things like feature flags, web sessions, and a way for us to enable maintenance mode across all of the servers. You've got NGX, the load balancer at the front, listening to Redis or checking Redis for information for things like how to set maintenance mode. For that, we just used elastic cache. It's basically hosted Redis as a service, and we just have to use that. That brings us to the front ends. So on old hosted, and this is part of the reason for that big diagram before, you had all of these services, a Chef, by first, or CID on its own server. Most of these services are pretty stateless. So we were able to combine these and in a normal Chef server install, all of these are combined onto the same machine. All the services are just on front ends, and you can horizontally scale them. So there's just a few other parts of the infrastructure left. First off, cookbook storage. When you upload your cookbooks, the actual physical data, where you have your recipes and attributes and all the actual cookbook data, that gets uploaded to S3. That made the migration process really easy, because in old hosted, it was in S3. In new hosted, it was in S3. We didn't have to do anything. Other components, load balancer in front of everything. We just used Amazon's ELB. That was a fairly easy decision. It just goes in front of the front ends. Load balancer is everything. And then there's one other machine we have in hosted that you may or may not have in a non-prem install. That is supportbox. And basically that is a machine we can log into. It configures us front end, but it's not in the load balancer. And we use that for troubleshooting, password resets, any issues you might come across with, or because we might need some manual fixes. We just use that as a support, do all that on the supportbox. So we've got the various components decided. Now we have to actually get things running. So the first thing we do is we put everything into auto scaling groups. Now, when you hear auto scaling groups, you think about machines magically growing, like you see the load growing and it spins up new machines. Problem is, our spiky load, that was for, it went jumped up at the top of every hour and then went right back down again. And it takes about 10 minutes to spin up a new node. So we don't actually use auto scaling groups to auto scale our infrastructure. We need to be at full capacity anyway. What we use auto scaling groups for is we set a static threshold and AWS makes sure that there's always 16 nodes or 14 nodes. Whatever we set it to. And that means it just reinforces the idea that we want to treat our servers as cattle and not pets. If we have a machine that's misbehaving, we can just kill it and AWS will just bring up a new machine to replace it and that is really, really nice when you get paged. So if we're using auto scaling groups, if you're using auto scaling groups, you need to build images and to do that, we use Packer. We build images every time we deploy a change, which isn't that often. Like new Chef server versions, something like that, or company changes. And when we do that, we destroy all of our old front ends or our old instances and we recreate new ones. So most of the setup that we do is actually done in Packer during the image build. There's about three types of things that we do afterwards. That is anything that's specific to the environment, so anything that needs to be different between acceptance and delivered. Or pre-prod and prod. Anything that's specific to the instance itself, things like host names or node names, specific Chef configuration and credentials, which they're actually kind of a specific case of environment and specific config, but we treat them separately because they kind of need special treatment there. All additional configuration after boot, it's done with Chef. So that configuration, when machine starts up, we have a user data script that sets up Chef and runs it on boot. So with AWS, you can set a shell script that it just runs on boot when the instance comes up. Now, we can't actually use the validatorless boot shot method. That's the one where you don't need a validator key. Because we need the machines to register themselves as part of the auto scaling group. So we just drop a validation key in Etsy Chef, tell Chef to run, the machine registers itself, and it goes on to configure things. We also use policy files to manage configuration. And that's been really useful for a number of reasons and you can read about the benefits of policy files. One big thing that we do is we make use of something called named run lists. And this is a separate run list for a node that you can run by specifying it by name. So what we do there in Packer, we say use the build run list and it just does the build time config. And then we've got our normal run list, which does all of the configuration with Chef, including the environment specific, the credentials, and the other things I mentioned earlier. So for the actual AWS resources, instances, RDS, Elastic Cache, we use Terraform. That manages all of our AWS resources and it makes provisioning really easy. It's your infrastructure or rather the instances and the actual AWS resources as code. And one of the ways that this is really, really helpful, about four days before we migrated, or rather four days before we were scheduled to migrate, we were happily set up in US West 2. One big problem with that, most of our customers were located in US East 1. So we were in US West 2 because of reasons, but someone made the decision to actually move us to US East 1 to be close to where all the customers are. And we were able to manage that migration in roughly one day. And that migration, that day, was mostly transferring data that we'd already got in place. And going through the Terraform config and the Chef config, just removing any assumptions we had about regions. So I mentioned earlier, when we do auto scaling groups, we build an AMI and then we destroy our instances and we create them. Problem is, if you do that, you destroy all your instances, host the Chef goes down. So what we actually do is we rebuild our images in Packer. We bring up the new instances in a separate auto scaling group that is not live. And we run a few tests and then once we get everything working, we verify it's all there, we swap out the auto scaling groups. And what that lets us do, it lets us test beforehand and it also lets us quickly roll back if things go wrong. So we only do this for the front ends. And one of the reasons for that is auto scaling groups and blue-green deploys, they work really, really well for stateless machines. You can just kill an instance and it will come back and no one really cares. For stateless services such as our back end, that's running your search, that has a lot of data and we need to be much more careful with that. We can't just kill an instance because that might be the instance where all the data is stored. So we just need to make sure that when we do our deploys and we need to change things, we make sure we kill the secretary back end first and then we failover and then we can do the old primary. If you have single machines, you can still use auto scaling groups. Now if you've got a single machine, chances are you don't really care about it going down because you really have AWS. In AWS, instances randomly get destroyed. And if you think that doesn't happen, you've got nothing coming. It does. But our support box, I mentioned before, that's just a machine we have to search into to do some troubleshooting or to reset passwords. If that goes down for five minutes, we don't care. But if it goes down for five minutes, we still want AWS to bring up a replacement for us. And that's happened numerous times in support boxes, which just disappeared. And I've logged in and I'm like, oh, this is a new box. And I'd never even known because we don't need to monitor it. We don't need to know because we don't really care that the support boxes are there until we actually decide we want to log in. So you can still use auto scaling groups for single machines. So we're now up to the migration process itself. We know how we want to get everything up and running AWS. Now we need to migrate. So like the project as a whole, we split up the migration into a few different tasks. Moving the database, specifically the Chef server database and the reporting database. Moving the search data, all the backend data, solo content. And finally, flipping the switch, flipping DNS, moving people over to the new infrastructure. First thing on that list is migrating the Chef server. So the Chef database server. The Chef database server is where all of your data stores were like nodes, eight-scots, roles, your clients on the code metadata, basically everything except for the code books themselves. And in all hosted, this was actually three databases. We had one for the Chef server itself and two others for a service called Bifrost, which did authentication. And a service for OCID, which also did authentication for a different service. In new host days, we decided we should merge all those databases into one, which is what an on-prem install looks like. So we chose ODS earlier to host the database. So one big problem with that, you can't do streaming replication from Postgres into ODS. I think you might be able to now because AWS has got this like database migration service. But at the time, we couldn't do that. We had to just copy all the data wholesale from our old database into ODS. Thankfully, databases are both small. All the data for everyone in hosted Chef is actually less than 100 gig. So we were able to just copy everything across and do a copy of the cross during migration. Next on that list was the reporting database. Now, reporting is the Chef server add-on, Chef server reporting add-on. And that keeps records of every Chef run you do against hosted, which is a lot of data. At the time of the migration, that was about two to three terabytes, which I know some databases can be bigger, but that was something that we certainly couldn't just copy across in five minutes. And ODS only supports up to about six terabytes. So allowing for some growth, we couldn't be in ODS. So for our reporting database, we decided to use just normally C2 instances running Postgres, which meant we had to manage those databases. But it also meant we could use streaming replication. And that meant joining migration for reporting, at least. We could just replicate all the data, and then when the migration came, we just fell over the database, and migration for that is pretty much instant. But before we could do the replication, you need to take a base backup. Now, old hosted data center, Burmese, we had gigabit connection. So you do your math, and comes out about one terabyte every three hours at gigabit speeds. Now, you never really get your full bandwidth for various reasons. So we're doing two to three years. Let's allow full day. Should be plenty of time. Three years later, we made a few mistakes by our calculations. First thing, we're transferring across the US. So you've got to deal with latency. That will slow down your transfer rates. Latency wasn't the biggest issue we had, though, because we're going from Chicago, so it's probably not all the way across the US, middle of the country to the east coast. But we also assumed that we had full gigabit. Normal hosted traffic, because, well, hosted, we still had to serve everybody. That ate about half of our bandwidth allocation. So we're already down to half. On top of that, reporting, that stores every chef run. And people querying every chef run they've ever done. Or at least querying every chef run for the past month. That's a lot of data. And the reporting server disks were constantly saturated. So that means there wasn't basically nothing left for us to transfer the data across. So the actual transfer rates we got were more like 10 megabytes a second, or roughly 18 megabits. And on top of that, our transfer, we were using all sync to begin with, that just died regularly. Like we transferred a couple of hours, it would just stop crashing out, and then we'd have to start again. So we used a few tricks to speed things up. We tried parallel or sync, some of these fancy tools you can get that claim to transfer your data across the country 10 times faster than also capable of to deal with the latency issue. But all of these things, it still took us about a week with those tricks to transfer everything. And by that time, Postgres has these things called wall files, which store the replication data while you're taking the base back up. But it only stores so much. In the time it took us to transfer everything, the wall files had rotated, and we couldn't start replication. So spent a week transferring everything, realized we can't start replication, repeat the process all over again. So about two weeks later, we finally get reporting transferred and replication started. Reporting done. Next service, solar. So as I mentioned before, solar holds all the search data. This is what gets queried when you run a search or in your recipes if you run the search function against nodes or against data bags, things like that. This, the actual data stored was much, much smaller than either reporting or the Chef database. And on top of that, solar has this built-in replication. It's not great replication. It's not streaming replication. It basically transfers the entire dataset once every hour. But it does a fair job, and the dataset we had for this was probably on the old, or about, say, 10 gigabytes rather than 100. So that meant we could choose the solar replication and things would just go back. And in case anyone's wondering, so Chef server, it has a command called reindex, where it will take all the data that you have in the Chef database, and it will populate solar from the data in the Chef server database. The reason we didn't use that is it just takes too long to do. For this dataset size of hosted, we're probably talking at least a day to reindex everything from scratch. So it's much easier to just transfer all the data as is across from one to the other. Next backend component is Redis, or Elastic Cash. That was even simpler. So if you recall from earlier, Redis basically stored feature flags, web sessions, whether we were in maintenance mode, and all of those things, the data is pretty static, and the data we had in old hosted, that was feature flag settings for some ancient feature we added years ago. So what we were able to do was just start the new Elastic Cash, or the new Redis database, with a completely fresh blank database with just the settings that we needed that we could manually put all the settings in for new hosted. That brings us to migration day itself. And we decided to migrate at night time. There's really a good downtime for hosted Chef. I think you might have noticed before, over that 12-hour period, the load was fairly flat with the spikes. But we decided to do it when at least most customers would not be looking at hosted Chef. But we start early in the morning. First task, before anything else, lower your DNS TTLs. So if anyone doesn't know, TTL in DNS is basically how long after you do DNS lookup, your laptop or your server, whoever's doing the lookup, will keep the name information without looking up again. Which means if the person goes and changes it, such as during a migration, you'll still have all the old information. So we loaded it to 60 seconds, which meant that when we flip the switch, most, if not all of our customers, would change over within the 60 seconds. Next we do test run. That's just test migration, and then we make a few final config changes. And then night time came. And we do our migration process. First thing, update status site. All of you know we're going to migrate. Then we enable maintenance mode. And then we do, we actually migrate the databases. This is just a procedure that we mentioned before, flipping switches where we got replication on, initiating data transfers where we don't have replication, and then using the sole replication. Then we test. We do a quick test with an organization. We took out maintenance mode, make sure everything worked. Then we do a quick load test, which we've tested earlier. We want to do a final load test, make sure everything didn't fall over. Then we flip the DNS, and then we update the status site. And we're migrated. So I mentioned load testing there. We did a homegrown tool called Chef Swarm that just simulates lots and lots of Chef runs at once. And we used it to simulate peak load, like top of the hour load, and new hosted. And we test against a single organization, like just test org that we had that was out of maintenance mode so we could still do it while everyone else was still in the maintenance mode. There's a few problems with that. First, be very, very careful about running your load test against active prod. I also mentioned that we tested against a single org. So we were like, we know this load test is not completely representative of an actual genuine load. So we expected that our load test would tell us everything's fine. And then we flip the switch, and the fear that I had was we flip the switch, real traffic come along, and the whole thing would fall over. Turns out the opposite happened. Join the migration. Now, we did the load test earlier in the day. We knew, according to the load test, we were good. We do our final load test, join the migration, just to make sure everything's okay, and everything falls over. Hosted just crashes and burns, and we're like, what the heck is going on? So we double check all of our procedures, like everything's okay. We're not sure what the heck is going on, but we think there's something wrong with the load test. So we knew we were good to go, flip the switch, and once we flip the switch, life traffic came along, and Hosted took the life traffic without any problem at all. Turns out that when we did that load test during the actual maintenance, we still had life traffic hitting the front ends, and then getting a maintenance mode response back at the same time we were doing the load test. What that meant was that the NGINX parts on the front ends, they were basically taking all of the current production traffic plus retries, and the load test at once, and that was why it fell over. So lesson learned there is be very, very careful with your load testing. It doesn't always tell you the truth. So we migrated. Load testing's complete. We flipped DNS, and everything's great. And these are the results. It's kind of hard to see there, but the top graph is right before the migration, and that is the top of the owl peak, which at 90% retires roughly 500 milliseconds for an average request, or for 90% our request, rather. Right after the migration, that went down to 50 milliseconds for the 90% is a factor of 10. And just to remind you, that's old Hosted. Big mess. Lots of services. Hard to understand. I'm not convinced I fully understood this architecture before the migration. Probably understood most of it, but not convinced myself. And that's the architecture afterwards. Much cleaner architecture, much easier to understand architecture. We've got much improved deployment method. We've got better ability to scale. We can just add new instances as we need them. And we're running the same Chef server packages as everyone here who is running Chef server. Any questions? Okay. So the question was, are we exposing Hosted Chef to hosts outside of AWS? And the answer is yes. We do have a lot of customers who are located in AWS. And sometimes before the migration we checked where most of our customers were coming from. But yes, Hosted Chef is available for anyone who wants to use Chef. And chances are, if you are using Chef to begin with, like learn Chef or Chef tutorial, you may have touched Hosted Chef just to get open-bunny real quickly. Someone else has a question? We point the front ends at a separate Chef server. So originally we just had a separate Chef server. Now as of about March this year, we're using Chef delivery. So that's a whole other talk. Okay. So the other question is, are you guys going to move to the Chef AJ product that you guys announced this week? Quite possibly yes. However, when we do that, I don't know. So we actually scheduled the maintenance window of about four hours. That was just to allow time to transfer everything over, because we need to transfer that over the whole Chef database. And also to allow time to troubleshoot any issues that came up. We probably got through the migration in about, probably a little under half of that time. So let's say one and a half hours. Any more questions? Yes? What's the total node count of the resource? The total node count is about 400,000 managed nodes. About how many organizations? About 100,000. Yep? How many of you keep the list perfectly? What's the staff? Well staffed. We have a team of about four or five people right now. They, so we used to be 100% host chef. This is actually, this is another big reason for the migration. In addition to the devs working on hosted chef, we were spending all of our time dealing with issues with host chef. We had these micro outages constantly and just dealing with these issues. Since we migrated, we've actually been able to be not 100% focused on hosted chef anymore. So we've moved on to other services. Any other questions? Okay. Well thank you very much everyone.
 First up is going to be Mark Harrison. He's a senior sysadmin at Chef. He is the sysadmin  in a company full of sysadmins. And he's going to talk about migrating hosted Chef to AWS.  Thank you. Can everyone hear me okay? Yep. Okay. Quick show of hands. How many people  here use hosted Chef? Okay. Keep your hand up and keep it up if you knew we migrated  last year before this talk. If you knew we migrated hosted Chef to AWS last year. Okay.  So I'm just curious how many people actually knew. Okay. For those who don't know, hosted  Chef is Chef software service product for Chef server. And it's pretty big. It's got  about 100,000 organizations. It's got about 400,000 nodes. And unlike most web services  or websites, it's got a very, very spiky load profile. And one thing about this is, here's  a normal website. So normal website, over the course of a day, you've got like the lull  in the morning and then your customers wake up. It starts to grow slowly. And then you  hit the peak of the day, which some people it's like midday for Netflix. It's probably  8pm. And then people go to bed and it starts to go back down again. This is the hosted  Chef, request second graph. And this is over 12 hours. Now, you can see there, but there's  spikes at the top of every hour. And then smaller spikes 30 minutes past the hour. Anyone  could guess why that might be? Anyone not from Chef? Go ahead. Precisely. People are  running Chef Client via Chrome. If you run Chef Client as a demon, then it will actually  pick a random time. But if you run it via Chrome, it can start immediately. And sometimes  there's a little bit of display and you can configure that. But if you don't configure  that, then it just runs immediately at the top of the hour. What that means is that we  have to cope with this huge increase in load at the top of every hour. And at the bottom.  Load at 15 minutes and 45. So yesterday, you all went to the keynote and you may have noticed  Seth mentioned hosted in 2010. And he mentioned this command 9vc to server create. And yet  I'm talking here about moving Chef to AWS and not from AWS. So just want to explain  a little bit there. In 2010, hosted Chef actually used to be in AWS. And then round about 2011,  we actually moved to Rackspace by metal because of performance issues. So it's actually somewhat  ironic that performance issues are what led us to migrate back to AWS last year. So why  did we actually move? Well, we were outgrowing our current infrastructure. We were hitting  scaling issues. But the reasons for moving to AWS are the reasons that all of you want  to move to the cloud. Cost. Flexibility. Basically the ability to get what you want when you  want it for as long as you want it. And only pay for what you need. And the ability to  scale up as needed. So no lead time. We don't have to wait two weeks for a new service to  arrive. Well, there's one other reason. Developer time. So the developers were spending lots  and lots of time supposing hosted. And the reason for this is hosted looked nothing like  an on-prem Chef install that you use. So also, we were using the same configuration, like  Chef cookbook, Chef recipes that we've been using for hosted since the time it was in  AWS previously. So we still had configuration from Chef 0.9 and old ways of doing things.  And it was time for us to modernize things. So here is a normal on-prem Chef install.  This is what any of you will install if you want to scale your Chef infrastructure or  Chef server. You've got load balancer at the beginning. Then you've got some front-ends,  one or more, more if you need to scale up. And then you've got a couple of back-ends  in a HAPer. This is the architecture diagram for old hosted. Now, you might not be able  to see it very well there, but we've got two different load balancers, engine X, we've got  three old long services, every different microservice that hosted Chef runs, all the Chef server  runs, that had its own machine or its own set of machines. So that's another big reason  that we wanted to move. So the devs were basically spending way too much time supporting that  big architecture. They were doing deployments, helping with troubleshooting. And we wanted  to be able to make hosted Chef look a lot more like the Chef service that all of you  install. And if we could do that, then we would get lots and lots of benefits. And we  could run the hosted Chef infrastructure without having to rely on devs providing custom code  parts because they're providing the code for everybody else. And this is how we did. This  is the architecture diagram for new hosted. And it's got a few services split out, but  overall, it's pretty close. It's a lot better than that. So we've decided we wanted to move.  We decided we're going to Edge of the West. And we decided roughly how we wanted to look.  Now we have to actually do the migration. So we split the migration up into about three  different tasks or projects. Getting things running in Edge of the West, actually migrating  the data, and then monitoring everything once we're done. Now, monitoring was two weeks  ago. So I'm going to focus on the first two. So first task was to set things up in Edge  of the West. So you run deep package, install Chef server code, Chef server CTL reconfigure,  Chef server CTL start, and you're done. Thank you, everybody. It's not actually that easy.  Otherwise, I won't be here talking to you today. So one of the first things that we  had to do was split out some of the backend services. So I mentioned earlier that you  had these just two backend services in a HAPer. The problem with that is you have your stateful  services. So you've got all the s that that's those all of your Chef data. And you've also  got search. Problem with that is to scale up. You basically need to buy a bigger instance.  So if anyone's seen those like new X1 giant Godzilla instances, we didn't want to have  to get one of them because the cost is like $1,000 an hour, something ridiculous like  that. So we split out our data and search into separate services. So for the Chef database,  we decided to use all the s. And thankfully, it's got the Amazon's got the magic service.  You push a button and you get this database hand delivered by magical Amazon elves. So  this best self this best problem self, right? Actually, all the s has worked out really,  really well for us so far. For the Chef database, at least, all the s has been working fairly  swimmingly for the reporting database, which I'll get to later. That's another story. Next  component in the backends search. And that has three main components. The solar service,  RabbitMQ, which solar holds all your search information when you run knife search. RabbitMQ  takes all the information when you make a change in Chef, like you save your node, it  posts it onto the queue. And then there's a service called ChefExpander, which takes  the changes off the queue and then posts the changes to solar. And if you're actually using  a brand new version of Chef server, using the new Chef head chair that's been released  in the past two weeks, all of this is actually just elastic search. What we decided to do  was just keep all these services together on the back end machine. And there's one other  component to that diagram that's part of the back end that I've not mentioned yet, and  that is Redis. Redis is used by the front end machines basically for things like feature  flags, web sessions, and a way for us to enable maintenance mode across all of the servers.  You've got NGX, the load balancer at the front, listening to Redis or checking Redis  for information for things like how to set maintenance mode. For that, we just used elastic  cache. It's basically hosted Redis as a service, and we just have to use that. That brings  us to the front ends. So on old hosted, and this is part of the reason for that big diagram  before, you had all of these services, a Chef, by first, or CID on its own server. Most of  these services are pretty stateless. So we were able to combine these and in a normal  Chef server install, all of these are combined onto the same machine. All the services are  just on front ends, and you can horizontally scale them. So there's just a few other parts  of the infrastructure left. First off, cookbook storage. When you upload your cookbooks, the  actual physical data, where you have your recipes and attributes and all the actual  cookbook data, that gets uploaded to S3. That made the migration process really easy, because  in old hosted, it was in S3. In new hosted, it was in S3. We didn't have to do anything.  Other components, load balancer in front of everything. We just used Amazon's ELB. That  was a fairly easy decision. It just goes in front of the front ends. Load balancer is  everything. And then there's one other machine we have in hosted that you may or may not  have in a non-prem install. That is supportbox. And basically that is a machine we can log  into. It configures us front end, but it's not in the load balancer. And we use that  for troubleshooting, password resets, any issues you might come across with, or because  we might need some manual fixes. We just use that as a support, do all that on the supportbox.  So we've got the various components decided. Now we have to actually get things running.  So the first thing we do is we put everything into auto scaling groups. Now, when you hear  auto scaling groups, you think about machines magically growing, like you see the load growing  and it spins up new machines. Problem is, our spiky load, that was for, it went jumped  up at the top of every hour and then went right back down again. And it takes about  10 minutes to spin up a new node. So we don't actually use auto scaling groups to auto scale  our infrastructure. We need to be at full capacity anyway. What we use auto scaling  groups for is we set a static threshold and AWS makes sure that there's always 16 nodes  or 14 nodes. Whatever we set it to. And that means it just reinforces the idea that we  want to treat our servers as cattle and not pets. If we have a machine that's misbehaving,  we can just kill it and AWS will just bring up a new machine to replace it and that is  really, really nice when you get paged. So if we're using auto scaling groups, if you're  using auto scaling groups, you need to build images and to do that, we use Packer. We build  images every time we deploy a change, which isn't that often. Like new Chef server versions,  something like that, or company changes. And when we do that, we destroy all of our old  front ends or our old instances and we recreate new ones. So most of the setup that we do  is actually done in Packer during the image build. There's about three types of things  that we do afterwards. That is anything that's specific to the environment, so anything that  needs to be different between acceptance and delivered. Or pre-prod and prod. Anything  that's specific to the instance itself, things like host names or node names, specific Chef  configuration and credentials, which they're actually kind of a specific case of environment  and specific config, but we treat them separately because they kind of need special treatment  there. All additional configuration after boot, it's done with Chef. So that configuration,  when machine starts up, we have a user data script that sets up Chef and runs it on boot.  So with AWS, you can set a shell script that it just runs on boot when the instance comes  up. Now, we can't actually use the validatorless boot shot method. That's the one where you  don't need a validator key. Because we need the machines to register themselves as part  of the auto scaling group. So we just drop a validation key in Etsy Chef, tell Chef  to run, the machine registers itself, and it goes on to configure things. We also use  policy files to manage configuration. And that's been really useful for a number of reasons  and you can read about the benefits of policy files. One big thing that we do is we make  use of something called named run lists. And this is a separate run list for a node that  you can run by specifying it by name. So what we do there in Packer, we say use the build  run list and it just does the build time config. And then we've got our normal run list, which  does all of the configuration with Chef, including the environment specific, the credentials,  and the other things I mentioned earlier. So for the actual AWS resources, instances,  RDS, Elastic Cache, we use Terraform. That manages all of our AWS resources and it makes  provisioning really easy. It's your infrastructure or rather the instances and the actual AWS  resources as code. And one of the ways that this is really, really helpful, about four  days before we migrated, or rather four days before we were scheduled to migrate, we were  happily set up in US West 2. One big problem with that, most of our customers were located  in US East 1. So we were in US West 2 because of reasons, but someone made the decision  to actually move us to US East 1 to be close to where all the customers are. And we were  able to manage that migration in roughly one day. And that migration, that day, was mostly  transferring data that we'd already got in place. And going through the Terraform config  and the Chef config, just removing any assumptions we had about regions.  So I mentioned earlier, when we do auto scaling groups, we build an AMI and then we destroy  our instances and we create them. Problem is, if you do that, you destroy all your instances,  host the Chef goes down. So what we actually do is we rebuild our images in Packer. We  bring up the new instances in a separate auto scaling group that is not live. And we run  a few tests and then once we get everything working, we verify it's all there, we swap  out the auto scaling groups. And what that lets us do, it lets us test beforehand and  it also lets us quickly roll back if things go wrong.  So we only do this for the front ends. And one of the reasons for that is auto scaling  groups and blue-green deploys, they work really, really well for stateless machines. You can  just kill an instance and it will come back and no one really cares. For stateless services  such as our back end, that's running your search, that has a lot of data and we need  to be much more careful with that. We can't just kill an instance because that might be  the instance where all the data is stored. So we just need to make sure that when we  do our deploys and we need to change things, we make sure we kill the secretary back end  first and then we failover and then we can do the old primary.  If you have single machines, you can still use auto scaling groups. Now if you've got  a single machine, chances are you don't really care about it going down because you really  have AWS. In AWS, instances randomly get destroyed. And if you think that doesn't happen, you've  got nothing coming. It does. But our support box, I mentioned before, that's just a machine  we have to search into to do some troubleshooting or to reset passwords. If that goes down for  five minutes, we don't care. But if it goes down for five minutes, we still want AWS to  bring up a replacement for us. And that's happened numerous times in support boxes,  which just disappeared. And I've logged in and I'm like, oh, this is a new box. And I'd  never even known because we don't need to monitor it. We don't need to know because  we don't really care that the support boxes are there until we actually decide we want  to log in. So you can still use auto scaling groups for single machines.  So we're now up to the migration process itself. We know how we want to get everything up and  running AWS. Now we need to migrate. So like the project as a whole, we split up the migration  into a few different tasks. Moving the database, specifically the Chef server database and  the reporting database. Moving the search data, all the backend data, solo content. And finally,  flipping the switch, flipping DNS, moving people over to the new infrastructure.  First thing on that list is migrating the Chef server. So the Chef database server.  The Chef database server is where all of your data stores were like nodes, eight-scots,  roles, your clients on the code metadata, basically everything except for the code books  themselves. And in all hosted, this was actually three databases. We had one for the Chef server  itself and two others for a service called Bifrost, which did authentication. And a service  for OCID, which also did authentication for a different service. In new host days, we  decided we should merge all those databases into one, which is what an on-prem install  looks like. So we chose ODS earlier to host the database. So one big problem with that,  you can't do streaming replication from Postgres into ODS. I think you might be able to now  because AWS has got this like database migration service. But at the time, we couldn't do that.  We had to just copy all the data wholesale from our old database into ODS. Thankfully,  databases are both small. All the data for everyone in hosted Chef is actually less than  100 gig. So we were able to just copy everything across and do a copy of the cross during  migration. Next on that list was the reporting database. Now, reporting is the Chef server  add-on, Chef server reporting add-on. And that keeps records of every Chef run you do  against hosted, which is a lot of data. At the time of the migration, that was about  two to three terabytes, which I know some databases can be bigger, but that was something  that we certainly couldn't just copy across in five minutes. And ODS only supports up  to about six terabytes. So allowing for some growth, we couldn't be in ODS. So for our  reporting database, we decided to use just normally C2 instances running Postgres, which  meant we had to manage those databases. But it also meant we could use streaming replication.  And that meant joining migration for reporting, at least. We could just replicate all the  data, and then when the migration came, we just fell over the database, and migration  for that is pretty much instant. But before we could do the replication, you need to take  a base backup. Now, old hosted data center, Burmese, we had gigabit connection. So you  do your math, and comes out about one terabyte every three hours at gigabit speeds. Now,  you never really get your full bandwidth for various reasons. So we're doing two to three  years. Let's allow full day. Should be plenty of time. Three years later, we made a few  mistakes by our calculations. First thing, we're transferring across the US. So you've  got to deal with latency. That will slow down your transfer rates. Latency wasn't the biggest  issue we had, though, because we're going from Chicago, so it's probably not all the  way across the US, middle of the country to the east coast. But we also assumed that we  had full gigabit. Normal hosted traffic, because, well, hosted, we still had to serve  everybody. That ate about half of our bandwidth allocation. So we're already down to half.  On top of that, reporting, that stores every chef run. And people querying every chef run  they've ever done. Or at least querying every chef run for the past month. That's a lot  of data. And the reporting server disks were constantly saturated. So that means there  wasn't basically nothing left for us to transfer the data across. So the actual transfer rates  we got were more like 10 megabytes a second, or roughly 18 megabits. And on top of that,  our transfer, we were using all sync to begin with, that just died regularly. Like we transferred  a couple of hours, it would just stop crashing out, and then we'd have to start again. So  we used a few tricks to speed things up. We tried parallel or sync, some of these fancy  tools you can get that claim to transfer your data across the country 10 times faster than  also capable of to deal with the latency issue. But all of these things, it still took us  about a week with those tricks to transfer everything. And by that time, Postgres has  these things called wall files, which store the replication data while you're taking  the base back up. But it only stores so much. In the time it took us to transfer everything,  the wall files had rotated, and we couldn't start replication. So spent a week transferring  everything, realized we can't start replication, repeat the process all over again. So about  two weeks later, we finally get reporting transferred and replication started. Reporting  done. Next service, solar. So as I mentioned before, solar holds all the search data. This  is what gets queried when you run a search or in your recipes if you run the search function  against nodes or against data bags, things like that. This, the actual data stored was  much, much smaller than either reporting or the Chef database. And on top of that, solar  has this built-in replication. It's not great replication. It's not streaming replication.  It basically transfers the entire dataset once every hour. But it does a fair job, and  the dataset we had for this was probably on the old, or about, say, 10 gigabytes rather  than 100. So that meant we could choose the solar replication and things would just go  back. And in case anyone's wondering, so Chef server, it has a command called reindex, where  it will take all the data that you have in the Chef database, and it will populate solar  from the data in the Chef server database. The reason we didn't use that is it just takes  too long to do. For this dataset size of hosted, we're probably talking at least a  day to reindex everything from scratch. So it's much easier to just transfer all the  data as is across from one to the other. Next backend component is Redis, or Elastic  Cash. That was even simpler. So if you recall from earlier, Redis basically stored feature  flags, web sessions, whether we were in maintenance mode, and all of those things, the data is  pretty static, and the data we had in old hosted, that was feature flag settings for  some ancient feature we added years ago. So what we were able to do was just start the  new Elastic Cash, or the new Redis database, with a completely fresh blank database with  just the settings that we needed that we could manually put all the settings in for new hosted.  That brings us to migration day itself. And we decided to migrate at night time. There's  really a good downtime for hosted Chef. I think you might have noticed before, over  that 12-hour period, the load was fairly flat with the spikes. But we decided to do it when  at least most customers would not be looking at hosted Chef. But we start early in the  morning. First task, before anything else, lower your DNS TTLs. So if anyone doesn't  know, TTL in DNS is basically how long after you do DNS lookup, your laptop or your server,  whoever's doing the lookup, will keep the name information without looking up again.  Which means if the person goes and changes it, such as during a migration, you'll still  have all the old information. So we loaded it to 60 seconds, which meant that when we  flip the switch, most, if not all of our customers, would change over within the 60 seconds.  Next we do test run. That's just test migration, and then we make a few final config changes.  And then night time came. And we do our migration process. First thing, update status site.  All of you know we're going to migrate. Then we enable maintenance mode. And then we do,  we actually migrate the databases. This is just a procedure that we mentioned before,  flipping switches where we got replication on, initiating data transfers where we don't  have replication, and then using the sole replication. Then we test. We do a quick test  with an organization. We took out maintenance mode, make sure everything worked. Then we  do a quick load test, which we've tested earlier. We want to do a final load test, make sure  everything didn't fall over. Then we flip the DNS, and then we update the status site.  And we're migrated. So I mentioned load testing there. We did a homegrown tool called Chef  Swarm that just simulates lots and lots of Chef runs at once. And we used it to simulate  peak load, like top of the hour load, and new hosted. And we test against a single organization,  like just test org that we had that was out of maintenance mode so we could still do it  while everyone else was still in the maintenance mode. There's a few problems with that. First,  be very, very careful about running your load test against active prod. I also mentioned  that we tested against a single org. So we were like, we know this load test is not completely  representative of an actual genuine load. So we expected that our load test would tell  us everything's fine. And then we flip the switch, and the fear that I had was we flip  the switch, real traffic come along, and the whole thing would fall over. Turns out the  opposite happened. Join the migration. Now, we did the load test earlier in the day. We  knew, according to the load test, we were good. We do our final load test, join the  migration, just to make sure everything's okay, and everything falls over. Hosted just  crashes and burns, and we're like, what the heck is going on? So we double check all of  our procedures, like everything's okay. We're not sure what the heck is going on, but we  think there's something wrong with the load test. So we knew we were good to go, flip  the switch, and once we flip the switch, life traffic came along, and Hosted took the life  traffic without any problem at all. Turns out that when we did that load test during  the actual maintenance, we still had life traffic hitting the front ends, and then getting  a maintenance mode response back at the same time we were doing the load test. What that  meant was that the NGINX parts on the front ends, they were basically taking all of the  current production traffic plus retries, and the load test at once, and that was why it  fell over. So lesson learned there is be very, very careful with your load testing. It doesn't  always tell you the truth. So we migrated. Load testing's complete. We flipped DNS, and  everything's great. And these are the results. It's kind of hard to see there, but the top  graph is right before the migration, and that is the top of the owl peak, which at 90%  retires roughly 500 milliseconds for an average request, or for 90% our request, rather. Right  after the migration, that went down to 50 milliseconds for the 90% is a factor of 10.  And just to remind you, that's old Hosted. Big mess. Lots of services. Hard to understand.  I'm not convinced I fully understood this architecture before the migration. Probably  understood most of it, but not convinced myself. And that's the architecture afterwards. Much  cleaner architecture, much easier to understand architecture. We've got much improved deployment  method. We've got better ability to scale. We can just add new instances as we need them.  And we're running the same Chef server packages as everyone here who is running Chef server.  Any questions? Okay.  So the question was, are we exposing Hosted Chef to hosts outside of AWS? And the answer  is yes. We do have a lot of customers who are located in AWS. And sometimes before the  migration we checked where most of our customers were coming from. But yes, Hosted Chef is  available for anyone who wants to use Chef. And chances are, if you are using Chef to  begin with, like learn Chef or Chef tutorial, you may have touched Hosted Chef just to get  open-bunny real quickly. Someone else has a question?  We point the front ends at a separate Chef server. So originally we just had a separate  Chef server. Now as of about March this year, we're using Chef delivery. So that's a whole  other talk.  Okay. So the other question is, are you guys going to move to the Chef AJ product that  you guys announced this week?  Quite possibly yes. However, when we do that, I don't know.  So we actually scheduled the maintenance window of about four hours. That was just to allow  time to transfer everything over, because we need to transfer that over the whole Chef  database. And also to allow time to troubleshoot any issues that came up. We probably got  through the migration in about, probably a little under half of that time. So let's  say one and a half hours. Any more questions? Yes?  What's the total node count of the resource?  The total node count is about 400,000 managed nodes.  About how many organizations?  About 100,000. Yep?  How many of you keep the list perfectly? What's the staff?  Well staffed. We have a team of about four or five people right now.  They, so we used to be 100% host chef. This is actually, this is another big reason for  the migration. In addition to the devs working on hosted chef, we were spending all of our  time dealing with issues with host chef. We had these micro outages constantly and just  dealing with these issues. Since we migrated, we've actually been able to be not 100% focused  on hosted chef anymore. So we've moved on to other services.  Any other questions? Okay. Well thank you very much everyone.
It_s_Time_for_Us_to_Move_0_131.png It_s_Time_for_Us_to_Move_132_263.png It_s_Time_for_Us_to_Move_264_479.png It_s_Time_for_Us_to_Move_480_527.png It_s_Time_for_Us_to_Move_528_635.png It_s_Time_for_Us_to_Move_636_683.png It_s_Time_for_Us_to_Move_684_1967.png It_s_Time_for_Us_to_Move_1968_2507.png It_s_Time_for_Us_to_Move_2508_2555.png It_s_Time_for_Us_to_Move_2556_2927.png It_s_Time_for_Us_to_Move_2928_4091.png It_s_Time_for_Us_to_Move_4092_4823.png It_s_Time_for_Us_to_Move_4824_5243.png It_s_Time_for_Us_to_Move_5244_5519.png It_s_Time_for_Us_to_Move_5520_6707.png It_s_Time_for_Us_to_Move_6708_7079.png It_s_Time_for_Us_to_Move_7080_7619.png It_s_Time_for_Us_to_Move_7620_8471.png It_s_Time_for_Us_to_Move_8472_9179.png It_s_Time_for_Us_to_Move_9180_9563.png It_s_Time_for_Us_to_Move_9564_9611.png It_s_Time_for_Us_to_Move_9612_9659.png It_s_Time_for_Us_to_Move_9660_11003.png It_s_Time_for_Us_to_Move_11004_12323.png It_s_Time_for_Us_to_Move_12324_13271.png It_s_Time_for_Us_to_Move_13272_14447.png It_s_Time_for_Us_to_Move_14448_15527.png It_s_Time_for_Us_to_Move_15528_16319.png It_s_Time_for_Us_to_Move_16320_21767.png It_s_Time_for_Us_to_Move_21768_23819.png It_s_Time_for_Us_to_Move_23820_24455.png It_s_Time_for_Us_to_Move_24456_25703.png It_s_Time_for_Us_to_Move_25704_29687.png It_s_Time_for_Us_to_Move_29688_29939.png It_s_Time_for_Us_to_Move_29940_30659.png It_s_Time_for_Us_to_Move_30660_36071.png It_s_Time_for_Us_to_Move_36072_38543.png It_s_Time_for_Us_to_Move_38544_39947.png It_s_Time_for_Us_to_Move_39948_43487.png It_s_Time_for_Us_to_Move_43488_45299.png It_s_Time_for_Us_to_Move_45300_47531.png It_s_Time_for_Us_to_Move_47532_48119.png It_s_Time_for_Us_to_Move_48120_48167.png It_s_Time_for_Us_to_Move_48168_50615.png It_s_Time_for_Us_to_Move_50616_51551.png It_s_Time_for_Us_to_Move_51552_51995.png It_s_Time_for_Us_to_Move_51996_57293.png