Beyond the Cookbook: Using Workflow to Bring Continuous Delivery to And Project
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Robinson, Jean-Paul
While we all know workflow provides an easy way to do continuous integration/delivery for cookbooks, we also all have other parts that need to be developed and maintained for a successful DevOps environment. This talk will provide information on how to extend the power of delivery to those other projects, and provide the basics needed to understand creating build cookbooks well enough to create one for any project. Topics covered: Understanding the build cookbook, what the phases of the build cookbook are, and how workflow uses them. Using dependencies to tie together both cookbook and non-cookbook items. Demo: Using workflow to build a small web application including building the Web server from source, and delivering to an end environment.
 Howdy, welcome again to Beyond the Cookbook using Chef Workflow to bring to you this delivery journey project. I'm JP Robinson. I'm a DevOps architect with SAP NS2. I'm sure most of you are here, saw Carmen's presentation this morning. And the main takeaway here is that we are a 100% independent subsidiary of SAP that caters exclusively within the federal space for US government entities. So how many of you here know what Chef Workflow is and have used it before? Okay, some of you. For any of you that don't, it's a continuous integration, continuous delivery pipeline. It basically is turnkey solution for if you have Chef Cookbooks and want to deliver them in a CI CD pipeline very easily. It's part of Chef Automate, and it actually does build cook, it uses the build cookbooks so that you can deploy things in a standardized and automated way using all the syntax you already know with a cookbook in the first place. But it only deploys cookbooks, right? No. You can use your build cookbook to do just about anything. So the build cookbook tells Workflow how to deploy your code, delivery trucks the standard turnkey example when you do Chef Generate Cookbook, it by default will start a new project for you with a dot delivery directory and a build cookbook that just goes through and calls delivery truck to run various phases. As I said before, normal Chef resources and can do just about anything a normal Chef cookbook can. So most of you who have used Workflow before are probably familiar with these phases. Usually it goes through verify, build, acceptance, union, rehearsal, and delivered. The idea here is also that verify happens before you merge, and there's an approval gate for merging it before it builds. The build is where most CI CD pipelines would say you're supposed to build your artifacts, and then it starts out to acceptance where it starts deploying and testing your artifacts. Once you pass acceptance, you have another gate to deliver it into your production environment passing through union, rehearsal, and delivered. Delivered being ultimately your production environment. So we can take those and kind of separate them into three distinct places. Pre-merge. Pre-merge is everything that happens from the moment you submit your pull request and say this is a change that I want to do. Post-merge is what happens once that code has been approved and merged into your master tree, and then deployment is the phases that are purely for deploying into production. So they deploy, they test, and that's all they really do. The verify phase, I'm going to go through what Delivery Truck does with it. It runs the R-spec Chef spec unit tests. It does a lint check with Rubicop or Cookstyle to make sure that your code is good in passing and no errors there. And then it does a final syntax check with KnifeCookbookTest, and also checks to make sure that you've updated your version number in the Cookbook because Delivery works by pinning the various versions within the environments that it delivers to. So if you haven't updated your version, there's nothing there to deliver. Then moves on into the build phase, which has the verify stage again. It runs all three of those verify stages again, post-merge, to make sure, for instance, that if you submitted your change and then somebody else came along and submitted their change, that you don't have any conflicts or anything that breaks the build after that point. The quality and security. Within Delivery Truck, these are left intentionally blank because the idea here is that you should write your own quality and security tests. Publish. Publish is the code according to the config JSON in your.delivery directory. With Delivery Truck, there are several options, including your local Chef server, your local supermarket server, or even the global supermarket server repository. Once it passes through build, it goes to acceptance. The idea here being acceptance is your first dev test environment. Pins the Cookbooks in the appropriate environment. It attempts to find anything that has this Cookbook assigned to it and deploys it via push jobs to make sure that Chef Client runs and it gets deployed out within that environment. Smoke and functional stages are also left intentionally blank. Again, the idea being you should write your own smoke test for each of your projects or your own functional test. The truth is these phases get repeated throughout union, rehearsal, and delivered. They're exactly the same going through each of them. The only difference is that union also runs these for anything that depends on this project. This is useful, for instance, if you've got a library Cookbook or when developing your own build Cookbooks, if you've got something that has libraries, for instance, you can actually tell it that the main project depends on those library projects and it will actually rerun with the new libraries to make sure that everything works. Which of these phases is really important? If you're going to write your own build Cookbook, well, which of these phases, where do I want to start? What do I want to do with them? That's really up to you. You should decide where things should happen within your project. If you've got a project, for instance, that you need to run Kitchen before you merge, then you should put that obviously into the verify section, either in unit, lint, or syntax. If you've got projects that should only be published after a major update, then you should probably check in the acceptance phase and fail acceptance to make if you haven't done a major version update or minor version update to make sure that it does not go through to your production environment. Basically you can also do things, for instance, if you wanted to say, well, I want to not have any gates whatsoever, I want to basically have it so everything goes through as soon as somebody submits it, you could technically do that by putting everything in the verify stage. It's probably a bad practice to do it, however. It depends on what you're building, basically. Do you need just a standard delivery truck for building your Cookbooks? Do you need something with a little more power? Or do you need to build something entirely different, your own vehicle? Some things to keep in mind as you do your own build Cookbooks. The phases are flexible. You can put whatever you want into any phase. For instance, if you want to publish in the provision phase, you could do that. You just need to keep in mind what happens in the pre-merge versus the post-merge, and then what happens in each of those deployment phases. Keeping the phases close to what they're named for the function will help you later if you come back to your project and go, well, where did I put that again? I don't remember. Some of the limits on the build Cookbook, it's worth keeping in mind that the Cookbooks for building are run as a non-privileged user on your build notes. That means, for instance, if you need to install packages to do whatever build you're going to do, you're probably going to have to provide that user with elevated privileges to do so. SUDU, DZDU, whatever you use. If you need to, say, compile something else entirely as root, you're going to have to provide it with the ability to do so. The other thing is that the phases, each of those phases, each of those sections, runs on different build notes. If you have 10 build notes, you're not likely to end up on the same build note for the next phase. Even if you do, each of them is isolated into its own directory on the build note. You can't depend that the state immediately after you finish one phase is going to be the state that you enter the next phase with. It completely checks out your project again so that it has a clean slate. Don't expect it to remember state at all. Building the build cookbook. Where should I start? I've got my project. I've got my source code that I want to make a build cookbook for. This isn't a Chef cookbook. I don't want to just do a Chef generate cookbook. I want to do a Chef generate build cookbook. It will actually add a.delivery directory with a completely empty cookbook. It sets out the recipes named for the phases which is required. But each of those phases will be empty. It won't include delivery truck automatically. How do I figure out what goes where? Same way you would building any automation project. Determine what your build steps are. Determine what you need to do, pre-merge, what you need to actually build and package it for deployment, and then how you want to actually do that deployment. Basically keep these in mind. Anything that you want to do before that code is merged into your code base, you want to do in the verify stage. Anything that you want to do for building your project, you probably want to put into the build phase. Acceptance is kind of a test run of the deployment. So it's really a deployment phase, but because of that gate there, I've kind of grouped it into post-merge. The next part of it is the deployment phases. How am I deploying this into production? How am I deploying this into union? How do I deploy this into my stage environment rehearsal? And then how do I deploy it ultimately into the production environment? So I'm going to do a demo. I've got a demo project, build Apache. It just basically builds Apache from the Apache Git repo based on a configuration file that tells it how I want it to build. It packages it for deployment and deploys it to nodes within my demo environment. The code for this can be found at my GitHub, which is github.com. Under the build Apache project. So that will be available. So what am I doing with it from my end? I've got a unit phase that will actually do a test compile to make sure that before it merges, it at least compiles. If it doesn't compile, obviously there's no reason for me to merge that in. It's not good code. The lint will actually check my configuration file that I set up for any errors and reject anything that doesn't have a good configuration file. Syntax, I've actually set it up to skip entirely for right now. The reason for that is I couldn't think of anything else to run through before I merged it. So I just want to show you guys the cookbook here. This is what the configuration file looks like. And we'll actually go ahead and make a change here to bump its build number so that it will build a new version of it. I've got here my required packages, which is just literally a list of the RPMs that are required for it to build because the cookbook goes through and installs those. Where to find the Apache source? Right now, like I said, it's pointed at the main Apache tree. You could fork this and point it to your own tree. APR and APR Utils, since those are also required to build Apache, where they get that source. I've also got an option down here to reduce the spam as it's building and basically not outputting anything from the make if I wanted it to. So I'm going to make... I'm going to actually go ahead and remember to branch first. So I'm going to create a branch for the demo, just like you normally would in your projects. Going to save my changes for that. I'm also going to come in here and update to skip the unit phase as well. And that's just because compiling Apache takes a bit of time. And if I do not skip that phase, we'll be sitting here for five minutes just watching it compile. So I'm going to go ahead and commit that and tell it to delivery review. So it will automatically pop up to the new change. And we'll start running through the phases, which right now it's only going to do lint. I'm going to go ahead and approve that and let it go on to the build phase. Well, I'll go over what it's actually doing. So as you can see in my unit recipe here, I'm basically just including the setup build and perform build actions that I have here. Setup build actually walks through, loads the configuration file, installs the required packages, and then sets up checking out each of those Git repositories for Apache and its required utilities. And then runs the build comps so that it's ready in any state that I can actually start doing the build for it. So set up the autocop setup. Perform build actually walks through, cleans up anything that may be there or left over for any reason, and then runs. slash configure with whatever options I specified in my configuration file that it read in. It then just runs make. So that's how it actually performs the build. So, come back to the presentation for a moment. The build phase. Verify would actually go through and do another compile post merge again to make sure nothing has changed since the first time we did it. Quality and security, I've also left those intentionally blank. I don't have any particular quality or security test I want to run within this demo. Publish. It does the full build. So this is now build number three. This time it will actually take the result of that build, package it up, and publish it to a web server that I have within my environment. So let's see where this has reached this point within the build phase. It is actually running the publish now, and you can see it here is actually running through spamming out the compile. So while it does that, so it runs through again this perform build setup. The other thing that it will do now is actually the package build recipe that I've set up, which will pull from a data bag, get some information like I've got an SSH key that it's using to copy the file over and publish it. Does the full build, installs it to a fake root directory to install the build locally, and then just basically does a tar to tar up that build file and push it out. It will then go and copy it over to the appropriate place on the deployment node for the repository. Like I said, this takes a while. Has a lot of spam to it. So I'm going to go ahead and move on for what it does in the next phases. So again, acceptance, you get a rehearsal delivered, all do the same thing. It will actually set an environment variable within the appropriate environment for here's where on that web server you need to go to find that new build that we just created, that new package. It will then do as the smoke phase, it will deploy the package locally from that package to make sure that the environment variable setting went okay, that the package actually will extract okay, and it's doing this locally on the build node itself, and then starts that Apache up, does a get request, just a HTTP get to the local Apache server, and then the functional test will actually go in, deploy a node within that environment that is running a cookbook that will deploy the code for it, deploy the code in the custom configuration, and then also the build node will send a get request to that server to check and see if it's operational. So just to go over those a little bit in detail. So the smoke test actually comes in, goes to the F server which this with server config that it keeps doing is actually set up in a cookbook called delivery sugar if any of you are familiar with that one. Delivery sugar has a lot of things that when you're writing your own custom build cookbooks, will actually make your life a lot easier because it will actually provides a good set of functions for you to work with, for working with your actual live chef server as opposed to just doing things locally on the build node itself. So it actually will come in here, figure out what the acceptance name is which is also within delivery sugar, and then I'm actually using a helper from delivery truck to retrieve that environment back from the chef server so that I can work with it later. So I've also got my build root here, I'm sorry this is not the provision phase. Provision phase, I'm doing the same type of thing here with the, Fetcher create the environment and then here's where I'm actually going through in setting the node attributes. So I set up the custom Apache attributes for the URL, where the doc root for that it should be and what build number it actually is on. And then it saves that environment. So back to the smoke test. The smoke test, like I said, creates the directory for it, pulls back the remote file, and then just extracts it. And as you can see, this is all the same type of code and same resources that you're familiar with from writing your own cookbooks, anything that you're already automating, you can do in here the same type of way. It then starts Apache locally with the appropriate environment for the libraries, sends an HTTP request again with a standard Chef resource, and as long as all that's correct, shuts down that Apache server. Very simple thing for it to do here. So the last part of that would be the functional test. Does much of the same thing with going in with server config, pulls back a data bag, actually does a node search here, same standard way you would do a node search in any of your other recipes, search for whichever terms you want, your recipe, custom Apache, Chef environment, whatever environment we're currently in for delivery. So if that's your acceptance environment, everything, project gets its own acceptance environment. If it's union, it'll pull back union as it's the union environment, and then it'll find each of those nodes. It will then go and use Knife SSH here to reach out to those nodes and just run Chef client here. Just so you guys can see the full command here. All I'm doing is executing Chef client and going through with my customer, with my user to get into that. Once it's run Chef client, once all those nodes have run Chef client, then it'll come through and for each of those nodes, it'll send a test request to the port that I've specified for it to listen on 8080 here, and just do a simple get request. If that passes, then the build passes. If any of those nodes fail to deploy it for any reason or anything like that, then obviously the CI CD pipeline fails and it stops at that point, go back, fix your errors, continue as you would normally. Let's go back and check and see how that build is doing. This was still back in the publish phase here, because it had lots of output for it. This is where it actually packaging it up here, creates the tar file, deploys it, and then moves on automatically into acceptance. Provision, and actually here's Ruby block for setting the current environment. You'll notice I didn't say anything about deploy. I'm not actually doing anything in the deploy phase. Smoke, it actually deploys out with a custom configuration file here. You can see it create the file, create the remote file, extract it, create the config file for it to use locally, and then send the get request up to local host. Then the functional test, it's just basically running through, sends it off to Knife SSH. There isn't a whole lot of output from that, but completes. That point, I can basically say deliver it, and it's going to go through and repeat those same phases for each of the environments. That's basically all there is to it. You're just using whatever Chef resources you already know to tell it, here's how I want you to build. Make configure, very simple to do. Any other thing else that you have to do, you can easily do it through here, just keeping in mind it runs on the build node and has some limits. Any questions for me at this point? Go ahead. I'm very impressed that we have verified or have taken syntax. That is a hard set order, and actually what it does is it runs all three of those phases at the same time for unit Linton syntax. Those three phases all run at the same time, they're all launched at the same time. If any of them fail, you obviously can't move on from verify. When it runs it in the build phase, it also will launch all three of those verify phases again at the same time before it moves on to quality or publish or anything else. Once it moves on from that, each of those phases is executed one by one. If you have cookbooks that depend on each other, even if they're not library type cookbooks, you've got a wrapper cookbook that depends on another cookbook. You're essentially using it as a library in that case, even though it's not a Chef library. In that case, you would want the wrapper cookbook to run with the attributes you're setting so that when you push it into union rehearsal delivered, you're not just testing, okay, this cookbook with the default attributes works. Make sense? Yeah, go ahead. Within the.delivery config where you're defining the build cookbook, you can pull that from Git, you can pull that. I believe you can pull that from the supermarket as well. It can be pulled from a variety of sources. Worst case scenario, you can just specify a Git slash slash URL for it. Anybody else have any other questions? I'm sorry? How many build nodes? Pipelines. This pipeline, this is just a demo, so it's only running the one pipeline. In my production environment, we have something like 100 projects in there. Some of them have more than one pipeline. Since we separate out, say, our Dev environment pipeline from our QA pipeline, so we have quite a few. Let me see here. Let me go back in time here if I can. The question was, how do I determine what caused part of it to fail, basically? Let me come back to one of my older ones. Here's one that failed, for instance, on the smoke test when I was writing it originally. I was up late writing it, and for some reason thought I needed to put the whole thing into a Ruby block, which obviously did not work so well. When it fails, it's just like a Chef Run failing, and it will give you a trace back. It will tell you, here's where I failed and why I failed. You can see it was processing this Ruby block that I had set up, undefined method, directory for Chef Resource Ruby block, because obviously that's a Chef resource and not a Ruby command. It will tell you where it was actually failing. It outputs Ruby block smoke test. Here's in the cookbook where I need to look at what it failed and why it failed, just like you would for any of your Chef Runs that fail. If you use this to set up a cookbook delivery, which is the example, if you forget to bump that metadata, it will very quickly fail out on the syntax check, and it will tell you explicitly in the error that it raises, you didn't bump the version number. You can do that with your own phases as well, where if it fails, especially if it's in a Ruby block, for instance, you can raise your own error message or log your own error message on certain failures. That's just like you would in the Chef Run. Any other questions? Yeah, I have one. So I'm very new to Chef Automate, and I noticed that on the pipeline, for each step, you have a recipe for that. Did you specify that in the default recipe in the cookbook, or did Chef Automate automatically pick that up? Which recipe are you asking about exactly? So in your cookbook, in your Chef, in your build cookbook, you already have a default recipe. Does that include all of the other recipes to go down the pipeline, or does Chef Automate automatically pick that up? This is my default recipe. Oh, don't let it. Is this my question? As you can see, it's fairly blank. You notice that each of the phases here within the build cookbook, and this is set up automatically when you do a Chef Generate build cookbook. Each of these phases are named for what the phase actually is. For instance, functional. When it runs functional, it runs the functional recipe. When it runs unit, it runs the unit recipe. The only place I'm doing includes here are for things like the unit recipe, where I'm actually including my two, my own custom recipes, because I divided the build into set up build, and perform build, and package build. I'm actually including those recipes. If I didn't, the only thing that would get run here is whatever's in unit, or whatever's in the next phase, smoke, publish, et cetera. I'm really new to this, but I'm trying to understand how much of these kinds of build cookbooks are publicly available? How much do you typically have to tweak to get it running? Okay, that depends on what you're doing. If you are doing purely cookbooks, then delivery truck is publicly available. As you can see, I have it pulled up here. I referenced it a few times. Delivery truck is basically the standard Chef workflow cookbook. It used to be called Chef Delivery, if you guys see my shirt. Delivery truck was basically the standard pipeline for move your cookbook through. This will deliver cookbooks into environments. It will pin them for you. It makes it very easy to take, this is my cookbook, and I want to deliver it into a production environment, have it pinned so I'm not worried about, well, did I update a version or not, which version am I actually pulling? It will actually take care of that for you. For writing other things, that's part of why I'm doing this presentation, is to show, number one, yes, workflow is very much able to handle whatever you want to throw at it. You do have to write some code for it. Like I said, mine is available on my GitHub at GitHub slash Dream Night. You can take that and modify it as you wish to support whatever kind of build you're doing, especially if it's a follows the auto-conf, configure, make, make, install type of build. If you have something more complex, you may have to go ahead and write your own complete code for it. If you guys are doing projects, I'd encourage you to do similar to, like with the supermarket, put your code out there for build cookbooks. Let it be out there because we can all point at different build cookbooks for different things. But if you've got, say, open source projects that you guys are working on and writing a cookbook to compile it and deploy it, put it out there. I've got a question regarding the early pipeline with the developers. I'm on the operations side. The developers we have, they push all their artifacts into Artifactory. When they go from Snapshot to Release, can that Release push into Artifactory? Can that trigger the Chef Automate piece and actually start into that verification to go out into the deployment phases? Basically, right now, the review on delivery is kicked off by that CLI command, Delivery Review. Delivery Review will actually take whatever the current state of the Git repository is and push it up to Delivery workflow, push it up to workflow, and then that kicks off the pipeline at the verify stage. Yeah. So now that you've worked with this, obviously more than a lot of us, does this feel like a natural fit, a good idea for building Apache or building VM images or building random thing? Versus using sort of the more traditional tools? Is this a natural fit or a force fit, in other words? I think it's a natural fit partly because I came into things from a very much Chef background where I'm writing a lot of cookbooks and things of that nature. If you already have your tools in place, you have your tools in place. This can replace them, but I can understand that trying to get people to change their ways, even trying to get people to adjust to say using Chef can be kind of a pain. Everybody's kind of pulled those teeth at some point, I'm sure. But it is a natural fit if you're writing cookbooks already to go ahead and write a build cookbook and use that as your pipeline. To me, it makes more sense than using other things where I have to worry about managing, say shell scripts or managing various different kinds of config files. I already know Chef, I can use it very naturally. By that same token, as far as a natural fit for your build process, I think this can build just about anything you want it to. Yesterday at the partner day, there was actually a demonstration of using it, they're using it to build out an entire demo environment, build AMIs with Packer, and then deploy them with an AWS. It's a very powerful tool. Do you have any experience or know of anybody that's making any build cookbooks with Terraform? I do not know for Terraform. I don't use Terraform specifically. I use Chef provisioning. I'm a little weak in that area. Maybe if anybody here has any experience with that, if you could come find me, that would be awesome. Okay, cool. Thanks. Anyone else? Okay, let's give a really big round of applause for John Paul. Thank you so much. Thank you so much. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.
 Howdy, welcome again to Beyond the Cookbook using Chef Workflow to bring to you this  delivery journey project.  I'm JP Robinson.  I'm a DevOps architect with SAP NS2.  I'm sure most of you are here, saw Carmen's presentation this morning.  And the main takeaway here is that we are a 100% independent subsidiary of SAP that caters  exclusively within the federal space for US government entities.  So how many of you here know what Chef Workflow is and have used it before?  Okay, some of you.  For any of you that don't, it's a continuous integration, continuous delivery pipeline.  It basically is turnkey solution for if you have Chef Cookbooks and want to deliver them  in a CI CD pipeline very easily.  It's part of Chef Automate, and it actually does build cook, it uses the build cookbooks  so that you can deploy things in a standardized and automated way using all the syntax you  already know with a cookbook in the first place.  But it only deploys cookbooks, right?  No.  You can use your build cookbook to do just about anything.  So the build cookbook tells Workflow how to deploy your code, delivery trucks the standard  turnkey example when you do Chef Generate Cookbook, it by default will start a new project  for you with a dot delivery directory and a build cookbook that just goes through and  calls delivery truck to run various phases.  As I said before, normal Chef resources and can do just about anything a normal Chef cookbook  can.  So most of you who have used Workflow before are probably familiar with these phases.  Usually it goes through verify, build, acceptance, union, rehearsal, and delivered.  The idea here is also that verify happens before you merge, and there's an approval  gate for merging it before it builds.  The build is where most CI CD pipelines would say you're supposed to build your artifacts,  and then it starts out to acceptance where it starts deploying and testing your artifacts.  Once you pass acceptance, you have another gate to deliver it into your production environment  passing through union, rehearsal, and delivered.  Delivered being ultimately your production environment.  So we can take those and kind of separate them into three distinct places.  Pre-merge.  Pre-merge is everything that happens from the moment you submit your pull request and  say this is a change that I want to do.  Post-merge is what happens once that code has been approved and merged into your master  tree, and then deployment is the phases that are purely for deploying into production.  So they deploy, they test, and that's all they really do.  The verify phase, I'm going to go through what Delivery Truck does with it.  It runs the R-spec Chef spec unit tests.  It does a lint check with Rubicop or Cookstyle to make sure that your code is good in passing  and no errors there.  And then it does a final syntax check with KnifeCookbookTest, and also checks to make  sure that you've updated your version number in the Cookbook because Delivery works by  pinning the various versions within the environments that it delivers to.  So if you haven't updated your version, there's nothing there to deliver.  Then moves on into the build phase, which has the verify stage again.  It runs all three of those verify stages again, post-merge, to make sure, for instance, that  if you submitted your change and then somebody else came along and submitted their change,  that you don't have any conflicts or anything that breaks the build after that point.  The quality and security.  Within Delivery Truck, these are left intentionally blank because the idea here is that you should  write your own quality and security tests.  Publish.  Publish is the code according to the config JSON in your.delivery directory.  With Delivery Truck, there are several options, including your local Chef server, your local  supermarket server, or even the global supermarket server repository.  Once it passes through build, it goes to acceptance.  The idea here being acceptance is your first dev test environment.  Pins the Cookbooks in the appropriate environment.  It attempts to find anything that has this Cookbook assigned to it and deploys it via  push jobs to make sure that Chef Client runs and it gets deployed out within that environment.  Smoke and functional stages are also left intentionally blank.  Again, the idea being you should write your own smoke test for each of your projects or  your own functional test.  The truth is these phases get repeated throughout union, rehearsal, and delivered.  They're exactly the same going through each of them.  The only difference is that union also runs these for anything that depends on this project.  This is useful, for instance, if you've got a library Cookbook or when developing your  own build Cookbooks, if you've got something that has libraries, for instance, you can  actually tell it that the main project depends on those library projects and it will actually  rerun with the new libraries to make sure that everything works.  Which of these phases is really important?  If you're going to write your own build Cookbook, well, which of these phases, where do I want  to start?  What do I want to do with them?  That's really up to you.  You should decide where things should happen within your project.  If you've got a project, for instance, that you need to run Kitchen before you merge,  then you should put that obviously into the verify section, either in unit, lint, or syntax.  If you've got projects that should only be published after a major update, then you should  probably check in the acceptance phase and fail acceptance to make if you haven't done  a major version update or minor version update to make sure that it does not go through to  your production environment.  Basically you can also do things, for instance, if you wanted to say, well, I want to not  have any gates whatsoever, I want to basically have it so everything goes through as soon  as somebody submits it, you could technically do that by putting everything in the verify  stage.  It's probably a bad practice to do it, however.  It depends on what you're building, basically.  Do you need just a standard delivery truck for building your Cookbooks?  Do you need something with a little more power?  Or do you need to build something entirely different, your own vehicle?  Some things to keep in mind as you do your own build Cookbooks.  The phases are flexible.  You can put whatever you want into any phase.  For instance, if you want to publish in the provision phase, you could do that.  You just need to keep in mind what happens in the pre-merge versus the post-merge, and  then what happens in each of those deployment phases.  Keeping the phases close to what they're named for the function will help you later  if you come back to your project and go, well, where did I put that again?  I don't remember.  Some of the limits on the build Cookbook, it's worth keeping in mind that the Cookbooks  for building are run as a non-privileged user on your build notes.  That means, for instance, if you need to install packages to do whatever build you're going  to do, you're probably going to have to provide that user with elevated privileges to do so.  SUDU, DZDU, whatever you use.  If you need to, say, compile something else entirely as root, you're going to have to  provide it with the ability to do so.  The other thing is that the phases, each of those phases, each of those sections, runs  on different build notes.  If you have 10 build notes, you're not likely to end up on the same build note for the next  phase.  Even if you do, each of them is isolated into its own directory on the build note.  You can't depend that the state immediately after you finish one phase is going to be  the state that you enter the next phase with.  It completely checks out your project again so that it has a clean slate.  Don't expect it to remember state at all.  Building the build cookbook.  Where should I start?  I've got my project.  I've got my source code that I want to make a build cookbook for.  This isn't a Chef cookbook.  I don't want to just do a Chef generate cookbook.  I want to do a Chef generate build cookbook.  It will actually add a.delivery directory with a completely empty cookbook.  It sets out the recipes named for the phases which is required.  But each of those phases will be empty.  It won't include delivery truck automatically.  How do I figure out what goes where?  Same way you would building any automation project.  Determine what your build steps are.  Determine what you need to do, pre-merge, what you need to actually build and package  it for deployment, and then how you want to actually do that deployment.  Basically keep these in mind.  Anything that you want to do before that code is merged into your code base, you want to  do in the verify stage.  Anything that you want to do for building your project, you probably want to put into  the build phase.  Acceptance is kind of a test run of the deployment.  So it's really a deployment phase, but because of that gate there, I've kind of grouped it  into post-merge.  The next part of it is the deployment phases.  How am I deploying this into production?  How am I deploying this into union?  How do I deploy this into my stage environment rehearsal?  And then how do I deploy it ultimately into the production environment?  So I'm going to do a demo.  I've got a demo project, build Apache.  It just basically builds Apache from the Apache Git repo based on a configuration file that  tells it how I want it to build.  It packages it for deployment and deploys it to nodes within my demo environment.  The code for this can be found at my GitHub, which is github.com.  Under the build Apache project.  So that will be available.  So what am I doing with it from my end?  I've got a unit phase that will actually do a test compile to make sure that before it  merges, it at least compiles.  If it doesn't compile, obviously there's no reason for me to merge that in.  It's not good code.  The lint will actually check my configuration file that I set up for any errors and reject  anything that doesn't have a good configuration file.  Syntax, I've actually set it up to skip entirely for right now.  The reason for that is I couldn't think of anything else to run through before I merged  it.  So I just want to show you guys the cookbook here.  This is what the configuration file looks like.  And we'll actually go ahead and make a change here to bump its build number so that it will  build a new version of it.  I've got here my required packages, which is just literally a list of the RPMs that are  required for it to build because the cookbook goes through and installs those.  Where to find the Apache source?  Right now, like I said, it's pointed at the main Apache tree.  You could fork this and point it to your own tree.  APR and APR Utils, since those are also required to build Apache, where they get that source.  I've also got an option down here to reduce the spam as it's building and basically not  outputting anything from the make if I wanted it to.  So I'm going to make...  I'm going to actually go ahead and remember to branch first.  So I'm going to create a branch for the demo, just like you normally would in your projects.  Going to save my changes for that.  I'm also going to come in here and update to skip the unit phase as well.  And that's just because compiling Apache takes a bit of time.  And if I do not skip that phase, we'll be sitting here for five minutes just watching  it compile.  So I'm going to go ahead and commit that  and tell it to delivery review.  So it will automatically pop up to the new change.  And we'll start running through the phases, which right now it's only going to do lint.  I'm going to go ahead and approve that and let it go on to the build phase.  Well, I'll go over what it's actually doing.  So as you can see in my unit recipe here, I'm basically just including the setup build  and perform build actions that I have here.  Setup build actually walks through, loads the configuration file, installs the required  packages, and then sets up checking out each of those Git repositories for Apache and its  required utilities.  And then runs the build comps so that it's ready in any state that I can actually start  doing the build for it.  So set up the autocop setup.  Perform build actually walks through, cleans up anything that may be there or left over  for any reason, and then runs. slash configure with whatever options I specified in my configuration  file that it read in.  It then just runs make.  So that's how it actually performs the build.  So, come back to the presentation for a moment.  The build phase.  Verify would actually go through and do another compile post merge again to make sure nothing  has changed since the first time we did it.  Quality and security, I've also left those intentionally blank.  I don't have any particular quality or security test I want to run within this demo.  Publish.  It does the full build.  So this is now build number three.  This time it will actually take the result of that build, package it up, and publish it  to a web server that I have within my environment.  So let's see where this has reached this point within the build phase.  It is actually running the publish now, and you can see it here is actually running through  spamming out the compile.  So while it does that, so it runs through again this perform build setup.  The other thing that it will do now is actually the package build recipe that I've set up,  which will pull from a data bag, get some information like I've got an SSH key that  it's using to copy the file over and publish it.  Does the full build, installs it to a fake root directory to install the build locally,  and then just basically does a tar to tar up that build file and push it out.  It will then go and copy it over to the appropriate place on the deployment node for the repository.  Like I said, this takes a while.  Has a lot of spam to it.  So I'm going to go ahead and move on for what it does in the next phases.  So again, acceptance, you get a rehearsal delivered, all do the same thing.  It will actually set an environment variable within the appropriate environment for here's  where on that web server you need to go to find that new build that we just created,  that new package.  It will then do as the smoke phase, it will deploy the package locally from that package  to make sure that the environment variable setting went okay, that the package actually  will extract okay, and it's doing this locally on the build node itself, and then starts  that Apache up, does a get request, just a HTTP get to the local Apache server, and  then the functional test will actually go in, deploy a node within that environment  that is running a cookbook that will deploy the code for it, deploy the code in the custom  configuration, and then also the build node will send a get request to that server to  check and see if it's operational.  So just to go over those a little bit in detail.  So the smoke test actually comes in, goes to the F server which this with server config  that it keeps doing is actually set up in a cookbook called delivery sugar if any of you  are familiar with that one.  Delivery sugar has a lot of things that when you're writing your own custom build cookbooks,  will actually make your life a lot easier because it will actually provides a good set  of functions for you to work with, for working with your actual live chef server as opposed  to just doing things locally on the build node itself.  So it actually will come in here, figure out what the acceptance name is which is also  within delivery sugar, and then I'm actually using a helper from delivery truck to retrieve  that environment back from the chef server so that I can work with it later.  So I've also got my build root here, I'm sorry this is not the provision phase.  Provision phase, I'm doing the same type of thing here with the, Fetcher create the  environment and then here's where I'm actually going through in setting the node attributes.  So I set up the custom Apache attributes for the URL, where the doc root for that it should  be and what build number it actually is on.  And then it saves that environment.  So back to the smoke test.  The smoke test, like I said, creates the directory for it, pulls back the remote file, and then  just extracts it.  And as you can see, this is all the same type of code and same resources that you're familiar  with from writing your own cookbooks, anything that you're already automating, you can do  in here the same type of way.  It then starts Apache locally with the appropriate environment for the libraries, sends an HTTP  request again with a standard Chef resource, and as long as all that's correct, shuts  down that Apache server.  Very simple thing for it to do here.  So the last part of that would be the functional test.  Does much of the same thing with going in with server config, pulls back a data bag,  actually does a node search here, same standard way you would do a node search in any of your  other recipes, search for whichever terms you want, your recipe, custom Apache, Chef  environment, whatever environment we're currently in for delivery.  So if that's your acceptance environment, everything, project gets its own acceptance  environment.  If it's union, it'll pull back union as it's the union environment, and then it'll find  each of those nodes.  It will then go and use Knife SSH here to reach out to those nodes and just run Chef  client here.  Just so you guys can see the full command here.  All I'm doing is executing Chef client and going through with my customer, with my user  to get into that.  Once it's run Chef client, once all those nodes have run Chef client, then it'll come  through and for each of those nodes, it'll send a test request to the port that I've  specified for it to listen on 8080 here, and just do a simple get request.  If that passes, then the build passes.  If any of those nodes fail to deploy it for any reason or anything like that, then obviously  the CI CD pipeline fails and it stops at that point, go back, fix your errors, continue  as you would normally.  Let's go back and check and see how that build is doing.  This was still back in the publish phase here, because it had lots of output for it.  This is where it actually packaging it up here, creates the tar file, deploys it, and  then moves on automatically into acceptance.  Provision, and actually here's Ruby block for setting the current environment.  You'll notice I didn't say anything about deploy.  I'm not actually doing anything in the deploy phase.  Smoke, it actually deploys out with a custom configuration file here.  You can see it create the file, create the remote file, extract it, create the config  file for it to use locally, and then send the get request up to local host.  Then the functional test, it's just basically running through, sends it off to Knife SSH.  There isn't a whole lot of output from that, but completes.  That point, I can basically say deliver it, and it's going to go through and repeat those  same phases for each of the environments.  That's basically all there is to it.  You're just using whatever Chef resources you already know to tell it, here's how I  want you to build.  Make configure, very simple to do.  Any other thing else that you have to do, you can easily do it through here, just keeping  in mind it runs on the build node and has some limits.  Any questions for me at this point?  Go ahead.  I'm very impressed that we have verified or have taken syntax.  That is a hard set order, and actually what it does is it runs all three of those phases  at the same time for unit Linton syntax.  Those three phases all run at the same time, they're all launched at the same time.  If any of them fail, you obviously can't move on from verify.  When it runs it in the build phase, it also will launch all three of those verify phases  again at the same time before it moves on to quality or publish or anything else.  Once it moves on from that, each of those phases is executed one by one.  If you have cookbooks that depend on each other, even if they're not library type cookbooks,  you've got a wrapper cookbook that depends on another cookbook.  You're essentially using it as a library in that case, even though it's not a Chef library.  In that case, you would want the wrapper cookbook to run with the attributes you're setting  so that when you push it into union rehearsal delivered, you're not just testing, okay,  this cookbook with the default attributes works.  Make sense?  Yeah, go ahead.  Within the.delivery config where you're defining the build cookbook, you can pull that from  Git, you can pull that.  I believe you can pull that from the supermarket as well.  It can be pulled from a variety of sources.  Worst case scenario, you can just specify a Git slash slash URL for it.  Anybody else have any other questions?  I'm sorry?  How many build nodes?  Pipelines.  This pipeline, this is just a demo, so it's only running the one pipeline.  In my production environment, we have something like 100 projects in there.  Some of them have more than one pipeline.  Since we separate out, say, our Dev environment pipeline from our QA pipeline, so we have  quite a few.  Let me see here.  Let me go back in time here if I can.  The question was, how do I determine what caused part of it to fail, basically?  Let me come back to one of my older ones.  Here's one that failed, for instance, on the smoke test when I was writing it originally.  I was up late writing it, and for some reason thought I needed to put the whole thing into  a Ruby block, which obviously did not work so well.  When it fails, it's just like a Chef Run failing, and it will give you a trace back.  It will tell you, here's where I failed and why I failed.  You can see it was processing this Ruby block that I had set up, undefined method, directory  for Chef Resource Ruby block, because obviously that's a Chef resource and not a Ruby command.  It will tell you where it was actually failing.  It outputs Ruby block smoke test.  Here's in the cookbook where I need to look at what it failed and why it failed, just  like you would for any of your Chef Runs that fail.  If you use this to set up a cookbook delivery, which is the example, if you forget to bump  that metadata, it will very quickly fail out on the syntax check, and it will tell you  explicitly in the error that it raises, you didn't bump the version number.  You can do that with your own phases as well, where if it fails, especially if it's in a  Ruby block, for instance, you can raise your own error message or log your own error message  on certain failures.  That's just like you would in the Chef Run.  Any other questions?  Yeah, I have one.  So I'm very new to Chef Automate, and I noticed that on the pipeline, for each step, you have  a recipe for that.  Did you specify that in the default recipe in the cookbook, or did Chef Automate automatically  pick that up?  Which recipe are you asking about exactly?  So in your cookbook, in your Chef, in your build cookbook, you already have a default  recipe.  Does that include all of the other recipes to go down the pipeline, or does Chef Automate  automatically pick that up?  This is my default recipe.  Oh, don't let it.  Is this my question?  As you can see, it's fairly blank.  You notice that each of the phases here within the build cookbook, and this is set up automatically  when you do a Chef Generate build cookbook.  Each of these phases are named for what the phase actually is.  For instance, functional.  When it runs functional, it runs the functional recipe.  When it runs unit, it runs the unit recipe.  The only place I'm doing includes here are for things like the unit recipe, where I'm  actually including my two, my own custom recipes, because I divided the build into set up build,  and perform build, and package build.  I'm actually including those recipes.  If I didn't, the only thing that would get run here is whatever's in unit, or whatever's  in the next phase, smoke, publish, et cetera.  I'm really new to this, but I'm trying to understand how much of these kinds of build  cookbooks are publicly available?  How much do you typically have to tweak to get it running?  Okay, that depends on what you're doing. If you are doing purely cookbooks, then delivery  truck is publicly available.  As you can see, I have it pulled up here.  I referenced it a few times.  Delivery truck is basically the standard Chef workflow cookbook.  It used to be called Chef Delivery, if you guys see my shirt.  Delivery truck was basically the standard pipeline for move your cookbook through.  This will deliver cookbooks into environments.  It will pin them for you.  It makes it very easy to take, this is my cookbook, and I want to deliver it into a production  environment, have it pinned so I'm not worried about, well, did I update a version or not,  which version am I actually pulling?  It will actually take care of that for you.  For writing other things, that's part of why I'm doing this presentation, is to show,  number one, yes, workflow is very much able to handle whatever you want to throw at it.  You do have to write some code for it.  Like I said, mine is available on my GitHub at GitHub slash Dream Night.  You can take that and modify it as you wish to support whatever kind of build you're doing,  especially if it's a follows the auto-conf, configure, make, make, install type of build.  If you have something more complex, you may have to go ahead and write your own complete  code for it.  If you guys are doing projects, I'd encourage you to do similar to, like with the supermarket,  put your code out there for build cookbooks.  Let it be out there because we can all point at different build cookbooks for different  things.  But if you've got, say, open source projects that you guys are working on and writing a  cookbook to compile it and deploy it, put it out there.  I've got a question regarding the early pipeline with the developers.  I'm on the operations side.  The developers we have, they push all their artifacts into Artifactory.  When they go from Snapshot to Release, can that Release push into Artifactory?  Can that trigger the Chef Automate piece and actually start into that verification to go  out into the deployment phases?  Basically, right now, the review on delivery is kicked off by that CLI command, Delivery  Review.  Delivery Review will actually take whatever the current state of the Git repository is  and push it up to Delivery workflow, push it up to workflow, and then that kicks off  the pipeline at the verify stage.  Yeah.  So now that you've worked with this, obviously more than a lot of us, does this feel like  a natural fit, a good idea for building Apache or building VM images or building random thing?  Versus using sort of the more traditional tools?  Is this a natural fit or a force fit, in other words?  I think it's a natural fit partly because I came into things from a very much Chef background  where I'm writing a lot of cookbooks and things of that nature.  If you already have your tools in place, you have your tools in place.  This can replace them, but I can understand that trying to get people to change their  ways, even trying to get people to adjust to say using Chef can be kind of a pain.  Everybody's kind of pulled those teeth at some point, I'm sure.  But it is a natural fit if you're writing cookbooks already to go ahead and write a  build cookbook and use that as your pipeline.  To me, it makes more sense than using other things where I have to worry about managing,  say shell scripts or managing various different kinds of config files.  I already know Chef, I can use it very naturally.  By that same token, as far as a natural fit for your build process, I think this can build  just about anything you want it to.  Yesterday at the partner day, there was actually a demonstration of using it, they're using  it to build out an entire demo environment, build AMIs with Packer, and then deploy them  with an AWS.  It's a very powerful tool.  Do you have any experience or know of anybody that's making any build cookbooks with Terraform?  I do not know for Terraform.  I don't use Terraform specifically.  I use Chef provisioning.  I'm a little weak in that area.  Maybe if anybody here has any experience with that, if you could come find me, that would  be awesome.  Okay, cool.  Thanks.  Anyone else?  Okay, let's give a really big round of applause for John Paul.  Thank you so much.  Thank you so much.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.
Beyond_the_Cookbook_0_131.png Beyond_the_Cookbook_132_371.png Beyond_the_Cookbook_372_563.png Beyond_the_Cookbook_564_863.png Beyond_the_Cookbook_864_2615.png Beyond_the_Cookbook_2616_4679.png Beyond_the_Cookbook_4680_8855.png Beyond_the_Cookbook_8856_17615.png Beyond_the_Cookbook_17616_18635.png Beyond_the_Cookbook_18636_18767.png Beyond_the_Cookbook_18768_23819.png Beyond_the_Cookbook_23820_25055.png Beyond_the_Cookbook_25056_27059.png Beyond_the_Cookbook_27060_36071.png Beyond_the_Cookbook_36072_36623.png Beyond_the_Cookbook_36624_36839.png Beyond_the_Cookbook_36840_36959.png Beyond_the_Cookbook_36960_37439.png Beyond_the_Cookbook_37440_37847.png Beyond_the_Cookbook_37848_39527.png Beyond_the_Cookbook_39528_43847.png Beyond_the_Cookbook_43848_43955.png Beyond_the_Cookbook_43956_44747.png Beyond_the_Cookbook_44748_47291.png Beyond_the_Cookbook_47292_52463.png Beyond_the_Cookbook_52464_52883.png Beyond_the_Cookbook_52884_53003.png Beyond_the_Cookbook_53004_53207.png Beyond_the_Cookbook_53208_53255.png Beyond_the_Cookbook_53256_53531.png Beyond_the_Cookbook_53532_53639.png Beyond_the_Cookbook_53640_53783.png Beyond_the_Cookbook_53784_53867.png Beyond_the_Cookbook_53868_53915.png Beyond_the_Cookbook_53916_54179.png Beyond_the_Cookbook_54180_54251.png Beyond_the_Cookbook_54252_54311.png Beyond_the_Cookbook_54312_54587.png Beyond_the_Cookbook_54588_54767.png Beyond_the_Cookbook_54768_55007.png Beyond_the_Cookbook_55008_55655.png Beyond_the_Cookbook_55656_55703.png Beyond_the_Cookbook_55704_55955.png Beyond_the_Cookbook_55956_56015.png Beyond_the_Cookbook_56016_56267.png Beyond_the_Cookbook_56268_56555.png Beyond_the_Cookbook_56556_56843.png Beyond_the_Cookbook_56844_57143.png Beyond_the_Cookbook_57144_57239.png Beyond_the_Cookbook_57240_57311.png Beyond_the_Cookbook_57312_57357.png