Progressive Delivery
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Sanchez, Carlos
Progressive Delivery makes it easier to adopt Continuous Delivery, by deploying new versions to a subset of users and evaluating their correctness and performance before rolling them to the totality of the users, and rolled back if not matching some key metrics. Canary deployments is one of the techniques in Progressive Delivery, used in companies like Facebook to roll out new versions gradually. But good news! you don't need to be Facebook to take advantage of it. We will demo how to create a fully automated Progressive Delivery pipeline with Canary deployments and rollbacks in Kubernetes using Jenkins X, an open source platform for cloud native CI/CD in Kubernetes, and Flagger, a project that uses Istio and Prometheus to automate Canary rollouts and rollbacks.
 Yes. Okay. Welcome. Perfect timing. Well, probably you're here because you think continuous delivery is a good thing to do, right? If not, you're probably in the wrong room. And I'm going to start with a story, right? Some years ago, we were doing all this automation. We were deploying with Puppet across multiple machines everywhere. And then you're doing this continuous delivery. Every time we committed some changes to Puppet, then this would get deployed until you get to a point where I made a change and these broke the machines. We do some testing, but this actually broke the machines in a way that not only they got broken, but it prevented new updates to get into the machines, right? So this is an example of how hard it is to do continuous delivery right. So my name is Carlos Anchez. I work at Adobe. As Victor said, the important part is the Twitter. See, Anchez, if you want to say good things, if you want to say bad things, that's not the right one. It's a different one. And we're here to talk to you about progressive delivery, which is a term that was coined a few years ago, just less than two years ago. It started on Launch Starly. This is a company that do feature flags. I'll talk about that. So they started with a term progressive delivery. Then Redmond, the analysts, James Governon started also talking about progressive delivery. And I was like, I'm totally still in this term. I love it. It's a subtly what it means. It gives you the right mindset about what we are trying to achieve. So it's a term that tries to encompass all these technologies that are not new. This is very important. This is not new technologies. It's just they're being around for a while. But it tries to avoid the problems with these all or nothing deployment capabilities or strategies. And basically, new versions that you deploy, they are not replacing existing versions, but you run them in parallel for a period of time in production. This is the important part. You get production traffic and you do evaluation of metrics to ensure that your deployment is good or not good. So until you do that, so you keep getting more traffic, get the metrics, look at the metrics, decide whether the deployment is good or not. And continuous delivery is hard. And progressive delivery is trying to make continuous delivery easier to adopt by reducing the risk. This is the key part. We are trying to reduce the risk. Nobody wants to get to waking up in the middle of the night because of deployment is going wrong. Nobody wants to decide, like, do not deploy on Fridays. Do not deploy before a holiday because then we start out. Do not deploy on Fridays. Then, well, maybe you shouldn't deploy on Thursdays either. So it's a bad game that you would want to get in. So reducing risk, but avoiding downtime and limiting the blast radius. So if something is wrong, then you have to make sure that it affects the less amount of users as possible. But it also gives you another benefit, which is the shorter time since you decide to do something until your production users can see some of your production users. So some of the techniques, if you've been to Victor's talk, this is more interesting because I have a Spanish accent, as he said. So some things that he also covered for those of you. I knew some people left the room and new people came in after waiting for their probably for an hour or two. So rolling updates, this is something that Kubernetes does natively. You can do a Qtl command. I mean, here it is in Kubernetes. There's pretty much a lot of people. So every time, this is not different from doing rollout updates on VMs. It's just doing it with containers with pods in Kubernetes. But it's natively included in Kubernetes. It's very easy to do. You deploy a new version and then you get new pods in with a new version and it starts taking old pods with the old version from the pool that served your requests to get your traffic. So every time you keep going one pod, two pods, three pods and so on, same thing we used to do with VMs back in the day, not much different other than it's more automated. Blue green deployment, what did you learn about blue green deployment from Victor's? Don't do it. Okay. So if you're doing blue green deployment, basically means having two instances of your production clusters or production hardware or infrastructure or whatever and having something in front of them and then you can quickly switch from one to another. So when you are ready to roll the new release, you point to the new one. If there's any problem, you just go to your DNS load balancer, whatever it is that you have in front and you point to the back. So you have immediate rollback, but all your users are getting the new version. So if there's something wrong, all your users are going to see it. Canary deployments is a specialized way, it's a better way of doing things, I guess, where you get a percentage of your users that get the new version. So you start sending a 1%, 2%, 10%, 20%, whatever is the number of your users to the new version. This is something that Facebook is doing, Netflix, I mean all the big companies are doing, but you don't have to be Facebook to do canary deployments and I'll show you that later. And you can select the users based on multiple options, right? You can just say randomly, send some percentage of users to the new version or you could say people from, what a lot of companies do is employees get the new version first before any public user gets it. So you get employees first, then you can use HTTP headers, IP location, anything you want. And another common trend that I've seen is like people, I think Facebook does this, they roll it out first to New Zealand. So for whatever reason, New Zealand has the statistic population very similar to the US, so they roll it out in New Zealand, they get the new versions first before it gets rolled to the US or the rest of the world. So I guess New Zealanders are the guinea pigs here, but that's something that you can do. Another option in progressive delivery is feature flex. This is something that LaunchDarly and other companies are doing, where this is very interesting because it allows you to do several things. One is exposing new features to specific people. So you could say, you could ship something to production disabled by default, and then you could look and use either HTTP header also or location base or a flag and your, I don't know, URL or something. And then you could get the new features, so you could see how that behaves in production. You could enable that feature for specific users, for a percentage of users. And it gives you also the speed because if you're familiar with the DevOps report from Dora, the companies, the teams that deploy faster to production or to master, merge things into master faster or commit directly to master or have short leaf branches are the most productive teams. So feature flex allows you to continuously get things into master without exposing users to those things. So let's say you have a feature that it takes you two months to do, you don't want to have a feature branch for two months because that's going to be a horrible idea. It's a horrible idea. Don't do that. And feature flex allows you to disable that, commit to master and at some point, either starting trying it with users, iterate, fix things, start showing to more users and so on. This is the other way you can do deployments. I don't recommend it. Containers don't really fit in. This is actually a picture. How did the deployment, they unloaded containers in some island in the Caribbean before and that's how a lot of companies look to some days, but just don't do it. And a very important part of progressive delivery is monitoring and I like to say monitoring is the new testing because with testing, you're basically just checking things that you think may fail, right? You have to think about, okay, what can fail? I'm going to write some tests to ensure that it doesn't fail. Monitoring will tell you how things fail that you didn't expect to fail, right? That's the only way you can know about those things. And it's the only way you're going to find out how your users, because your users are always going to break your application in very new and innovative ways. So this is the way you catch that. And it also allows you to have a reaction to these issues automatically. So you have some data coming in, you have real data. So you know, this saying if all we have is opinions, let's go with mine, unless we have data, let's go with the data, right? So this will give you the actual data where you can do things based on that data. And the bad part is like progressive delivery requires a good number of metrics. You need to instrument your application, you need to know what means success, what means failure. These metrics, it's not only HTTP metrics or app time or things like that, but also could be business metrics, right? If at some point you have a metric that says how many items are people adding to their shopping cart, right? You're saying this metric going, well, okay. And at some point people are not getting things to their shopping cart, then you may think, okay, there's something wrong with the application. Everything gets served, but maybe they get an empty page or some page that I'm not getting error from that. So you have to catch that. And I love this quote from DevOps Para, to make error is human, to propagate error to serve in an automated way, does DevOps? So actually in my previous company, I'm not going to name names, and we had to do this demo. The CEO was going to do this demo at the big conference, and we were running these huge clusters with, it was like 2000 Jenkins masters running, I don't know, terabytes of memory, CPUs and everything. And at some point I'm getting on a flight to go to San Francisco to the conference, and I'm checking my Slack before the flight, and it's like, oh, I deleted the wrong cluster by mistake. So this was one day before, two days before. So I think if you haven't automatically destroyed something by mistake, it means that you are not automating enough, okay? You got to get to this point where you're saying, okay, I did something really bad automatically. That's the point where you're saying, okay, I mean, automation is good. No blame, no blame in postmortems, right? So Jenkins X, what does Jenkins X come here? Jenkins X, who is using Jenkins X? Okay, who is using Jenkins? Who is using Jenkins on Kubernetes? With a Kubernetes plugin, probably, yeah? Okay, that's something I wrote a few years ago. I'm glad you used it. Don't blame me, okay? So Jenkins X is the new project. I like to say that Jenkins X is to Jenkins like Java is to JavaScript. They have nothing in common, okay? Jenkins X runs natively on Kubernetes. Did I mention Kubernetes in this talk? I don't remember. And it uses a bunch of technologies that you probably know and some new ones. So it uses Docker, of course, everything runs on Kubernetes. A scaffold, rough, I'll talk about those. And Tecton Pro. So Tecton is the pipeline engine in Kubernetes, for Kubernetes. It uses spots and containers to run pipeline steps. This is a project from the Continuous Delivery Foundation now. And it uses this, basically, it's a model that you've probably seen with GitHub Actions and other tools where you have a pipeline, every step runs in a different container. So you have isolation and you have a scale. You have more pipelines, then you get more containers and everything scales. There's no central point. It scales as much as your Kubernetes cluster does. Prowl is a project that comes from the Kubernetes itself that implements chat ops. So it handles the GitHub webhooks. And then you can talk, you probably seen this now, it's very popular where you got a pull request and the people start slash something, slash something and then slash test and the test shows you the test results, slash retest and you can rerun things. So chat ops. Helm is the package manager for Kubernetes. De facto, standard. Scaffold to build Docker images. This approach is from Google that allows you to do typical Docker builds but also has support for other backends to the Docker image builds. Genki says it allows you to use Canico. I like Canico a lot. It allows you to do builds without the Docker demons. So it's safe to use in Kubernetes or at least safer. And you don't have to open your cluster to everybody to have access to the Docker demo. And also, if you are a maven user, you can use, I'm a long time member, well I committed to maven for a long time. Gip optimizes, you can have a maven plugin and you can run NVIDIA and something and then it will build a Docker image for you and it will optimize the layers with your dependencies. So when maven downloads the internet, it gets optimized with different layers for the Docker images. So when you change the dependencies, it's not having to rebuild everything from scratch. It caches things. And draft is a project from Microsoft that generates the Docker file and helps you to do a project. This is something that Genki uses when you import a project into Genki's X or when you generate a new project, it will basically create the side, oh, you have a Java project so I'm going to create you a Docker file and help charts using the Docker image for the open JDK or other open JDK and it will generate all this scaffolding for you. What I'm talking about Genki's X because I'm going to show you how to do progressive delivery with Genki's X. There's a web page that goes through all the details if you want to try it yourself. I'm going to show you a demo. It uses Istio. I mean, who's using Istio? Here? Some people. Are you happy? Yes? Okay. Probably because you don't have to maintain it, right? You don't update it, right? That's it. Don't touch it. Okay. So what I'm going to use from Istio is it's a service match. It provides you all these cool things, allows you a connection between services, allows you to do things like say, oh, I want these services to go encrypted, so securing connection between services, allows you to control the traffic. This is what we're going to use for canary deployments. We want to control what traffic goes to what service, the old version, the new version, and it also allows you to do observation of your application or your service without having to touch your application. So this is what we're going to do to get the metrics. Istio uses Prometheus to store all these metrics. So this is the important bit. You don't have to touch your application. When you run on Istio, your application is automatically going to get metrics from it, all the HTTP metrics, all the traffic is going through Istio, and you're going to get these metrics for free. You don't have to be Facebook to use this. You can have Istio and all the metrics automatically without touching your application or going to Prometheus. And we're going to use Flagger. Can you say, do you know what Flagger is? No. So... I'm doing Flagger for one year. He's a creator of Flagger. And automates the promotion of canary deployments using Istio to shift the traffic, to control the traffic, and looking at the Prometheus metrics to decide whether your application and the canary version is behaving correctly or not. So it's going to go through the different stages. Updating, saying, add some traffic to canary service, look at the metrics. Is this okay? Add more traffic. It's okay. Add more traffic. It's okay. Add more traffic until you finish the rollout. Of course, this is all configured with Jamel that you know and you love. You need some extra tools, but okay. I mean, with all the dongles you need for MacBooks, one more is okay. This is cheaper than anything that Apple will sell you. So in GenkiSex, you just need to enable the canary section. You have to say canary enabled through. And just recently, this was merged, and new projects will have a lot of this already included. But you have to say what host is your traffic coming in? And you use, because we are using Istio, Istio is going to look at your headers and say, oh, this URL is going to the service. And we use canary analysis so we can say how much, how often do we want to check for the metrics? And how many metrics do we want to fail? How many times do we want the metrics to fail before we roll back? In this case, it's five. And the step weight and mass weight is what are the steps in my canary deployment? So this means 10%, 20%, 30%, 40%, and 50, and then do a full rollout. So you can configure all these things in Jamel, so no problem. And an interesting part is what metrics do I want to check, right? So you can use any metrics that are in Prometheus. In this case, there are two metrics that Istio gives you for free. You don't have to touch your application. You don't have to do Facebook to do canary deployments. So you do request access rate and say that 99% of your requests are successful, non-500 errors, check every 60 seconds, and request duration means that your P99, you want them under 500 milliseconds. So you can use any of these metrics. You can use your business metrics. If you get business metrics into Prometheus, let's say, number of items in the shopping cart, you can use that too. And then you can just profit, right? GenkiSex has this way, this concept of environment, some promotion between environments. So in GenkiSex, you have pull request environments. Every time you create a pull request, you get a new Kubernetes namespace. It gets deployed as the pull request. So you get one for pull requests. You get another one that is stating where master is always deployed. So stating always is updated with master. And then you have a production where you manually promote things. So in this case, I'm saying, take my application, croc hunter Java, version 130 into environment production. And this is going to deploy to production. In GenkiSex, by default, this will just roll update my deployment. When I have the canary enabled through, this is going to do a canary deployment. So how this all looks like, what time is it? Okay, it's 45, right? Okay, I'm going to make sure I have time for the demo. So my traffic is coming through the Istio virtual service. I have two services running primary and canary. Primary is the old version, canary is the new version. Flagger is sitting there looking at the Prometheus metrics and updating the virtual service, sending more traffic to one or another. I'll show you that. And if you don't have as enough Slack channels, you can have another one where you can say, show me how my rollout is happening because everybody wants more Slack channels and more Slack messages. This is how you can do it. Also Microsoft Teams. Also Microsoft Teams. Well, the demo is using Quarkus, if you are interested. Okay, so let me go to the demo. So I have this application, this is a Staggi, no, okay, let's go to production first. This is Istio, my URL is going through Istio. This is version 76. And this is an amazing application. I'm going to do the same demos, sort of the same demos that Victor did, but more fun with animals, okay? Because Victor's demos were really boring. And I'm going to say now, if he's not here, it didn't work. So I can shoot crocodiles with lasers. What could be more fun than that? But of course, there's some people that complain, you know, this is not very environmentally friendly shooting crocodiles with lasers. So we made a change in master. I did a change before coming here. This is version 77. And instead of shooting crocodiles with lasers, I'm throwing fish to the crocodiles. Okay? This will be more politically correct, everything. So we're not getting into trouble. Okay? So what I want to do, I want to roll out this into production. So I'm going to do JX. I can do get applications. I'm going to show you at the top. Okay? So it tells me, Jenkins X tells me Staggi is 77, production is 76. So I'm going to do JX, promote croc hunter Java version, 077 environment production. And what this is going to do is open a Jenkins X, oh, okay, unable to cherry pick. Because I was doing, I was messing with it before. Let me do it just manually. This is the Jenkins X is GitOps. So everything is in Git. If I go to my production environment, it's a GitHub repo, Staging environment is another GitHub repo. So I can go directly to make it faster here and say, no, 77. I want the 77. And it's the same thing. Either I do it from the command line or I can do it here. Let's create a pull request, create a pull request. So this is basically going to go through my promotion build. It's going to make sure that my home charts compile and do some testing there. And the most important thing, I have chat ops. If this is telling me this PR is approved or not, in this case, in the environment we have auto approval. So if this passes the build, this will work. I can do the most interesting things that you can do with GitOps with chat ops is you can run commands. So you can do meow and then you get a picture of a cat. So that's probably the number one feature. But you can do slash test, slash root test. And you can add your own things. So this is running the promotion build. While this is running, I have at the top, I have something getting the version from my application. So you see 76. I have flagger here at the bottom. There's some stack trace. Don't worry. I'll tell you later what happened. And so this is going to be doing this. And more interesting stuff. I have flagger includes a graphana dashboard where I can see on the left hand side, this is my primary. On the right hand side is the canary. So I can see all these metrics compare one to another. So request volume, request duration, CPU usage, memory usage, whatever. So we could also come here and look at things. I thought it was late, but not that much. Okay. So this is still not merged. Let me see. I think I have tecton here. We can look at the pipeline that is building the Helm chart. Okay. There it is. This is still doing the build. While this runs, this is all running on GKE, Kubernetes engine. I have a cluster. Right now it has three machines. That may be where it will take more time. I have auto-scaling enabled. So it was going to go up and down depending on the number of things that I'm running before I was, last time I checked it was four nodes. So I'm guessing it may need to go up to four before to the build. But we'll see. What else can I show you? Okay. This is still building. We can take a look at the pods. JX production. Okay. We have still the primary one running. It hasn't created the other one yet. One advice. If you want to be a 7x engineer, you can alias Qubectl to K. Then you save typing seven times later. There's a project that will create aliases for use. When I do KGPO, that means Qubectl gets pods. So I'm saving a lot of time. You just need to remember all of them. Okay. So this is still working. Is this merged? Okay. This is merged. So it's got a, okay. This is doing now the deployment. And what else can I show you? I can see here also what workloads I have running on my cluster. I have Istio. I have Junkin. And here are my applications. Production, staging, production. These are the ones that I'm deployed with Junkin. And this is this. Let's take a look at the pods again. Still not running. What's the trigger point? Trigger point. Oh, that's what is querying my application. The cool request that I'm showing is just hitting. Because that's one of the problems. If you do canary deployments and you don't have traffic, then it's not going to work very well because there's not going to be any metrics. I have something that is basically just doing curls against this thing. Okay. Now it's doing something. Isn't it? Ready? Zero of one. Yes. Okay. It has started. Why is this what I'm doing here? Let's take a look at the Flagger logs. Here it is. So here at the bottom, you don't see it says, new revision detected scaling up my canary. So it has detected that I have deployed a new version. And it has created this call copy for my canary. So I have both canary and the old version running. And this is now starting the pods. Okay. Now I have two pods. Forget about the trigger at the bottom. That's the one I'm doing the close from. I have the canary is the first one. Primary is the old one. So 70 seconds ago, it has started the canary. And eventually, Flagger will start saying I'm sending 10% of the traffic to the new version while this is happening. Yes. So this is running on Quarkus. Quarkus will also allow you to compile to native. So you don't even need the JVM or any libraries in your Docker image if you want to do that. So that's a very interesting project if you want to do small Java images on Kubernetes or Docker. I will tell you to buy my book, but it's not my book. It's Victor's book. But I wrote the chapter on progressive delivery. Let's see. Okay. Here it is. Advancing canary weight 10%. And if you can see here, some requests are getting version 77 while some requests are getting 76. So one in 10 should get the new version. If I go to my production one and I refresh enough times without caching, that's the other thing. There's caching involved, browser caching, things like that. Okay. One in 10. 76. 76. 76. Maybe we should wait until this 20%. Yeah, that's the other option. It should work. Should. Just timing. Okay. I'm not a chosen one. Okay. Let's do this. Network. Disable cache. Anyway, 77. Okay. So I have my version in production. Some people are getting this version. Some people are getting the old version. And remember, you don't have to be Facebook to do canary deployments. Anybody can do it. So I'll take two questions before finally doing things. Yes. Yes, you can do with Istio. You can do all sort of things that you want. You can say HTTP headers. You can say IP location should work too. I think cookies, everything. Yes. Well, what happens when the rollback happens, everybody gets the old version anyway. We stop looking at the headers or anything. Actually, it kills the pod with the new version. So there's no new version to go to. So everything goes back to the old version. Yeah. Yes. Somebody else. Yes. That depends. That's the big concern I have about progressive delivery and rollouts in general. You fix one problem. Now you have a new and different problem. More interesting problem. So what happens when you have multiple commits to master, multiple things going on, multiple feature flags, whatever. Now you need some sort of dashboard that allows you to see how am I progressing things, where are the different things happening, stuff like that. So that becomes the next challenge. I'm going to note that nobody asked about databases because Victor already answered about that because that's the typical question. How do you do this with canary deployments with databases? So make sure that your application is able to live with two versions. So what typically people do is you have the breaking changes you make them with enough time. So if your application can only talk to this version of the database, you make your application now talk to two versions of the database and you deploy your application and then later on you change your database and do things like this. So one thing I want you to get out here and remember is like you don't have to be Facebook to do canary deployments. I don't know if I mentioned that before. And you can use, all these tools are available there. It's all open source. You can use Jenkins X, you can use SysT, you can use Flagger, you can use Prometheus, you can use something else but I hope you give it a try and if it works for you, good. Thank you.
 Yes. Okay. Welcome. Perfect timing. Well, probably you're here because you think continuous  delivery is a good thing to do, right? If not, you're probably in the wrong room. And I'm  going to start with a story, right? Some years ago, we were doing all this automation. We  were deploying with Puppet across multiple machines everywhere. And then you're doing  this continuous delivery. Every time we committed some changes to Puppet, then this would get  deployed until you get to a point where I made a change and these broke the machines.  We do some testing, but this actually broke the machines in a way that not only they got  broken, but it prevented new updates to get into the machines, right? So this is an example  of how hard it is to do continuous delivery right. So my name is Carlos Anchez. I work  at Adobe. As Victor said, the important part is the Twitter. See, Anchez, if you want to  say good things, if you want to say bad things, that's not the right one. It's a different  one. And we're here to talk to you about progressive delivery, which is a term that  was coined a few years ago, just less than two years ago. It started on Launch Starly.  This is a company that do feature flags. I'll talk about that. So they started with a term  progressive delivery. Then Redmond, the analysts, James Governon started also talking about progressive  delivery. And I was like, I'm totally still in this term. I love it. It's a subtly what  it means. It gives you the right mindset about what we are trying to achieve. So it's a term  that tries to encompass all these technologies that are not new. This is very important.  This is not new technologies. It's just they're being around for a while. But it tries to avoid  the problems with these all or nothing deployment capabilities or strategies. And basically,  new versions that you deploy, they are not replacing existing versions, but you run them  in parallel for a period of time in production. This is the important part. You get production  traffic and you do evaluation of metrics to ensure that your deployment is good or not  good. So until you do that, so you keep getting more traffic, get the metrics, look at the  metrics, decide whether the deployment is good or not. And continuous delivery is hard.  And progressive delivery is trying to make continuous delivery easier to adopt by reducing  the risk. This is the key part. We are trying to reduce the risk. Nobody wants to get to  waking up in the middle of the night because of deployment is going wrong. Nobody wants  to decide, like, do not deploy on Fridays. Do not deploy before a holiday because then  we start out. Do not deploy on Fridays. Then, well, maybe you shouldn't deploy on Thursdays  either. So it's a bad game that you would want to get in. So reducing risk, but avoiding  downtime and limiting the blast radius. So if something is wrong, then you have to make  sure that it affects the less amount of users as possible. But it also gives you another  benefit, which is the shorter time since you decide to do something until your production  users can see some of your production users. So some of the techniques, if you've been  to Victor's talk, this is more interesting because I have a Spanish accent, as he said.  So some things that he also covered for those of you. I knew some people left the room and  new people came in after waiting for their probably for an hour or two. So rolling updates,  this is something that Kubernetes does natively. You can do a Qtl command. I mean, here it  is in Kubernetes. There's pretty much a lot of people. So every time, this is not different  from doing rollout updates on VMs. It's just doing it with containers with pods in Kubernetes.  But it's natively included in Kubernetes. It's very easy to do. You deploy a new version  and then you get new pods in with a new version and it starts taking old pods with the old  version from the pool that served your requests to get your traffic. So every time you keep  going one pod, two pods, three pods and so on, same thing we used to do with VMs back  in the day, not much different other than it's more automated.  Blue green deployment, what did you learn about blue green deployment from Victor's?  Don't do it. Okay. So if you're doing blue green deployment, basically means having two  instances of your production clusters or production hardware or infrastructure or whatever and  having something in front of them and then you can quickly switch from one to another.  So when you are ready to roll the new release, you point to the new one. If there's any problem,  you just go to your DNS load balancer, whatever it is that you have in front and you point  to the back. So you have immediate rollback, but all your users are getting the new version.  So if there's something wrong, all your users are going to see it. Canary deployments is  a specialized way, it's a better way of doing things, I guess, where you get a percentage  of your users that get the new version. So you start sending a 1%, 2%, 10%, 20%, whatever  is the number of your users to the new version. This is something that Facebook is doing,  Netflix, I mean all the big companies are doing, but you don't have to be Facebook to  do canary deployments and I'll show you that later. And you can select the users based  on multiple options, right? You can just say randomly, send some percentage of users to  the new version or you could say people from, what a lot of companies do is employees get  the new version first before any public user gets it. So you get employees first, then  you can use HTTP headers, IP location, anything you want. And another common trend that I've  seen is like people, I think Facebook does this, they roll it out first to New Zealand.  So for whatever reason, New Zealand has the statistic population very similar to the US,  so they roll it out in New Zealand, they get the new versions first before it gets rolled  to the US or the rest of the world. So I guess New Zealanders are the guinea pigs here, but  that's something that you can do. Another option in progressive delivery is feature  flex. This is something that LaunchDarly and other companies are doing, where this is very  interesting because it allows you to do several things. One is exposing new features to specific  people. So you could say, you could ship something to production disabled by default, and then  you could look and use either HTTP header also or location base or a flag and your, I  don't know, URL or something. And then you could get the new features, so you could see  how that behaves in production. You could enable that feature for specific users, for  a percentage of users. And it gives you also the speed because if you're familiar with  the DevOps report from Dora, the companies, the teams that deploy faster to production  or to master, merge things into master faster or commit directly to master or have short  leaf branches are the most productive teams. So feature flex allows you to continuously  get things into master without exposing users to those things. So let's say you have a feature  that it takes you two months to do, you don't want to have a feature branch for two months  because that's going to be a horrible idea. It's a horrible idea. Don't do that. And feature  flex allows you to disable that, commit to master and at some point, either starting  trying it with users, iterate, fix things, start showing to more users and so on. This  is the other way you can do deployments. I don't recommend it. Containers don't really  fit in. This is actually a picture. How did the deployment, they unloaded containers in  some island in the Caribbean before and that's how a lot of companies look to some days,  but just don't do it. And a very important part of progressive delivery is monitoring  and I like to say monitoring is the new testing because with testing, you're basically just  checking things that you think may fail, right? You have to think about, okay, what can fail?  I'm going to write some tests to ensure that it doesn't fail. Monitoring will tell you  how things fail that you didn't expect to fail, right? That's the only way you can know about  those things. And it's the only way you're going to find out how your users, because  your users are always going to break your application in very new and innovative ways.  So this is the way you catch that. And it also allows you to have a reaction to these  issues automatically. So you have some data coming in, you have real data. So you know,  this saying if all we have is opinions, let's go with mine, unless we have data, let's go  with the data, right? So this will give you the actual data where you can do things based  on that data. And the bad part is like progressive delivery requires a good number of metrics.  You need to instrument your application, you need to know what means success, what means  failure. These metrics, it's not only HTTP metrics or app time or things like that, but  also could be business metrics, right? If at some point you have a metric that says how  many items are people adding to their shopping cart, right? You're saying this metric going,  well, okay. And at some point people are not getting things to their shopping cart, then  you may think, okay, there's something wrong with the application. Everything gets served,  but maybe they get an empty page or some page that I'm not getting error from that. So you  have to catch that. And I love this quote from DevOps Para, to make error is human, to  propagate error to serve in an automated way, does DevOps? So actually in my previous  company, I'm not going to name names, and we had to do this demo. The CEO was going  to do this demo at the big conference, and we were running these huge clusters with, it  was like 2000 Jenkins masters running, I don't know, terabytes of memory, CPUs and everything.  And at some point I'm getting on a flight to go to San Francisco to the conference, and  I'm checking my Slack before the flight, and it's like, oh, I deleted the wrong cluster  by mistake. So this was one day before, two days before. So I think if you haven't automatically  destroyed something by mistake, it means that you are not automating enough, okay? You got  to get to this point where you're saying, okay, I did something really bad automatically.  That's the point where you're saying, okay, I mean, automation is good. No blame, no blame  in postmortems, right? So Jenkins X, what does Jenkins X come here? Jenkins X, who is  using Jenkins X? Okay, who is using Jenkins? Who is using Jenkins on Kubernetes? With a  Kubernetes plugin, probably, yeah? Okay, that's something I wrote a few years ago. I'm glad  you used it. Don't blame me, okay? So Jenkins X is the new project. I like to say that Jenkins  X is to Jenkins like Java is to JavaScript. They have nothing in common, okay? Jenkins  X runs natively on Kubernetes. Did I mention Kubernetes in this talk? I don't remember.  And it uses a bunch of technologies that you probably know and some new ones. So it uses  Docker, of course, everything runs on Kubernetes. A scaffold, rough, I'll talk about those. And  Tecton Pro. So Tecton is the pipeline engine in Kubernetes, for Kubernetes. It uses spots  and containers to run pipeline steps. This is a project from the Continuous Delivery  Foundation now. And it uses this, basically, it's a model that you've probably seen with  GitHub Actions and other tools where you have a pipeline, every step runs in a different  container. So you have isolation and you have a scale. You have more pipelines, then you  get more containers and everything scales. There's no central point. It scales as much  as your Kubernetes cluster does. Prowl is a project that comes from the Kubernetes itself  that implements chat ops. So it handles the GitHub webhooks. And then you can talk, you  probably seen this now, it's very popular where you got a pull request and the people  start slash something, slash something and then slash test and the test shows you the  test results, slash retest and you can rerun things. So chat ops. Helm is the package manager  for Kubernetes. De facto, standard. Scaffold to build Docker images. This approach is from  Google that allows you to do typical Docker builds but also has support for other backends  to the Docker image builds. Genki says it allows you to use Canico. I like Canico a lot. It  allows you to do builds without the Docker demons. So it's safe to use in Kubernetes  or at least safer. And you don't have to open your cluster to everybody to have access to  the Docker demo. And also, if you are a maven user, you can use, I'm a long time member,  well I committed to maven for a long time. Gip optimizes, you can have a maven plugin  and you can run NVIDIA and something and then it will build a Docker image for you and it  will optimize the layers with your dependencies. So when maven downloads the internet, it gets  optimized with different layers for the Docker images. So when you change the dependencies,  it's not having to rebuild everything from scratch. It caches things. And draft is a  project from Microsoft that generates the Docker file and helps you to do a project.  This is something that Genki uses when you import a project into Genki's X or when you  generate a new project, it will basically create the side, oh, you have a Java project  so I'm going to create you a Docker file and help charts using the Docker image for the  open JDK or other open JDK and it will generate all this scaffolding for you. What I'm talking  about Genki's X because I'm going to show you how to do progressive delivery with Genki's  X. There's a web page that goes through all the details if you want to try it yourself.  I'm going to show you a demo. It uses Istio. I mean, who's using Istio? Here? Some people.  Are you happy? Yes? Okay. Probably because you don't have to maintain it, right? You  don't update it, right? That's it. Don't touch it. Okay. So what I'm going to use from Istio  is it's a service match. It provides you all these cool things, allows you a connection  between services, allows you to do things like say, oh, I want these services to go encrypted,  so securing connection between services, allows you to control the traffic. This is what we're  going to use for canary deployments. We want to control what traffic goes to what service,  the old version, the new version, and it also allows you to do observation of your application  or your service without having to touch your application. So this is what we're going to  do to get the metrics. Istio uses Prometheus to store all these metrics. So this is the  important bit. You don't have to touch your application. When you run on Istio, your application  is automatically going to get metrics from it, all the HTTP metrics, all the traffic  is going through Istio, and you're going to get these metrics for free. You don't have  to be Facebook to use this. You can have Istio and all the metrics automatically without  touching your application or going to Prometheus. And we're going to use Flagger. Can you say,  do you know what Flagger is? No. So... I'm doing Flagger for one year. He's a creator  of Flagger. And automates the promotion of canary deployments using Istio to shift the  traffic, to control the traffic, and looking at the Prometheus metrics to decide whether  your application and the canary version is behaving correctly or not. So it's going to  go through the different stages. Updating, saying, add some traffic to canary service,  look at the metrics. Is this okay? Add more traffic. It's okay. Add more traffic. It's  okay. Add more traffic until you finish the rollout. Of course, this is all configured  with Jamel that you know and you love. You need some extra tools, but okay. I mean, with  all the dongles you need for MacBooks, one more is okay. This is cheaper than anything  that Apple will sell you. So in GenkiSex, you just need to enable the canary section.  You have to say canary enabled through. And just recently, this was merged, and new projects  will have a lot of this already included. But you have to say what host is your traffic  coming in? And you use, because we are using Istio, Istio is going to look at your headers  and say, oh, this URL is going to the service. And we use canary analysis so we can say how  much, how often do we want to check for the metrics? And how many metrics do we want to  fail? How many times do we want the metrics to fail before we roll back? In this case,  it's five. And the step weight and mass weight is what are the steps in my canary deployment?  So this means 10%, 20%, 30%, 40%, and 50, and then do a full rollout. So you can configure  all these things in Jamel, so no problem. And an interesting part is what metrics do  I want to check, right? So you can use any metrics that are in Prometheus. In this case,  there are two metrics that Istio gives you for free. You don't have to touch your application.  You don't have to do Facebook to do canary deployments. So you do request access rate  and say that 99% of your requests are successful, non-500 errors, check every 60 seconds, and  request duration means that your P99, you want them under 500 milliseconds. So you can use  any of these metrics. You can use your business metrics. If you get business metrics into  Prometheus, let's say, number of items in the shopping cart, you can use that too. And  then you can just profit, right? GenkiSex has this way, this concept of environment, some  promotion between environments. So in GenkiSex, you have pull request environments. Every  time you create a pull request, you get a new Kubernetes namespace. It gets deployed  as the pull request. So you get one for pull requests. You get another one that is stating  where master is always deployed. So stating always is updated with master. And then you  have a production where you manually promote things. So in this case, I'm saying, take  my application, croc hunter Java, version 130 into environment production. And this  is going to deploy to production. In GenkiSex, by default, this will just roll update my  deployment. When I have the canary enabled through, this is going to do a canary deployment.  So how this all looks like, what time is it? Okay, it's 45, right? Okay, I'm going to make  sure I have time for the demo. So my traffic is coming through the Istio virtual service.  I have two services running primary and canary. Primary is the old version, canary is the  new version. Flagger is sitting there looking at the Prometheus metrics and updating the  virtual service, sending more traffic to one or another. I'll show you that. And if you  don't have as enough Slack channels, you can have another one where you can say, show me  how my rollout is happening because everybody wants more Slack channels and more Slack messages.  This is how you can do it.  Also Microsoft Teams. Also Microsoft Teams. Well, the demo is using Quarkus, if you are  interested. Okay, so let me go to the demo. So I have this application, this is a Staggi,  no, okay, let's go to production first. This is Istio, my URL is going through Istio. This  is version 76. And this is an amazing application. I'm going to do the same demos, sort of the  same demos that Victor did, but more fun with animals, okay? Because Victor's demos were  really boring. And I'm going to say now, if he's not here, it didn't work. So I can shoot  crocodiles with lasers. What could be more fun than that? But of course, there's some  people that complain, you know, this is not very environmentally friendly shooting crocodiles  with lasers. So we made a change in master. I did a change before coming here. This is  version 77. And instead of shooting crocodiles with lasers, I'm throwing fish to the crocodiles.  Okay? This will be more politically correct, everything. So we're not getting into trouble.  Okay? So what I want to do, I want to roll out this into production. So I'm going to  do JX. I can do get applications. I'm going to show you at the top. Okay? So it tells  me, Jenkins X tells me Staggi is 77, production is 76. So I'm going to do JX, promote croc  hunter Java version, 077 environment production. And what this is going to do is open a Jenkins  X, oh, okay, unable to cherry pick. Because I was doing, I was messing with it before.  Let me do it just manually. This is the Jenkins X is GitOps. So everything is in Git. If I  go to my production environment, it's a GitHub repo, Staging environment is another GitHub  repo. So I can go directly to make it faster here and say, no, 77. I want the 77. And it's  the same thing. Either I do it from the command line or I can do it here. Let's create a pull  request, create a pull request. So this is basically going to go through my promotion  build. It's going to make sure that my home charts compile and do some testing there.  And the most important thing, I have chat ops. If this is telling me this PR is approved  or not, in this case, in the environment we have auto approval. So if this passes the  build, this will work. I can do the most interesting things that you can do with GitOps with chat  ops is you can run commands. So you can do meow and then you get a picture of a cat. So  that's probably the number one feature. But you can do slash test, slash root test. And  you can add your own things. So this is running the promotion build. While this is running,  I have at the top, I have something getting the version from my application. So you see  76. I have flagger here at the bottom. There's some stack trace. Don't worry. I'll tell you  later what happened. And so this is going to be doing this. And more interesting stuff.  I have flagger includes a graphana dashboard where I can see on the left hand side, this  is my primary. On the right hand side is the canary. So I can see all these metrics compare  one to another. So request volume, request duration, CPU usage, memory usage, whatever.  So we could also come here and look at things. I thought it was late, but not that much.  Okay. So this is still not merged. Let me see. I think I have tecton here. We can look at  the pipeline that is building the Helm chart. Okay. There it is. This is still doing the  build. While this runs, this is all running on GKE, Kubernetes engine. I have a cluster.  Right now it has three machines. That may be where it will take more time. I have auto-scaling  enabled. So it was going to go up and down depending on the number of things that I'm  running before I was, last time I checked it was four nodes. So I'm guessing it may need  to go up to four before to the build. But we'll see. What else can I show you? Okay. This  is still building. We can take a look at the pods. JX production. Okay. We have still the  primary one running. It hasn't created the other one yet. One advice. If you want to  be a 7x engineer, you can alias Qubectl to K. Then you save typing seven times later.  There's a project that will create aliases for use. When I do KGPO, that means Qubectl  gets pods. So I'm saving a lot of time. You just need to remember all of them. Okay. So  this is still working. Is this merged? Okay. This is merged. So it's got a, okay. This  is doing now the deployment. And what else can I show you? I can see here also what workloads  I have running on my cluster. I have Istio. I have Junkin. And here are my applications.  Production, staging, production. These are the ones that I'm deployed with Junkin. And  this is this. Let's take a look at the pods again. Still not running. What's the trigger  point? Trigger point. Oh, that's what is querying my application. The cool request that I'm  showing is just hitting. Because that's one of the problems. If you do canary deployments  and you don't have traffic, then it's not going to work very well because there's not  going to be any metrics. I have something that is basically just doing curls against  this thing. Okay. Now it's doing something. Isn't it? Ready? Zero of one. Yes. Okay. It  has started. Why is this what I'm doing here? Let's take a look at the Flagger logs. Here  it is. So here at the bottom, you don't see it says, new revision detected scaling up my  canary. So it has detected that I have deployed a new version. And it has created this call  copy for my canary. So I have both canary and the old version running. And this is now  starting the pods. Okay. Now I have two pods. Forget about the trigger at the bottom. That's  the one I'm doing the close from. I have the canary is the first one. Primary is the old  one. So 70 seconds ago, it has started the canary. And eventually, Flagger will start  saying I'm sending 10% of the traffic to the new version while this is happening. Yes.  So this is running on Quarkus. Quarkus will also allow you to compile to native. So you  don't even need the JVM or any libraries in your Docker image if you want to do that. So  that's a very interesting project if you want to do small Java images on Kubernetes or Docker.  I will tell you to buy my book, but it's not my book. It's Victor's book. But I wrote the  chapter on progressive delivery. Let's see. Okay. Here it is. Advancing canary weight  10%. And if you can see here, some requests are getting version 77 while some requests  are getting 76. So one in 10 should get the new version. If I go to my production one  and I refresh enough times without caching, that's the other thing. There's caching involved,  browser caching, things like that. Okay. One in 10. 76. 76. 76. Maybe we should wait until  this 20%. Yeah, that's the other option. It should work. Should. Just timing. Okay. I'm  not a chosen one. Okay. Let's do this. Network. Disable cache. Anyway, 77. Okay. So I have  my version in production. Some people are getting this version. Some people are getting  the old version. And remember, you don't have to be Facebook to do canary deployments. Anybody  can do it. So I'll take two questions before finally doing things. Yes. Yes, you can do  with Istio. You can do all sort of things that you want. You can say HTTP headers. You can  say IP location should work too. I think cookies, everything. Yes. Well, what happens when  the rollback happens, everybody gets the old version anyway. We stop looking at the headers  or anything. Actually, it kills the pod with the new version. So there's no new version  to go to. So everything goes back to the old version. Yeah. Yes. Somebody else. Yes. That  depends. That's the big concern I have about progressive delivery and rollouts in general.  You fix one problem. Now you have a new and different problem. More interesting problem.  So what happens when you have multiple commits to master, multiple things going on, multiple  feature flags, whatever. Now you need some sort of dashboard that allows you to see how  am I progressing things, where are the different things happening, stuff like that. So that  becomes the next challenge. I'm going to note that nobody asked about databases because  Victor already answered about that because that's the typical question. How do you do  this with canary deployments with databases? So make sure that your application is able  to live with two versions. So what typically people do is you have the breaking changes  you make them with enough time. So if your application can only talk to this version  of the database, you make your application now talk to two versions of the database  and you deploy your application and then later on you change your database and do things  like this. So one thing I want you to get out here and remember is like you don't have  to be Facebook to do canary deployments. I don't know if I mentioned that before. And  you can use, all these tools are available there. It's all open source. You can use Jenkins  X, you can use SysT, you can use Flagger, you can use Prometheus, you can use something  else but I hope you give it a try and if it works for you, good. Thank you.
progressive_delivery_0_359.png progressive_delivery_360_2003.png progressive_delivery_2004_5171.png progressive_delivery_5172_8939.png progressive_delivery_8940_14363.png progressive_delivery_14364_18515.png progressive_delivery_18516_20579.png progressive_delivery_20580_22079.png progressive_delivery_22080_26459.png progressive_delivery_26460_28847.png progressive_delivery_28848_32903.png progressive_delivery_32904_33707.png progressive_delivery_33708_34259.png progressive_delivery_34260_34439.png progressive_delivery_34440_34487.png progressive_delivery_34488_34547.png progressive_delivery_34548_35207.png progressive_delivery_35208_35339.png progressive_delivery_35340_35915.png progressive_delivery_35916_35975.png progressive_delivery_35976_36095.png progressive_delivery_36096_36275.png progressive_delivery_36276_36959.png progressive_delivery_36960_37271.png progressive_delivery_37272_38063.png progressive_delivery_38064_38267.png progressive_delivery_38268_38687.png progressive_delivery_38688_38867.png progressive_delivery_38868_39263.png progressive_delivery_39264_39311.png progressive_delivery_39312_40091.png progressive_delivery_40092_40811.png progressive_delivery_40812_41363.png progressive_delivery_41364_42383.png progressive_delivery_42384_42791.png progressive_delivery_42792_43775.png progressive_delivery_43776_43859.png progressive_delivery_43860_43907.png progressive_delivery_43908_43979.png progressive_delivery_43980_44159.png progressive_delivery_44160_44759.png progressive_delivery_44760_44915.png progressive_delivery_44916_47747.png progressive_delivery_47748_48563.png progressive_delivery_48564_50855.png progressive_delivery_50856_55523.png progressive_delivery_55524_55646.png