Giving your Heroku App highly-available PostgreSQL
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Varghese, Jake
As you’re building out your app, you know how important it is to have your database up at all times. You need your PostgreSQL database to be HA. Jake from Compose is going to show you how to migrate your data to a HA PG service in less than 20 minutes. This is a sponsored talk by Compose, an IBM company
 My name is Jake. I'm with Compose and today I'm going to show you how to migrate your Heroku app over to IBM's cloud HA post-gray sequel. Before we talk about how to do that, I want to talk about how we actually attain high availability. And the way we do that is with an app we created called Governor. Governor sits on top of SCD, which is our distribution coordinator. SCD is connected to two post-gray sequel nodes in leader follower format. SCD is the one that's maintaining its leader and follower format. And we use Wall Street and Replication to keep the nodes in sync. In front of the two nodes, we have an HA proxy load balancer. This load balancer is connected to both nodes and to the governor. So it knows which one is the leader and which one is the follower. If any changes happen to the nodes, the proxy will hold the request until the new leader is shown. So your app won't have any idea that a failover just happened. That's how we keep you with zero downtime. Your app actually connects to the HA proxy node. And so we released this into open source a while ago and it's been forked a couple of times and one of the newest forks that came out is Petroni. Petroni adds to governor by allowing you to use ZooKeeper and Console as your distribution coordinator. And it's very stable and quite good. Now we actually use Petroni for all our PG deployments. So that's how we actually attain high availability. So let me show you the demo that we created to migrate over. This demo that I created is an artificial intelligence that's totally real and not made up at all. And what this intelligence does is it will take an image and stick it into one of four Star Trek captain posters. So it takes your image and puts it in there. So let's take a look at what this looks like. So I'll go ahead and upload an image here. Bam. That's Picard as me, obviously. I'm more of a Cisco guy, but this one's not that bad. That one is way better. I'm going to keep that one as my wallpaper, actually. And then obviously we have the other two captains, Kirk, I'm not that fond of. Okay. I do like the hairline, though. It's a lot nicer. But then we go to Janeway and clearly the AI needs tweaks. So that's our app. Luckily we don't have to do anything about the tweak in the AI for what we want to show you today. So for today, we're going to migrate over the data from production, captainly, over to our compose cloud, PostgreSQL. So the way we're going to do that is first thing we're going to do is pull a backup from Heroku and capture that down. So go ahead and capture that and then we'll download it. Hopefully it doesn't take too long. The awesome images that I showed you before? No, that's totally real. It was made by an artificial intelligence, for sure. And not by one of our awesome designers. All right. So now we'll pull the download here real quick. And so now we got the export downloaded. And I'm going to take you over to our Postgre configuration details page. Let me get there real quick. So here's the details page for the Postgre deployment that we set up under IBM cloud. Here's the connection string for you. One thing I would like to point out, our deployments come with the entire server. So you can put as many databases as you want to on it. We load all the PG deployments with a default database called compose. This has a couple of things in it, like your admin and some other things in it. And then I've already created the captainly staging and the captainly production databases. So we'll go ahead and use PG restore to restore that. So right here we have the standard PG restore, a couple of options for no owners. Here's the host from the connection string. All of our deployments use nonstandard ports for security reasons. So you're going to need to use the minus P option. Then you'll have the admin as the username. And then we're going to import into captainly staging and the dump that we want to import into. Let me go ahead and copy and paste the password because it's going to ask me for that. Cool. So now that's importing. One thing we're going to need to do on staging is we're going to need to drop the Heroku Postgre add on. The reason for this is because the database underscore URL config variable is automatically attached to any add on that uses it. So you can't actually reconfigure it with the add on. So we have to remove the add on and we'll go ahead and remove that real quick. Cool. Now let's double check that. All right, so database URL is gone. Now we can go ahead and set the database URL. Then everyone can see how badly I type. And we'll use basically the exact same connection string except we're going to change this last part to captainly staging. And let's see here. All right, still processing the data almost done. Cool. We're on the sequence. So now we can go ahead and set this. Oh, I need to do set not ST. Cool. So let's double check that. All right, so the database is already set. So let's also go into the console and make sure the connection is up and running. Cool. Import is done. Perfect. So now we got data, everything is connected. So last check, we're going to go to captainly staging and refresh, make sure we get no errors. Perfect. So staging, our staging test is done. So now all we have to do is pull the stuff into production. That's going to be pretty easy since we already did staging. All we have to do is change this part right here to production and let that import. Got to get password. All right, and we'll do the exact same thing with the captainly production Hiroku instance. And we'll remove. Cool. Let's double check that. All right, database URL is gone. Now we can set it. And again, we're going to basically use the exact same thing we did before, except we're going to change minus R to production and we're going to change this to production. Perfect. Now let's go ahead and run the console. How far we got in here? Not too bad. So now let's double check with the console. Perfect. So now we've gotten production pulled over. Now let's just double check one more time in the app. And that's it. We just migrated all the data over to the new deployment. That's all we had to do. Without downtime to which app? Right. You have to do the config variable switch. So that's going to give you some downtime. But filling up the databases, you don't have to switch over at that point. So you can just wait until everything's done and then switch over. I don't know how fast your big Hiroku app will switch over all the config variables to all the servers. It should be zero downtime from what I remember of Hiroku. But I can't do that. Okay. Yeah. So you can. I think you can sync one of our deployments into another deployment. I have to double check on that because the way we set up the configuration, I don't know if that will actually work that well because one will be a leader, one will be a follower, then a coming in from another leader. It might throw everything off. So I'm not too sure about that. But hit me up at the booth and I can ask one of the tech guys when we get there. Okay. Yeah. So this is done. And so you see up here where it says search. Nothing, there's nothing really to search right now in this app. But let's say I wanted to add a search. If it was super simple, I would just scan across the Postgre tables. But if it's anything more difficult than that, I'm probably going to want my own indexer. And you can come into cloud here and just create a resource. And let's say elastic search. And we'll put in US South. Ladies prefer to then we just create. I'm not going to click create right now because I don't want to provision another elastic search. But you would just click create and then it would look something like this once it was done. And this would be the overview of your elastic search deployment, which would have the connection strings for you to put into ES underscore URL or elastic search URL, whatever the config variable is for your app. And yeah. So that's basically all there is to migrating your app over. That's it. So if any of you all have any more questions, let me know. Yeah, that's it for this. So we try to keep it really short since it was sponsored and we don't want to just sell you and pitch everything. So we just decided to show you what it can do. That's it. All right. Thanks, guys.
 My name is Jake. I'm with Compose and today I'm going to show you how to migrate your  Heroku app over to IBM's cloud HA post-gray sequel. Before we talk about how to do that,  I want to talk about how we actually attain high availability. And the way we do that  is with an app we created called Governor. Governor sits on top of SCD, which is our distribution  coordinator. SCD is connected to two post-gray sequel nodes in leader follower format. SCD  is the one that's maintaining its leader and follower format. And we use Wall Street and  Replication to keep the nodes in sync. In front of the two nodes, we have an HA proxy  load balancer. This load balancer is connected to both nodes and to the governor. So it knows  which one is the leader and which one is the follower. If any changes happen to the nodes,  the proxy will hold the request until the new leader is shown. So your app won't have  any idea that a failover just happened. That's how we keep you with zero downtime. Your app  actually connects to the HA proxy node. And so we released this into open source a while  ago and it's been forked a couple of times and one of the newest forks that came out is  Petroni. Petroni adds to governor by allowing you to use ZooKeeper and Console as your distribution  coordinator. And it's very stable and quite good. Now we actually use Petroni for all  our PG deployments. So that's how we actually attain high availability. So let me show you  the demo that we created to migrate over. This demo that I created is an artificial  intelligence that's totally real and not made up at all. And what this intelligence does  is it will take an image and stick it into one of four Star Trek captain posters. So  it takes your image and puts it in there. So let's take a look at what this looks like.  So I'll go ahead and upload an image here. Bam. That's Picard as me, obviously. I'm more  of a Cisco guy, but this one's not that bad. That one is way better. I'm going to keep  that one as my wallpaper, actually. And then obviously we have the other two captains,  Kirk, I'm not that fond of. Okay. I do like the hairline, though. It's a lot nicer. But  then we go to Janeway and clearly the AI needs tweaks. So that's our app. Luckily we don't  have to do anything about the tweak in the AI for what we want to show you today. So  for today, we're going to migrate over the data from production, captainly, over to our  compose cloud, PostgreSQL. So the way we're going to do that is first thing we're going  to do is pull a backup from Heroku and capture that down. So go ahead and capture that and  then we'll download it. Hopefully it doesn't take too long. The awesome images that I showed  you before? No, that's totally real. It was made by an artificial intelligence, for sure.  And not by one of our awesome designers. All right. So now we'll pull the download here  real quick. And so now we got the export downloaded. And I'm going to take you over to our Postgre  configuration details page. Let me get there real quick. So here's the details page for  the Postgre deployment that we set up under IBM cloud. Here's the connection string for  you. One thing I would like to point out, our deployments come with the entire server.  So you can put as many databases as you want to on it. We load all the PG deployments with  a default database called compose. This has a couple of things in it, like your admin  and some other things in it. And then I've already created the captainly staging and  the captainly production databases. So we'll go ahead and use PG restore to restore that.  So right here we have the standard PG restore, a couple of options for no owners. Here's  the host from the connection string. All of our deployments use nonstandard ports for  security reasons. So you're going to need to use the minus P option. Then you'll have  the admin as the username. And then we're going to import into captainly staging and  the dump that we want to import into. Let me go ahead and copy and paste the password  because it's going to ask me for that. Cool. So now that's importing. One thing we're going  to need to do on staging is we're going to need to drop the Heroku Postgre add on. The  reason for this is because the database underscore URL config variable is automatically attached  to any add on that uses it. So you can't actually reconfigure it with the add on. So we have  to remove the add on and we'll go ahead and remove that real quick. Cool. Now let's double  check that. All right, so database URL is gone. Now we can go ahead and set the database  URL. Then everyone can see how badly I type. And we'll use basically the exact same connection  string except we're going to change this last part to captainly staging. And let's see here.  All right, still processing the data almost done. Cool. We're on the sequence. So now  we can go ahead and set this. Oh, I need to do set not ST. Cool. So let's double check  that. All right, so the database is already set. So let's also go into the console and  make sure the connection is up and running. Cool. Import is done. Perfect. So now we got  data, everything is connected. So last check, we're going to go to captainly staging and  refresh, make sure we get no errors. Perfect. So staging, our staging test is done. So now  all we have to do is pull the stuff into production. That's going to be pretty easy since we already  did staging. All we have to do is change this part right here to production and let  that import. Got to get password. All right, and we'll do the exact same thing with the  captainly production Hiroku instance. And we'll remove. Cool. Let's double check that.  All right, database URL is gone. Now we can set it. And again, we're going to basically  use the exact same thing we did before, except we're going to change minus R to production  and we're going to change this to production. Perfect. Now let's go ahead and run the console.  How far we got in here? Not too bad. So now let's double check with the console. Perfect.  So now we've gotten production pulled over. Now let's just double check one more time  in the app. And that's it. We just migrated all the data over to the new deployment. That's  all we had to do. Without downtime to which app? Right. You have to do the config variable  switch. So that's going to give you some downtime. But filling up the databases, you don't have  to switch over at that point. So you can just wait until everything's done and then switch  over. I don't know how fast your big Hiroku app will switch over all the config variables  to all the servers. It should be zero downtime from what I remember of Hiroku. But I can't  do that. Okay. Yeah. So you can. I think you can sync one of our deployments into another  deployment. I have to double check on that because the way we set up the configuration,  I don't know if that will actually work that well because one will be a leader, one will  be a follower, then a coming in from another leader. It might throw everything off. So  I'm not too sure about that. But hit me up at the booth and I can ask one of the tech  guys when we get there. Okay. Yeah. So this is done. And so you see up here where it says  search. Nothing, there's nothing really to search right now in this app. But let's say  I wanted to add a search. If it was super simple, I would just scan across the Postgre  tables. But if it's anything more difficult than that, I'm probably going to want my  own indexer. And you can come into cloud here and just create a resource. And let's say  elastic search. And we'll put in US South. Ladies prefer to then we just create. I'm not  going to click create right now because I don't want to provision another elastic search.  But you would just click create and then it would look something like this once it was  done. And this would be the overview of your elastic search deployment, which would have  the connection strings for you to put into ES underscore URL or elastic search URL, whatever  the config variable is for your app. And yeah. So that's basically all there is to migrating  your app over. That's it. So if any of you all have any more questions, let me know.  Yeah, that's it for this. So we try to keep it really short since it was sponsored and  we don't want to just sell you and pitch everything. So we just decided to show you what it can  do. That's it. All right. Thanks, guys.
Giving_your_Heroku_0_4439.png Giving_your_Heroku_4440_5051.png Giving_your_Heroku_5052_6767.png Giving_your_Heroku_6768_7007.png Giving_your_Heroku_7008_7139.png Giving_your_Heroku_7140_7199.png Giving_your_Heroku_7200_7247.png Giving_your_Heroku_7248_7343.png Giving_your_Heroku_7344_8231.png Giving_your_Heroku_8232_10871.png Giving_your_Heroku_10872_16631.png Giving_your_Heroku_16632_16763.png Giving_your_Heroku_16764_16811.png Giving_your_Heroku_16812_16878.png