Delivering Autonomous Rails Apps behind Corporate Firewalls
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Merwin, Nick
When a Fortune 500 company wants to use your app but can't risk sharing sensitive data in the cloud, you'll need to package and deploy an autonomous version of it behind their firewall (aka the Fog). We’ll explore some methods to make this possible including standalone VM provisioning, codebase security, encrypted package distribution, seat based licensing and code updates.
 Thanks for coming out late Thursday, guys. It's been a long couple days. I'm Nick Merwin, a founder of coveralls.io, and hope you guys have seen some of our badges around town, GitHub, Readme, et cetera. I don't know. We're kind of all over the place hopefully right now. But today we're going to discuss how it went about figuring out how to get coveralls, a SaaS app, into the hands of customers that wanted to run it within their corporate networks. It was pretty untreated territory for me, but I hope this general overview and what we'll see later with just a really basic prototype will help you guys get some confidence that you could also take your app to the next level by offering a hosted version to a customer base that maybe you never thought was possible. So in that sense, it's kind of going to be partially a business model discussion and some just general sysadmining stuff, and then of course all kind of viewed through the lens of Rails since coveralls is a Rails app. All right, so why would you want to set your app free into the dark world out there outside of your cushy deployment environment? So currently let's say your app is hosted on Heroku or VPS. Coveralls is on DigitalOcean. And if you have a subscription-based service where users are paying by the month and maybe there's a few tiers and usage-based add-ons, and it's coming along the crewing users and it seems like your potential customer base is covered, but you might be neglecting an even more lucrative and even preferable customer base, which would be the enterprise, quote-unquote. And this could be monolithic companies where dev teams have a much harder time getting upper management to embrace cloud-based tools, or perhaps the issue is just that setting up a $5-a-month service fee takes a mountain of vendor approval paperwork. So let's chat about how it can make sense to offer a hosted version of your app in addition to your cloud service. So there are three kind of tenants that I ran across in figuring this out, or as to the why you would want to do this. And the first one is security, and it's pretty obvious that we all worry that when you use a cloud app, you essentially give up control of your data, and you assume that the devs on the other side hopefully use Bcrypt on your password, but you never really know for sure and just hope there's never a real data breach. But beyond passwords, it could be any sort of data that your company's security department has forbidden from leaving their networks. And in fact, it could preclude the possibility of them ever using your app. This could be maybe a medical company that wants to use your data mining tool that can't upload anything to the cloud due to HIPAA compliance issues, or maybe it's a dev team that wants to use your source code analysis tools but can't risk sharing or leaking proprietary code. There's also just general business apps like CRM's, internal tasking, chatting, scheduling that would have plenty of info that those companies wouldn't want to share with the competitor. But by running a hosted version of your app within their networks, they can just be more confident that they won't ever have their accounts hacked or their data compromised out on the open internet. So second one is a bit more esoteric. It's reliability, and this is just kind of a worst case thing. If your app that you're giving them is a DevOps tool that's part of the build or deployment pipeline, then they're not going to want there to be any unplanned maintenance downtime. So for instance, with GitHub or Travis, if something goes down, something goes wrong, you don't have control of it, and it blocks your pushes or your builds that could screw up your deployment. And so if you're running it internally, then your dev teams and your sysadmins will always be able to coordinate to make sure everything's good to go when it comes to crunch time. The lastly is cost, and the traditional subscription model where it's just pay by the month doesn't really work when you've given over your app to a customer. So oftentimes those potential customers are also less price sensitive and could be willing to pay a premium for your host of service and some sort of extra white glove support service. But once you deliver the app to them, you have to assume that it won't be able to phone home to check the subscription status. So instead of a time-based subscription, like a monthly fee, you probably are going to need to do some sort of seat-based usage or something else that's beyond just a monthly or perhaps yearly. And then once you're thinking about seats, then you can do seat packs as tiers, as in the number of active users that can access the app as it's living within their network. You could also build some sort of self-destruct or shut-down mechanism that will force the customer to purchase or renew the license after a set amount of time, like six months or a year, just to make sure that they come back and they keep a valid license. So there's a lot of different opportunity to work out business model kinks beyond just a traditional SaaS model. So before we go into the how we figured this out, let me just mention Coveralls and what it does and why we took it to the enterprise. So Coveralls, really quick. Coveralls is a tool for code cover tracking and notifications. We can annotate your pull requests with successor failure statuses based on changes in your coverage percentage or just send messages to your chat or email with updates. This helps dev teams make sure they don't deliver untested code, especially in a production. And on site you can see line by line coverage of the code base, the cloud version for open sources free, but we charge for private repos. And the rub comes because we don't, while we don't actually store source on our servers, we do store private scoped OAuth tokens for GitHub. And so for some companies, that's just a non-starter. We can't have, they can't allow us to see within their private repos. And then some of them are already using GitHub Enterprise, so they're not using Cloud GitHub to start out with, and the two are not interoperable. So what we did is gathered interest from a potential customer base over a matter of months, and it became clear that there was enough of an interest that we should try to do this and try to make it happen. And so that's what brought us here. So there are some pretty big main hurdles that I found in converting your general cloud Rails app to a fog one where it's living within, below the cloud layer. So delivery and installation, let's talk about delivery first. The best user experience for your customers would be to have the least amount of work to get your app up and running in the environment, obviously. So three download files would be, I think, is the most consolidated way that we can achieve this. And that's first the virtual machine that the app is going to run on, and that could just be a Linux box, as I'll show you, a virtual Linux box. And that's typically between 800 and a gigabyte size download. The next one would be the packaged up app files itself, and that's a decently high number, just between 50 and 100 or more, depending on how many gems you have, because all the gems need to be bundled and vended into the app package itself, as we'll see. And then lastly, a license file that's specific to the customer and is generated for every customer. And so out of those three, your customers would probably only need to download the VM, like else in a blue moon when you do big upgrades to the underlying system or have new dependencies. The app package is something that they would download whenever there's updates, and we'll get to that in the license file would be around the same amount quicker, like when they go to trial to active or what. So actually, let's talk about those incremental updates for a sec. With the standard deployment, obviously, when your code is hosted up on Heroku, getting features and fixes out to your users is as easy as getting pushed, not so in this case when your app is running completely out somewhere in the wild where you have no access to it. So that's where the app packages come in. It's a smaller download that has just the most up-to-date bug fixes, features, et cetera, that is a much quicker download than having to re-download and configure an entire virtual machine. So also for the license files, one last thing, it makes sense that this is the part where you'll probably need to build a secondary app in addition to your main Cloud one that will just service those customers and charge them subscription fees and have trial accounts and all that stuff. So that's where the download license file and then download the other two in the secondary app. All right. So installation-wise, the networking config is the first thing that they're going to see and since they're not, you don't want them to have to log into the VM themselves. The best way to get around this is to build a tiny little menu-driven Ruby app that gets presented on boot and I'll show you guys how we did this. And that way, the customer won't need to know any specifics about Linux networking. In addition to that, external access just has to be assumed as not going to be a possibility. So hopefully if they're, the reason why they're purchasing this hosted version is that so they can be completely confident in the security behind it. They won't give it any external access, you know, that'll be totally walled off in a sandbox in their network, hopefully. So the assumption of there not being any way to access it from the outside is probably valid. Last one is just process management and that's how are we going to get background jobs and the server itself to stay alive and start at boot. All right. So more of these hurdles. So product support, generally you can see immediately when a user hits a 500 error, it's going to, you're going to get an email and you're going to be able to go on to your error tracking service, air brake, whatever, and take a look. In this case, there's no way to know when that happens. There's no way to get pinged. So you're just going to get an email from your customer saying something's not right, what's going on. So you need to provide them a way to be able to send you details about the exception and so we're going to look at how to just keep those exceptions somewhere that we can then send over or let your customer send to you in like a text to an email or something. So that goes same for logging. If something's really funky on it, you're going to be able to see if there's any weird parameters coming through the routes, especially when they're not causing an exception per say. Lastly there's resource management. If it seems like things are running slowly on their site, there needs to be some way to address that. I'm not going to go over that in this talk because I feel like that's more of a sysadmin-ing issue, but that's something for homework, I guess. So lastly is the intellectual property question and this is a big one. And because you're giving them a VM, they can unpack the VM and mount the disk and look at everything you put there. So even though it comes in one nice file, it's still extractable and you can just load up another VM in virtual box of VMware and then mount the disk that you had delivered them. So that means they're going to be able to look at everything as though they were a root user. So I feel like database access, unless, I don't know, there might be some way to get around this, but you should just assume that they're going to have access to the database itself and it doesn't really matter. I don't know. That's the bigger question here, along with code obfuscation. Since Ruby, there's no way to really, really fully protect your source code once it's being executed in somebody else's environment and it has to actually work itself, you can write a code obfuscator as a deterrent and that's what we've done for CoverAll's Enterprise. But I think, I mean, that prevents people from immediately reading it if they just mount your disk and look through your source file tree. But it's probably safest to just cover this with your license and say something like, the license you're purchasing from us covers only the use of the software modification or redistribution or not permitted. So it just kind of comes down to legalese as being your last line of defense here. And if this keeps you up at night, then maybe the best thing to remember is that your customers for your app should be most interested in getting updates, getting bug fixes, new features and support rather than breaking into your source code, reading it, copying it, spreading out on the Internet. So that could be its own talk about how to actually achieve some level of code obfuscation, but we're not going to go over that really today. Okay. So let's get into the nitty-gritty here. So this is a pretty hectic diagram, but it's basically showing the general architecture of how the app is running within our VM, which is Ubuntu. And we chose Ubuntu because it's just widely used and relatively easy to configure. So the main components that live within it are the network config app, which is the first thing that customers will see when you boot it up. And that lets you do things like select static networking or DHCP and then set the name servers and then reboot, shut down, et cetera. So it's just kind of a simple little starter app. And then behind our web server, we'll go into why we chose Passenger for it. There's going to be a reinstalled app that I just made in Sinatra. It's a really simple app that facilitates the unpacking of the package file that contains the actual app or the app updates and the license file. And so within that, there's, within behind the web server, we'll live your Rails app also. So the Rails app itself, some of the things that these three components I think are the main ones that set it apart from your standard cloud hosted app or where you started from to begin with to get it to this point. So the license file reader is going to read and cache your license at the boot when your Rails app boots and then it's checked on every page view. And then you can do things like enable the disabled features or lock the app down. Second one is data import and export because when you need to download a brand new upgraded virtual machine, say it's got new dependencies for new features, then you're going to need to give your customers a way to get their data out, the old one, and get it in the new one. So that's things like dumping the database. If your app has uploads, then you're going to need to pull those together if the uploads are living on the VM. And then anything else that the users might have uploaded or changed that has a state on the VM. And the third one is support package generation and downloads. So we'll show how we can use the Rails rescue from in the most general terms just to collect all the exceptions that happen in controllers at least. And they can be archived into a temp directory and then when an admin for the app as it's living on the VM is ready to or needs some support, they can hit a link that generates a archived file that we can encrypt also and then email it or dropbox it over to us. Okay, so setting up the environment first. For development, I think it's probably easiest to use virtual box. It's free and pretty simple and it's really easy to export and appliance from your machine called an OVA file and that's just a nice little, it's a tar or zip or something with a.OVA extension. That's just something easy you can link to and serve from your CDN. Also Ubuntu, just because of its, how widely it's used, hopefully we can trust its built in security settings and standards. So we kind of get that out of the box and our customers get that confidence out of the box. So when you provision it for your Rails app, you obviously want to use the minimum amount of dependencies and touches to it so that you can just keep the download size smallest. If you don't need to install something like ImageMagic, then just don't do it. There's no point. And it's pretty easy to end up with a two gigabyte or more virtual machine, whereas you probably could have kept it slimmed completely down to a one gigabyte. So also when you're doing development on the virtual machine, it helps to have two separate versions of it, one that you're going to use just for packaging up your app because you're going to need to have the full app sitting somewhere on the virtual machine so that you can run a bundle to vendor the gems into the app itself to get ready to be completely packaged up. It's not something you should do in your local development environment. Say if you're developing on OS X and you vendor your gems in that environment, they're not going to run on Linux or once you're in Linux. So all right, next let's talk about networking configuration. That's just a little screenshot of what a user will be prompted with when they, or a customer when they load up the virtual machine the first time or every time, really. You can still SSH into it. It's not like completely blocking everything off, but this is just sitting within a file in Ubuntu called the TTY, the teletype. And it's pretty damn simple to set up. And really you're just, to make this work, the bare minimum is just having some system calls that write to the file, the interface file, or the DNS file. So there's also shut down and reboot options. So here's some, I don't know if that, can you guys read that? It's pretty small. I have it on a text editor too. So yeah, like I was saying, it's a menu-driven app and it's just basically collecting information from system calls and then displaying it and then letting you input things just using GetS as a very simple Ruby app and then letting you just reboot the entire computer. So it's pretty standard, simple, no rocket science here. So the next part of the puzzle is the server itself. And I chose Passenger for it because it just seemed the most dead simple to put your app in a directory and have Passenger just start serving it. Once you hit touch temporary start, then Passenger will load the new code and it doesn't really care if the code isn't there to begin with. It's not going to completely blow up. It's just going to 404. So that means the loader app that we'll talk about in the next slide can easily just extract the packaged up Rails app into a directory that's already been preconfigured for Passenger to serve the app from. And as you can see, sort of online 14, that's where we're running the loader app. So as soon as you boot up the VM for the first time, those error pages will actually, if the Rails app is not found, it'll redirect you to the setup page. The setup page is where you can upload your package file and your license file and it'll be extracted into its eventual, like, living place for good. So the installer app itself is just a simple Sinatra app that takes the package and the license file, puts them in the place where they'll live. And this one, you want to have pre-installed on the VM for distribution because it's going to be doing the work to get Rails up and running. So we can go over a bit of what is actually happening in the Sinatra app. I'm using a simple encryption library called gibberish that just is a nice abstraction over, I believe, open SSL. And then this horribly insecure shared key up there, ASDF, ASDF. And if you do have your code obfuscated, then it doesn't really matter that the key is right there and it's just best to assume that if people are going to be looking into your code, like, they're going to figure out everything about it to begin with. So you don't really need, like, a huge key. So what it does is it first will decrypt the license file to make sure that it's valid or it will just use that license file. Let's say if you're upgrading your license, it can just put that in your Rails app and your Rails app will start picking up the new license. Maybe it's gone from trial to live. So if the package itself is present, the package file, then it'll decrypt it and place it in the directory where a passenger is preconfigured to read it from or to load it from, and then it'll just touch the restart file and redirect you straight to root, and so you'll be good to go. So right, that index is just how simple it can be. I mean, this is like the bare minimum of what it takes. So tweaks to the app itself that we discussed, the license file, in a regular customer subscription app, right, there's all the data, or the customer specifics in the database, and that's checked on every page load. But we can't do that here because there's no way for the subscription side that you're selling them on your sales app to have any effect to the Rails app that's running on their side. So when they get a new license file, it contains within it just an encrypted JSON file, and it'll live in the temp directory and get loaded on boot every time. So we also want to add some of the Rails secrets, well, all of them, and for the simple example app, I'm running Devise also. So the secret token in Devise Secret can be read from the license file on boot, and you want to do that because you don't want one customer to be able to tamper with another customer's cookies. It's a really kind of extreme use case or possibility, but still just best practices. All right, let's see. So this is a really basic module that demonstrates how to read and write the license file. It's using the same encryption key as the loader, and this would obviously be obfuscated, hopefully somewhat, just for a deterrent, but it gives you an idea of how simple it is. It's the first thing to get loaded in the initializer so that the rest of Rails can use it when it's booting up. And from here, we can have calls out to it all over the app, checking the trial period, disabling functionality, displaying messages to link out to the web if they need to upgrade their license. And another big one here is the seat limit, and the seat limit is something that can be checked, say, in a user validator, like, validate seats and just do a count against the license seat number. In that case, also, for administrators, you'd also want to provide a way for them to manage the users, of course, as though they were using your cloud app, so they would be able to deactivate or delete users so it's a free upseats if they needed to. All right, so next is the support package. And this is the really simple implementation of an exception tracker, and it's going to put exceptions in little encrypted files with a cleaned backtrace into your temp directory to be ready for download by the admin when the time comes, when they start hitting 500s. And that just encrypts it back up and tars it and just is able to be emailed to you or dropbox to you, depending on how big it is. All right, lastly is the data import and export management mechanism. So this is just a dead simple PG restore and PG dump. These are Postgres commands. This would be in a controller. So the first one is purely just taking whatever you spit at it and attempting to shove it into the database, and export is just giving you an entire dump. So nothing really tricky here, but it's definitely an important part of being able to upgrade between virtual machines. For background jobs, not going to get into too much, but we used form and export, which allows you to take your proc file and generate upstart processes, and those can be used by Ubuntu, and so that will help, that will let them be executed on boot and stay running, hopefully. I think the next step here would be for your app itself to show the background job status within the app itself and some sort of dashboard, and of course that wouldn't be something you would show on your cloud app, but just something to think about. So lastly, when all of your pieces are in place, what's the best way to make it actually distributable? Obviously you can pare it down to a single script because it's just like a Git push, this would be something that deploys to your CDN, maybe updates your users that there's a new version of it ready for download. So some basic things to do here, this is super basic rake task that just grabs the version, has a bundle, a vendor bundle that throws everything in your vendor directory, and then tars everything up that's pertinent, excludes what's not, and then encrypts it all using your shared key, and that's it. So some of the things that it could do are obfuscation at this step, if you needed to run a post processor on it, then it would happen here, and then last one would be actually uploading it to your CDN and then notifying your users. And I keep a version file in the root just so that all these scripts and random things can use it because each download of the package is going to have some sort of version attached to it that your user is going to be able to want to go back and forth or be able to identify. So that was pretty much it for what you can get up and running as a prototype. And beyond that, once you actually have it in customers' hands, they might, if they're already using AWS, they might want to see it as an Amazon image, which can be spun up without actually having to download anything. It's pretty trivial to convert an OVA file to an AMI. Amazon provides some command line tools to make this simple, so you could also put that into your packaging script. Actually no, this would be a different script that would run purely on your VM when you're ready to cut a new version. And then once you have an AMI listed on Amazon, it's pretty trivial to just hit the public button and then it's searchable in the public registry. So some more things that are going to be the next, taking the next step beyond what we've covered. Resource management, there should be an easy way for your admin to see how your VM is performing and this could be part of the support package also. Clustering, perhaps there's, you could set up a mechanism to run multiple VMs at once for better performance. Mail server, which is not that tricky, but could be important since in your cloud app, you're probably using, you know, SendGrid or MailGun or whatever. Those aren't going to be accessible from the VM. So you should be able to, your admin should be able to specify a mail server. Incremental VM updates, you could also include bash scripts to change pieces of the VM around when you import a new package file. So when that gets exported or untarred and migrations are ran, those migrations could also include, say, new dependencies that were stored inside the package because you can't just run an app to get install from there. You have to assume that your VM doesn't have any access to the outside world. So we talked a little about the SAS app that would run in the cloud in addition to your main one to sell license files, facilitate that. And last one is enterprise sales, but that's just a joke because who knows about that? Like, that's kind of the, that's the big question mark, which I have no idea about either and is definitely uncharted territory. So I think, I think we're running out of time, so maybe we should just do any questions. Yeah, the question was what are we doing for obfuscation? That's still kind of a work in progress. Right now it's a Ruby C extension and the way that GitHub does it is they've compiled their own Ruby to do it, so it's at a lower level than just a jam extension. And so figuring out how to do that right now, but it's not, it's not something that's widely discussed. Yeah, well, the question is how, how do we reduce the amount of patches that we need to ship because there could be maybe a bug that's specific to one customer, but we don't want to have to have everybody else come and download it all at once. Yeah, yeah, there's really not a good way to tighten up the feedback loop beyond having to bundle and, or to package up and then release brand new versions for every little bug piss. I mean, I think that's, that's more of a customer relations question where it's, if it's a small bug, like maybe it can wait into a bigger point release if it's being fixed just for their environment or if it's something that's business critical and it just doesn't work for them then, and they're paying customer, then of course, no matter how small it is, it needs to, how small the change is, a new version has to be released and everybody will see the new version up there. Yeah, it could, it could be incremental like that where the package in, in the migration will actually do the updates. But probably that, that would be for smaller things like maybe you can include a.dev that can be loaded in that's not too huge, but for OS level that would necessitate cutting a brand new VM and asking your customers to go download a new one then do the whole import data export dance. Scaling support teams. Well, we haven't really had to scale ours yet because we haven't had so many customers that it's become overwhelming, but I mean, I, I feel like we're going to just need to scale in traditional ways, but it's yet to be seen what extra sort of hurdles we're going to have with supporting multiple versions of the package and the VM and having this kind of asynchronous support flow where we get, you know, emails or drop boxes of just packages of logs and 500 errors. So that's, it's definitely uncharted territory also. So that there's two pretty simple little apps up there that are just prototypes. It's for COM calling enterprise communications just like let's you post, you know, it's like a simple little blog app. But it shows off some of the licensed file reading and pretty much all the, the screen grabs from the presentation were from that app except for the app loader which is the little Sinatra app. Anything else? All right. Thanks for coming guys. Thank you very much.
 Thanks for coming out late Thursday, guys.  It's been a long couple days.  I'm Nick Merwin, a founder of coveralls.io, and hope you guys have seen some of our badges  around town, GitHub, Readme, et cetera.  I don't know.  We're kind of all over the place hopefully right now.  But today we're going to discuss how it went about figuring out how to get coveralls, a  SaaS app, into the hands of customers that wanted to run it within their corporate networks.  It was pretty untreated territory for me, but I hope this general overview and what  we'll see later with just a really basic prototype will help you guys get some confidence that  you could also take your app to the next level by offering a hosted version to a customer  base that maybe you never thought was possible.  So in that sense, it's kind of going to be partially a business model discussion and  some just general sysadmining stuff, and then of course all kind of viewed through the lens  of Rails since coveralls is a Rails app.  All right, so why would you want to set your app free into the dark world out there outside  of your cushy deployment environment?  So currently let's say your app is hosted on Heroku or VPS.  Coveralls is on DigitalOcean.  And if you have a subscription-based service where users are paying by the month and maybe  there's a few tiers and usage-based add-ons, and it's coming along the crewing users and  it seems like your potential customer base is covered, but you might be neglecting an  even more lucrative and even preferable customer base, which would be the enterprise, quote-unquote.  And this could be monolithic companies where dev teams have a much harder time getting  upper management to embrace cloud-based tools, or perhaps the issue is just that setting up  a $5-a-month service fee takes a mountain of vendor approval paperwork.  So let's chat about how it can make sense to offer a hosted version of your app in addition  to your cloud service.  So there are three kind of tenants that I ran across in figuring this out, or as to  the why you would want to do this.  And the first one is security, and it's pretty obvious that we all worry that when you use  a cloud app, you essentially give up control of your data, and you assume that the devs  on the other side hopefully use Bcrypt on your password, but you never really know for  sure and just hope there's never a real data breach.  But beyond passwords, it could be any sort of data that your company's security department  has forbidden from leaving their networks.  And in fact, it could preclude the possibility of them ever using your app.  This could be maybe a medical company that wants to use your data mining tool that can't  upload anything to the cloud due to HIPAA compliance issues, or maybe it's a dev team  that wants to use your source code analysis tools but can't risk sharing or leaking proprietary  code.  There's also just general business apps like CRM's, internal tasking, chatting, scheduling  that would have plenty of info that those companies wouldn't want to share with the competitor.  But by running a hosted version of your app within their networks, they can just be more  confident that they won't ever have their accounts hacked or their data compromised  out on the open internet.  So second one is a bit more esoteric.  It's reliability, and this is just kind of a worst case thing.  If your app that you're giving them is a DevOps tool that's part of the build or deployment  pipeline, then they're not going to want there to be any unplanned maintenance downtime.  So for instance, with GitHub or Travis, if something goes down, something goes wrong,  you don't have control of it, and it blocks your pushes or your builds that could screw  up your deployment.  And so if you're running it internally, then your dev teams and your sysadmins will always  be able to coordinate to make sure everything's good to go when it comes to crunch time.  The lastly is cost, and the traditional subscription model where it's just pay by the month doesn't  really work when you've given over your app to a customer.  So oftentimes those potential customers are also less price sensitive and could be willing  to pay a premium for your host of service and some sort of extra white glove support  service.  But once you deliver the app to them, you have to assume that it won't be able to phone  home to check the subscription status.  So instead of a time-based subscription, like a monthly fee, you probably are going to need  to do some sort of seat-based usage or something else that's beyond just a monthly or perhaps  yearly.  And then once you're thinking about seats, then you can do seat packs as tiers, as in  the number of active users that can access the app as it's living within their network.  You could also build some sort of self-destruct or shut-down mechanism that will force the  customer to purchase or renew the license after a set amount of time, like six months  or a year, just to make sure that they come back and they keep a valid license.  So there's a lot of different opportunity to work out business model kinks beyond just  a traditional SaaS model.  So before we go into the how we figured this out, let me just mention Coveralls and what  it does and why we took it to the enterprise.  So Coveralls, really quick.  Coveralls is a tool for code cover tracking and notifications.  We can annotate your pull requests with successor failure statuses based on changes in your  coverage percentage or just send messages to your chat or email with updates.  This helps dev teams make sure they don't deliver untested code, especially in a production.  And on site you can see line by line coverage of the code base, the cloud version for open  sources free, but we charge for private repos.  And the rub comes because we don't, while we don't actually store source on our servers,  we do store private scoped OAuth tokens for GitHub.  And so for some companies, that's just a non-starter.  We can't have, they can't allow us to see within their private repos.  And then some of them are already using GitHub Enterprise, so they're not using Cloud GitHub  to start out with, and the two are not interoperable.  So what we did is gathered interest from a potential customer base over a matter of months,  and it became clear that there was enough of an interest that we should try to do this  and try to make it happen.  And so that's what brought us here.  So there are some pretty big main hurdles that I found in converting your general cloud  Rails app to a fog one where it's living within, below the cloud layer.  So delivery and installation, let's talk about delivery first.  The best user experience for your customers would be to have the least amount of work  to get your app up and running in the environment, obviously.  So three download files would be, I think, is the most consolidated way that we can achieve  this.  And that's first the virtual machine that the app is going to run on, and that could  just be a Linux box, as I'll show you, a virtual Linux box.  And that's typically between 800 and a gigabyte size download.  The next one would be the packaged up app files itself, and that's a decently high number,  just between 50 and 100 or more, depending on how many gems you have, because all the  gems need to be bundled and vended into the app package itself, as we'll see.  And then lastly, a license file that's specific to the customer and is generated for every  customer.  And so out of those three, your customers would probably only need to download the VM, like  else in a blue moon when you do big upgrades to the underlying system or have new dependencies.  The app package is something that they would download whenever there's updates, and we'll  get to that in the license file would be around the same amount quicker, like when they go  to trial to active or what.  So actually, let's talk about those incremental updates for a sec.  With the standard deployment, obviously, when your code is hosted up on Heroku, getting  features and fixes out to your users is as easy as getting pushed, not so in this case  when your app is running completely out somewhere in the wild where you have no access to it.  So that's where the app packages come in.  It's a smaller download that has just the most up-to-date bug fixes, features, et cetera,  that is a much quicker download than having to re-download and configure an entire virtual  machine.  So also for the license files, one last thing, it makes sense that this is the part where  you'll probably need to build a secondary app in addition to your main Cloud one that  will just service those customers and charge them subscription fees and have trial accounts  and all that stuff.  So that's where the download license file and then download the other two in the secondary  app.  All right.  So installation-wise, the networking config is the first thing that they're going to see  and since they're not, you don't want them to have to log into the VM themselves.  The best way to get around this is to build a tiny little menu-driven Ruby app that gets  presented on boot and I'll show you guys how we did this.  And that way, the customer won't need to know any specifics about Linux networking.  In addition to that, external access just has to be assumed as not going to be a possibility.  So hopefully if they're, the reason why they're purchasing this hosted version is that so  they can be completely confident in the security behind it.  They won't give it any external access, you know, that'll be totally walled off in a sandbox  in their network, hopefully.  So the assumption of there not being any way to access it from the outside is probably  valid.  Last one is just process management and that's how are we going to get background jobs and  the server itself to stay alive and start at boot.  All right.  So more of these hurdles.  So product support, generally you can see immediately when a user hits a 500 error, it's going  to, you're going to get an email and you're going to be able to go on to your error tracking  service, air brake, whatever, and take a look.  In this case, there's no way to know when that happens.  There's no way to get pinged.  So you're just going to get an email from your customer saying something's not right,  what's going on.  So you need to provide them a way to be able to send you details about the exception and  so we're going to look at how to just keep those exceptions somewhere that we can then  send over or let your customer send to you in like a text to an email or something.  So that goes same for logging.  If something's really funky on it, you're going to be able to see if there's any weird  parameters coming through the routes, especially when they're not causing an exception per  say.  Lastly there's resource management.  If it seems like things are running slowly on their site, there needs to be some way  to address that.  I'm not going to go over that in this talk because I feel like that's more of a sysadmin-ing  issue, but that's something for homework, I guess.  So lastly is the intellectual property question and this is a big one.  And because you're giving them a VM, they can unpack the VM and mount the disk and look  at everything you put there.  So even though it comes in one nice file, it's still extractable and you can just load  up another VM in virtual box of VMware and then mount the disk that you had delivered  them.  So that means they're going to be able to look at everything as though they were a root  user.  So I feel like database access, unless, I don't know, there might be some way to get  around this, but you should just assume that they're going to have access to the database  itself and it doesn't really matter.  I don't know.  That's the bigger question here, along with code obfuscation.  Since Ruby, there's no way to really, really fully protect your source code once it's being  executed in somebody else's environment and it has to actually work itself, you can write  a code obfuscator as a deterrent and that's what we've done for CoverAll's Enterprise.  But I think, I mean, that prevents people from immediately reading it if they just mount  your disk and look through your source file tree.  But it's probably safest to just cover this with your license and say something like,  the license you're purchasing from us covers only the use of the software modification  or redistribution or not permitted.  So it just kind of comes down to legalese as being your last line of defense here.  And if this keeps you up at night, then maybe the best thing to remember is that your customers  for your app should be most interested in getting updates, getting bug fixes, new features  and support rather than breaking into your source code, reading it, copying it, spreading  out on the Internet.  So that could be its own talk about how to actually achieve some level of code obfuscation,  but we're not going to go over that really today.  Okay.  So let's get into the nitty-gritty here.  So this is a pretty hectic diagram, but it's basically showing the general architecture  of how the app is running within our VM, which is Ubuntu.  And we chose Ubuntu because it's just widely used and relatively easy to configure.  So the main components that live within it are the network config app, which is the first  thing that customers will see when you boot it up.  And that lets you do things like select static networking or DHCP and then set the name servers  and then reboot, shut down, et cetera.  So it's just kind of a simple little starter app.  And then behind our web server, we'll go into why we chose Passenger for it.  There's going to be a reinstalled app that I just made in Sinatra.  It's a really simple app that facilitates the unpacking of the package file that contains  the actual app or the app updates and the license file.  And so within that, there's, within behind the web server, we'll live your Rails app  also.  So the Rails app itself, some of the things that these three components I think are the  main ones that set it apart from your standard cloud hosted app or where you started from  to begin with to get it to this point.  So the license file reader is going to read and cache your license at the boot when your  Rails app boots and then it's checked on every page view.  And then you can do things like enable the disabled features or lock the app down.  Second one is data import and export because when you need to download a brand new upgraded  virtual machine, say it's got new dependencies for new features, then you're going to need  to give your customers a way to get their data out, the old one, and get it in the new  one.  So that's things like dumping the database.  If your app has uploads, then you're going to need to pull those together if the uploads  are living on the VM.  And then anything else that the users might have uploaded or changed that has a state  on the VM.  And the third one is support package generation and downloads.  So we'll show how we can use the Rails rescue from in the most general terms just to collect  all the exceptions that happen in controllers at least.  And they can be archived into a temp directory and then when an admin for the app as it's  living on the VM is ready to or needs some support, they can hit a link that generates  a archived file that we can encrypt also and then email it or dropbox it over to us.  Okay, so setting up the environment first.  For development, I think it's probably easiest to use virtual box.  It's free and pretty simple and it's really easy to export and appliance from your machine  called an OVA file and that's just a nice little, it's a tar or zip or something with  a.OVA extension.  That's just something easy you can link to and serve from your CDN.  Also Ubuntu, just because of its, how widely it's used, hopefully we can trust its built  in security settings and standards.  So we kind of get that out of the box and our customers get that confidence out of the  box.  So when you provision it for your Rails app, you obviously want to use the minimum amount  of dependencies and touches to it so that you can just keep the download size smallest.  If you don't need to install something like ImageMagic, then just don't do it.  There's no point.  And it's pretty easy to end up with a two gigabyte or more virtual machine, whereas  you probably could have kept it slimmed completely down to a one gigabyte.  So also when you're doing development on the virtual machine, it helps to have two separate  versions of it, one that you're going to use just for packaging up your app because you're  going to need to have the full app sitting somewhere on the virtual machine so that you  can run a bundle to vendor the gems into the app itself to get ready to be completely packaged  up.  It's not something you should do in your local development environment.  Say if you're developing on OS X and you vendor your gems in that environment, they're not  going to run on Linux or once you're in Linux.  So all right, next let's talk about networking configuration.  That's just a little screenshot of what a user will be prompted with when they, or a  customer when they load up the virtual machine the first time or every time, really.  You can still SSH into it.  It's not like completely blocking everything off, but this is just sitting within a file  in Ubuntu called the TTY, the teletype.  And it's pretty damn simple to set up.  And really you're just, to make this work, the bare minimum is just having some system  calls that write to the file, the interface file, or the DNS file.  So there's also shut down and reboot options.  So here's some, I don't know if that, can you guys read that?  It's pretty small.  I have it on a text editor too.  So yeah, like I was saying, it's a menu-driven app and it's just basically collecting information  from system calls and then displaying it and then letting you input things just using GetS  as a very simple Ruby app and then letting you just reboot the entire computer.  So it's pretty standard, simple, no rocket science here.  So the next part of the puzzle is the server itself.  And I chose Passenger for it because it just seemed the most dead simple to put your app  in a directory and have Passenger just start serving it.  Once you hit touch temporary start, then Passenger will load the new code and it doesn't really  care if the code isn't there to begin with.  It's not going to completely blow up.  It's just going to 404.  So that means the loader app that we'll talk about in the next slide can easily just extract  the packaged up Rails app into a directory that's already been preconfigured for Passenger  to serve the app from.  And as you can see, sort of online 14, that's where we're running the loader app.  So as soon as you boot up the VM for the first time, those error pages will actually,  if the Rails app is not found, it'll redirect you to the setup page.  The setup page is where you can upload your package file and your license file and it'll  be extracted into its eventual, like, living place for good.  So the installer app itself is just a simple Sinatra app that takes the package and the  license file, puts them in the place where they'll live.  And this one, you want to have pre-installed on the VM for distribution because it's going  to be doing the work to get Rails up and running.  So we can go over a bit of what is actually happening in the Sinatra app.  I'm using a simple encryption library called gibberish that just is a nice abstraction  over, I believe, open SSL.  And then this horribly insecure shared key up there, ASDF, ASDF.  And if you do have your code obfuscated, then it doesn't really matter that the key is right  there and it's just best to assume that if people are going to be looking into your code,  like, they're going to figure out everything about it to begin with.  So you don't really need, like, a huge key.  So what it does is it first will decrypt the license file to make sure that it's valid  or it will just use that license file.  Let's say if you're upgrading your license, it can just put that in your Rails app and  your Rails app will start picking up the new license.  Maybe it's gone from trial to live.  So if the package itself is present, the package file, then it'll decrypt it and place it in  the directory where a passenger is preconfigured to read it from or to load it from, and then  it'll just touch the restart file and redirect you straight to root, and so you'll be good  to go.  So right, that index is just how simple it can be.  I mean, this is like the bare minimum of what it takes.  So tweaks to the app itself that we discussed, the license file, in a regular customer subscription  app, right, there's all the data, or the customer specifics in the database, and that's checked  on every page load.  But we can't do that here because there's no way for the subscription side that you're  selling them on your sales app to have any effect to the Rails app that's running on  their side.  So when they get a new license file, it contains within it just an encrypted JSON file, and  it'll live in the temp directory and get loaded on boot every time.  So we also want to add some of the Rails secrets, well, all of them, and for the simple example  app, I'm running Devise also.  So the secret token in Devise Secret can be read from the license file on boot, and you  want to do that because you don't want one customer to be able to tamper with another  customer's cookies.  It's a really kind of extreme use case or possibility, but still just best practices.  All right, let's see.  So this is a really basic module that demonstrates how to read and write the license file.  It's using the same encryption key as the loader, and this would obviously be obfuscated,  hopefully somewhat, just for a deterrent, but it gives you an idea of how simple it  is.  It's the first thing to get loaded in the initializer so that the rest of Rails can  use it when it's booting up.  And from here, we can have calls out to it all over the app, checking the trial period,  disabling functionality, displaying messages to link out to the web if they need to upgrade  their license.  And another big one here is the seat limit, and the seat limit is something that can be  checked, say, in a user validator, like, validate seats and just do a count against the license  seat number.  In that case, also, for administrators, you'd also want to provide a way for them to manage  the users, of course, as though they were using your cloud app, so they would be able  to deactivate or delete users so it's a free upseats if they needed to.  All right, so next is the support package.  And this is the really simple implementation of an exception tracker, and it's going to  put exceptions in little encrypted files with a cleaned backtrace into your temp directory  to be ready for download by the admin when the time comes, when they start hitting 500s.  And that just encrypts it back up and tars it and just is able to be emailed to you or  dropbox to you, depending on how big it is.  All right, lastly is the data import and export management mechanism.  So this is just a dead simple PG restore and PG dump.  These are Postgres commands.  This would be in a controller.  So the first one is purely just taking whatever you spit at it and attempting to shove it  into the database, and export is just giving you an entire dump.  So nothing really tricky here, but it's definitely an important part of being able to upgrade  between virtual machines.  For background jobs, not going to get into too much, but we used form and export, which  allows you to take your proc file and generate upstart processes, and those can be used by  Ubuntu, and so that will help, that will let them be executed on boot and stay running,  hopefully.  I think the next step here would be for your app itself to show the background job status  within the app itself and some sort of dashboard, and of course that wouldn't be something you  would show on your cloud app, but just something to think about.  So lastly, when all of your pieces are in place, what's the best way to make it actually  distributable?  Obviously you can pare it down to a single script because it's just like a Git push,  this would be something that deploys to your CDN, maybe updates your users that there's  a new version of it ready for download.  So some basic things to do here, this is super basic rake task that just grabs the version,  has a bundle, a vendor bundle that throws everything in your vendor directory, and then  tars everything up that's pertinent, excludes what's not, and then encrypts it all using  your shared key, and that's it.  So some of the things that it could do are obfuscation at this step, if you needed to  run a post processor on it, then it would happen here, and then last one would be actually  uploading it to your CDN and then notifying your users.  And I keep a version file in the root just so that all these scripts and random things  can use it because each download of the package is going to have some sort of version attached  to it that your user is going to be able to want to go back and forth or be able to identify.  So that was pretty much it for what you can get up and running as a prototype.  And beyond that, once you actually have it in customers' hands, they might, if they're  already using AWS, they might want to see it as an Amazon image, which can be spun up  without actually having to download anything.  It's pretty trivial to convert an OVA file to an AMI.  Amazon provides some command line tools to make this simple, so you could also put that  into your packaging script.  Actually no, this would be a different script that would run purely on your VM when you're  ready to cut a new version.  And then once you have an AMI listed on Amazon, it's pretty trivial to just hit the public  button and then it's searchable in the public registry.  So some more things that are going to be the next, taking the next step beyond what we've  covered.  Resource management, there should be an easy way for your admin to see how your VM is performing  and this could be part of the support package also.  Clustering, perhaps there's, you could set up a mechanism to run multiple VMs at once  for better performance.  Mail server, which is not that tricky, but could be important since in your cloud app,  you're probably using, you know, SendGrid or MailGun or whatever.  Those aren't going to be accessible from the VM.  So you should be able to, your admin should be able to specify a mail server.  Incremental VM updates, you could also include bash scripts to change pieces of the VM around  when you import a new package file.  So when that gets exported or untarred and migrations are ran, those migrations could  also include, say, new dependencies that were stored inside the package because you can't  just run an app to get install from there.  You have to assume that your VM doesn't have any access to the outside world.  So we talked a little about the SAS app that would run in the cloud in addition to your  main one to sell license files, facilitate that.  And last one is enterprise sales, but that's just a joke because who knows about that?  Like, that's kind of the, that's the big question mark, which I have no idea about either and  is definitely uncharted territory.  So I think, I think we're running out of time, so maybe we should just do any questions.  Yeah, the question was what are we doing for obfuscation?  That's still kind of a work in progress.  Right now it's a Ruby C extension and the way that GitHub does it is they've compiled  their own Ruby to do it, so it's at a lower level than just a jam extension.  And so figuring out how to do that right now, but it's not, it's not something that's widely  discussed.  Yeah, well, the question is how, how do we reduce the amount of patches that we need  to ship because there could be maybe a bug that's specific to one customer, but we don't  want to have to have everybody else come and download it all at once.  Yeah, yeah, there's really not a good way to tighten up the feedback loop beyond having  to bundle and, or to package up and then release brand new versions for every little bug piss.  I mean, I think that's, that's more of a customer relations question where it's, if it's a small  bug, like maybe it can wait into a bigger point release if it's being fixed just for  their environment or if it's something that's business critical and it just doesn't work  for them then, and they're paying customer, then of course, no matter how small it is,  it needs to, how small the change is, a new version has to be released and everybody will  see the new version up there.  Yeah, it could, it could be incremental like that where the package in, in the migration  will actually do the updates.  But probably that, that would be for smaller things like maybe you can include a.dev that  can be loaded in that's not too huge, but for OS level that would necessitate cutting  a brand new VM and asking your customers to go download a new one then do the whole import  data export dance.  Scaling support teams.  Well, we haven't really had to scale ours yet because we haven't had so many customers  that it's become overwhelming, but I mean, I, I feel like we're going to just need to  scale in traditional ways, but it's yet to be seen what extra sort of hurdles we're going  to have with supporting multiple versions of the package and the VM and having this  kind of asynchronous support flow where we get, you know, emails or drop boxes of just  packages of logs and 500 errors.  So that's, it's definitely uncharted territory also.  So that there's two pretty simple little apps up there that are just prototypes.  It's for COM calling enterprise communications just like let's you post, you know, it's like  a simple little blog app.  But it shows off some of the licensed file reading and pretty much all the, the screen  grabs from the presentation were from that app except for the app loader which is the  little Sinatra app.  Anything else?  All right.  Thanks for coming guys.  Thank you very much.
Delivering_Autonomous_Rails_Apps_0_47.png Delivering_Autonomous_Rails_Apps_48_155.png Delivering_Autonomous_Rails_Apps_156_311.png Delivering_Autonomous_Rails_Apps_312_2027.png Delivering_Autonomous_Rails_Apps_2028_3695.png Delivering_Autonomous_Rails_Apps_3696_9071.png Delivering_Autonomous_Rails_Apps_9072_9215.png Delivering_Autonomous_Rails_Apps_9216_10691.png Delivering_Autonomous_Rails_Apps_10692_11171.png Delivering_Autonomous_Rails_Apps_11172_11507.png Delivering_Autonomous_Rails_Apps_11508_22811.png Delivering_Autonomous_Rails_Apps_22812_22919.png Delivering_Autonomous_Rails_Apps_22920_30935.png Delivering_Autonomous_Rails_Apps_30936_32159.png Delivering_Autonomous_Rails_Apps_32160_33083.png Delivering_Autonomous_Rails_Apps_33084_35159.png Delivering_Autonomous_Rails_Apps_35160_38723.png Delivering_Autonomous_Rails_Apps_38724_40559.png Delivering_Autonomous_Rails_Apps_40560_43499.png Delivering_Autonomous_Rails_Apps_43500_47891.png Delivering_Autonomous_Rails_Apps_47892_48095.png Delivering_Autonomous_Rails_Apps_48096_49199.png Delivering_Autonomous_Rails_Apps_49200_51995.png Delivering_Autonomous_Rails_Apps_51996_52043.png Delivering_Autonomous_Rails_Apps_52044_52103.png Delivering_Autonomous_Rails_Apps_52104_52151.png Delivering_Autonomous_Rails_Apps_52152_52307.png Delivering_Autonomous_Rails_Apps_52308_52415.png Delivering_Autonomous_Rails_Apps_52416_52559.png Delivering_Autonomous_Rails_Apps_52560_52667.png Delivering_Autonomous_Rails_Apps_52668_52727.png Delivering_Autonomous_Rails_Apps_52728_52811.png Delivering_Autonomous_Rails_Apps_52812_53207.png Delivering_Autonomous_Rails_Apps_53208_53279.png Delivering_Autonomous_Rails_Apps_53280_53675.png Delivering_Autonomous_Rails_Apps_53676_53735.png Delivering_Autonomous_Rails_Apps_53736_53783.png Delivering_Autonomous_Rails_Apps_53784_54035.png Delivering_Autonomous_Rails_Apps_54036_54215.png Delivering_Autonomous_Rails_Apps_54216_54311.png Delivering_Autonomous_Rails_Apps_54312_54491.png Delivering_Autonomous_Rails_Apps_54492_54539.png Delivering_Autonomous_Rails_Apps_54540_54707.png Delivering_Autonomous_Rails_Apps_54708_54779.png Delivering_Autonomous_Rails_Apps_54780_54899.png Delivering_Autonomous_Rails_Apps_54900_54995.png Delivering_Autonomous_Rails_Apps_54996_55127.png Delivering_Autonomous_Rails_Apps_55128_55187.png Delivering_Autonomous_Rails_Apps_55188_55619.png Delivering_Autonomous_Rails_Apps_55620_55775.png Delivering_Autonomous_Rails_Apps_55776_55907.png Delivering_Autonomous_Rails_Apps_55908_55967.png Delivering_Autonomous_Rails_Apps_55968_56375.png Delivering_Autonomous_Rails_Apps_56376_56495.png Delivering_Autonomous_Rails_Apps_56496_56543.png Delivering_Autonomous_Rails_Apps_56544_56903.png Delivering_Autonomous_Rails_Apps_56904_56951.png Delivering_Autonomous_Rails_Apps_56952_57143.png Delivering_Autonomous_Rails_Apps_57144_57227.png Delivering_Autonomous_Rails_Apps_57228_57275.png Delivering_Autonomous_Rails_Apps_57276_57323.png Delivering_Autonomous_Rails_Apps_57324_57371.png Delivering_Autonomous_Rails_Apps_57372_57551.png Delivering_Autonomous_Rails_Apps_57552_79627.png