Open Sourcing: Real Talk
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Evans, Andrew
Hired open-sources some useful abstractions from our Majestic Monolith® and we've learned a lot. Some tough lessons, and some cool knowledge. We'll cover: When & where should you pull something out of the code? Does it really help? What things are important to think about? What if it never takes off? We'll also look at some design patterns from our open-source work.
 We got a lot to cover here, so I'm going to go ahead and get started. This is open sourcing, real talk. So who am I? I'm Andrew Evans. I work for Hired. I've been there a little over two years. I've worked for all kinds of different startups for like nine, ten years. I've been working with Ruby on Rails since about 1.8 or so. If anybody remembers some fun Twitter-related arguments back then, good times. And I've been writing bugs in PHP or QBasic or all sorts of things for longer than I care to remember. So what is Hired? Hired is the best way to get a tech job. As a candidate, you make a profile. We put you live in our two-sided marketplace. And then companies ask you to interview with them, and they have to give you salary, equity, everything else up front, and you decide if it's worth responding and having that conversation. As a company, this is the best and fastest way to get great candidates into your pipeline. But this is not a pitch talk, so I'm not going to focus on that. Instead, I'm going to talk about open sourcing. This is something that Hired's been doing basically since the beginning. We have 46 public repos on GitHub. A lot of those are forks where we've needed to make some modifications or thought about modifying things. But some of those are open source projects that maybe you can use. These are just a couple. We have a PubSub layer over Sidekick. We have a query builder for Elasticsearch. We have Fortitude, which is our CSS framework, which is built to scale. It's built for teams. We even have some smaller ones. We have a RackMiddleware that will log stats from the Puma web server. So if you're using that, it can be helpful for gathering more info. And we have a tiny thing, which will change the color of the lights in your office. If you're using something like a Phillips Hue, we turn our lights red when the master build fails. So it's a good incentive to get on Slack and figure out what happened. I'm just going to dive deep into these two so we can talk a little bit about what producing these projects were like. I'm going to cover whether it would be worthwhile or not for you to do any open source work, extracting anything out of your app, and making it public. I'm going to talk about some of the challenges that we faced in doing that. And I'm going to talk about how it played out across our team, like how did that affect the dynamics. And I'm going to try and get a little technical and talk about what we learned about Ruby and RubyGems and Rails. Ruby will have some time for Q&A at the end. So why should you open source things? Well, if you've got some logic in your application, then clearly every other Rails app is doing pretty much the same thing. So if we all share our code, then we could all do this stuff automatically. When you open source something, the community will fix bugs in it. They'll add features. They'll document your things. It's free labor. It's amazing. You'll get developer street cred. You'll get lots of respect and appreciation. You'll get 10K followers on Twitter and hundreds of thousands of GitHub stars. And people will open issues and say, well, there's no bugs or problems with this. I just want to say thank you and we love you. So literally none of that is true. You may have bits of application code that everyone needs across basically every Rails app, in which case what's it like working at Thoughtbot? But for the most part, you may be doing some pretty specialized stuff. Most of your app is business logic, so that lives in your business app. So real talk about our projects in particular. I pulled some data off of Best Gems. Best Gems basically scrapes data off rubygems.org every day and tracks like total number of downloads, average number of downloads, et cetera. And it gives your gem a rank out of the 130,000 plus gems on rubygems. I can see ours are not quite in the top 100 or 1,000. We're probably not going to be on the public leaderboards anytime soon. Our average daily downloads, I adjusted and got rid of weekends and like holidays for that. But basically it's a little bit over the range we'd expect for how many times a day we deploy to Heroku. So hopefully these aren't all just Git push Heroku. Oh, look, we got more downloads. I also pulled some data off GitHub. The number of stars and followers we have is a bit more encouraging. There's probably some people outside of our company that care about these repos. But we're not getting people opening issues just saying how great we are. I found some gems that are similar on the Best Gems ranking. So one of them is a framework that lets you build bots for Google Wave. Does anybody remember that? That existed for a hot second. One of them is a command line interface to the pirate bay. I think the founders announced a few months ago that they'd been sued and thrown in jail too many times, so that's dead. One of them combines ActiveAdmin and Trailblazer. Trailblazer is the object-oriented, explicit over implicit, loosely coupled, well-organized framework that you can add on top of your Rails app. ActiveAdmin is literally the opposite of every one of those values. So I'm not sure what you get by combining them. And the last one is a command line app that gives you a random Mitch Hedberg quote. I used to use that all the time. I mean, I still do, but I used to, too. So what we found is if you don't market your gem, then people don't really just find it organically. We haven't done a lot of promotion for these. I don't think we post about them on our blog too often. This may be the first talk where we've gone deep into our open-source stuff at RailsConf. So we haven't gotten a lot of people that just found this out of the blue and said this is the greatest thing. Was it still worth it for us to open-source this code from our application? I'll dive in a little bit and we'll see. So Reactor is our enhanced interface to Sidekick. This is how we do a lot of our background processing at Hired. Sidekick, if you don't know, is the premier background processing library for Ruby and Rails application that was made by Mike Purhem, that guy. Mike is awesome. Thank you very much. It's been a wonderful system for us at Hired and it's been rock solid. So the thing that we were looking at when considering Reactor was when you have, say, a candidate updating their profile, you may want to do a couple of different things in the background. You might want to notify the talent advocate that's working with them. You might want to bust a cache or update a saved search or something like that. So with Sidekick, you have worker classes for these things and you perform a sync and those things will happen in the background outside of your web request. But if you're doing a whole bunch of things, then you may have a whole bunch of different workers that you have to tell to do the thing. So you face a choice. Do you want to copy and paste all of those calls everywhere that you're updating a candidate's profile? Or you can have one worker that fires off and then that calls other workers, which might then call other workers. For the most part, this isn't a huge deal. I think it's a recommended practice. But if you're like me and you tend to forget things, then you might throw a cycle into your workers and they'll spin up 100 or 10,000 jobs. However many activate until you can shut down Sidekick and figure out what happened. So basically, with that pattern, you end up with this big graph of things happening throughout Sidekick. You have to kind of model that in your head and it can make it a little difficult to track down where exactly specific jobs are coming from. So Reactor is a publish and subscribe system. Basically you publish an event. That event has a name, which is just a symbol. You can pass it any arbitrary data. There's also an extension so that you can have active record objects publish an event. And then those objects are serialized and deserialized and available within your event code. So this will kick off a single Sidekick worker, which then locates all these subscribers to that particular event name. And it will enqueue a job for each of those. We wrote a convenience method for our controllers called action event. This will publish an event from the current user and it will merge in parameters or any other data that we think would be useful for tracing, tracking, analytics, et cetera. So what do subscribers look like? You include this subscriberable module. That gives you the class method on event. So you say on event, you give it the event name and then you give it a block of what to do when that event happens. So in our first example, we pull out the actor, which is whatever published the event. We get their ID and then we can bust a cache related to that particular record. You can also set a wild card handler. So this thing will respond to all events. And we use that for logging site activity to Postgres or to anywhere else that we want to have that analytics data available. There's more options to it. Instead of passing a block, you can pass it an event name. If your objects and functionality are a little more complicated and you want to unit test them, this can make it a little bit easier. You can also pass options to it like delays and perform later. So for example, once an interview request happens from this subscriber, then two days later we'll send an update. And you can do some logic in that, like figure out if it's canceled and then abort. We also have a system for creating model-based subscribers. So in this case, we create a subscriber called admin notifier. This is going to live in our Postgres database. And we define the on fire code to say what happens when this event fires. The neat thing about this is you can have it subscribe to multiple different events from the database. So if I'm an admin and I want to be notified when these three different things happen, then I can just do that through our admin interface and not have to deploy new code to handle that. These can be a little bit magical. So their use has fallen off a little bit. Finally, you can have records publish events based on the data in those records. So for example, if we have an interview that has a start at and an end at time, then we can tell it to publish events at those times. We can even pass it some conditions and some other things similar to validations and callbacks. And the nice thing is if the model gets updated, the start at or end at changes, then it will automatically reschedule that for us. So did it help that we took this code and instead of just building it in our application, we put it in an open source repo? Well when you do that, you get this nice green field project. And it actually changes your desktop wallpaper to the old Windows XP wallpaper. Green fields are great. You don't have to worry about your legacy code. You can just start developing fresh. And it keeps your code separate outside of Kansas out of your application. So that produces a little bit of friction to updating the library, which it turns out can be a really good thing. The code for reactor is kind of out of sight, out of mind for day-to-day feature development for us. So it discourages making small arbitrary changes. If I want to say, like, oh, this should be calling instance methods, not class methods, or this or that, I'm less likely to take that on because I have to context switch and then publish to RubyGems and go through this slightly higher friction process. Because it turns it into a lower churn section of code, it keeps the pattern and convention clear. This action event is used all over our application, and basically it stays the same everywhere and it has just kept working. We had to make some updates for Rails 5, and I'm about to make some updates for Sidekick 5, but for the most part, we don't have to think about it too much. Also, when you have this in a public repository instead of your private application repository, that's going on your permanent record on GitHub. That might add just a little bit more pressure to do it right. You might be a little more attentive to things like documenting all of your methods and making sure that your test coverage is 100% and that your code climate score isn't going down. Finally, when you're publishing through RubyGems specifically, you have to be a little more thoughtful about your versioning. You really have to think about, okay, is this going to break anything? Is this backwards incompatible? Or is this just a feature or a patch release that isn't going to break anything existing but maybe provide a new feature? Also, when you're building outside your application and you're not thinking so much about features, it gives you a chance to think more clearly about the pattern of what you're building. So instead of thinking about what do I need to do this feature, you think about what is the minimum information that this open source library needs to do its job. You also have to think about how RubyGems and Rails and Bundler and these other tools work. One thing I stumbled on to pretty early is that requiring Rails is harder than it sounds. I looked through things like device and paperclip and Sidekick and they all require Rails in totally different ways, which I found amusing. Some of these libraries require Rails for all of their functionality and some of them have a few Rails-specific extensions. That caused me to dive a little deeper into, okay, well, what is a rarity? Is that something I need? Where do I put the require statements? How do I manage the dependencies for that? And then it makes you think about whether you want to use Rails-specific extensions inside your gem that's living outside your application. So do we want to use ActiveSupportsConcerns pattern, which is a pretty nice way of extending objects or do we want to use the basic Ruby include and extend patterns? And then you think a little bit more about the overall design pattern. So when we're thinking about React or we're thinking, oh, I just want this action event method in my code so I can just put events all over the place. But when you're designing a gem from the ground up, it's like, well, okay, what's the name for that pattern? So when you go through design pattern, blog post, or the Gang of Forerbook or whatever, you say, oh, this is a message bus. So we're going to make channels that events get passed along. These events are going to have arbitrary data. Some people get this confused with event delegation, particularly if you've done a lot of jQuery. You can have multiple listeners for particular events there. But those actually form a chain. And any of the listeners in that chain can abort the entire process. With the message bus, all of the subscribers get the message regardless of what each of them are doing individually. So if you wanted to see some examples of subscribers and message buses written in Rails, the shiny new action cable has a pretty well put together, well documented example of this. I have a link here to the subscriber map. Or if you want something a bit more core, active support notifications has been in Rails for quite a while. If you're wondering how all of your database statements get from the application code to your development console, this is what's doing it. And this has a link to subscribers in the class that's responsible for organizing them and mapping them. So TLDR, publishing the reactor gem, didn't make us famous, didn't really help establish any of those selfish goals that we put out. But it did help our team establish a convention. It has quietly kept working for years. And it's allowed us to write that code in a way that's smaller and more loosely coupled and better organized. It's also a good excuse when anyone wants to work on it to go in and look at the pattern, look at the examples, and see how does Message Bus work, what's cool about this. The other one I wanted to talk about was Stretchy. This is a composable query builder for Elasticsearch. Elasticsearch is a pretty amazing technology for full text search. It's based on Lucene. It's similar to Solar. Basically, when you want something more advanced than a relational database I like statement, then this is a good place to go. And yes, we spent an inordinate amount of time thinking about the name for this because Nogem is going to take off and become popular unless you have a really clever pun as the name for it. So I wrote the first version of this. The goals that I was thinking of were I wanted to have an active record-ish query syntax that we can use for Elasticsearch. I wanted to have these immutable query objects which make handling and caching responses really easy to manage. I wanted this to be just a query builder for indexing our models to Elasticsearch. We already have reactor so we can index on specific events. We have serializers. So I figured we'd just keep our Elasticsearch gem out of all of that business. I wanted to make this flexible enough for pretty complicated Elasticsearch queries. Because this is just a query builder, it's taking in a bunch of method calls and then it's producing some JSON that we're going to send to Elasticsearch. That means we don't really need any dependencies, which is pretty nice. We don't have to require all of Rails for this one thing. And the end goal of all of that is we have these composable scopes that we can use in the same way that we're familiar with from ActiveRecord. So here's an example of what that looks like. This is a query that should get us back Lynn Conway. She basically invented modern microprocessors. So it's a good person to find in our candidate database. You can see here we have the where method. Pretty similar to ActiveRecord. You can throw some fairly arbitrary parameters at it and it will figure out what the right query is to apply those. But then there's some more Elasticsearch specific stuff. This match query, for example, is like the most basic full text matching relevant sorting query in Elasticsearch. And then we have these boost methods. That gets into what's called a function score query. It lets you tweak how your search results are going to be ranked. When you look at the query that's generated, it's this giant message JSON thing. When I first started working with Elasticsearch, I tried to figure out how to make these giant JSON documents composable so we could mix and match them in different ways. Turned out to be really hard, which is what inspired this whole ActiveRecord-like approach. So some other example methods. I won't get too deep into these, but you tell it like what index you're going to search, what type you want to get back. And then you can use where for filters. You can use match for full text matching. You can or multiple filters together by using the should clause. This comes straight from the Elasticsearch DSL. And then you can add boost to tweak your relevance algorithms. And finally, if what Stretchy has isn't enough, then you can go lower level. You can say this is just a query and then throw some raw JSON at it. And it will figure out where that needs to go in your final query that's going to be sent over. Elasticsearch has a whole lot of stuff in it. It has about 7 million tools for doing different kinds of searches and figuring out relevance and producing the set of results that you want. So it became obvious pretty early we couldn't support all of that. So the goal was just to get to a system where we could do what we needed to in the app and then hopefully expand that as we got more and more features and hired. So how do I get started making an active record chainable interface? I pulled out this 2010 blog post that I've had bookmarked forever, RailsTips.org. Thanks, John Unimaker. This stuff still works. It's still pretty much the same in Ruby 2.3. Hopefully you have a class that class has some data, maybe an array of nodes, let's say. And each of your chainable methods returns a new instance of that class with some new stuff added to it. So when you get down, you're like, okay, we're building up this array and then we're going to compile it to this JSON query. What does that look like? Is there a name for that? How does it work? First out, it's called the visitor pattern. I was looking around at how other repos do this. I looked at active record and oh, that uses A-REL. And A-REL has a visitor pattern. So that has a collector class and then it has a tree manager for figuring out what was collected and compiling it. And then it has multiple different visitor classes to take that tree of abstract nodes and compile it into a query for, say, MySQL or Postgres or Oracle, I guess, if you're into that kind of thing. This is an example of what that tree of abstract nodes is going to look like. You have a select node and then that can have multiple child nodes, like what are you selecting, where are you getting it from, and then any filters that are going to be applied to your query. And then those filters can have child nodes and those have child nodes. So the visitor will traverse all of those and compile it into the final string. So Stretchy didn't need that full stack. There's really only one elastic search. We decided because time, we only wanted to support the latest version. So we can basically combine the visiting and compile steps. But the pattern was still pretty much the same. Some challenges we ran into in having this query builder live outside of our repo and kind of its own thing. It was a little confusing to the rest of the team initially. If you look at other gems for integrating elastic search into your Rails app, they do a lot more. They have search Rails and search kick and chewy. They all tie into active model. They have some other miscellaneous helpers. And Stretchy doesn't do any of that. So other developers were like, how do I do this or that with Stretchy? And it's like, well, we don't actually do that. And it turns out when you have this API-like active record, the whole point is to make it easier so you don't have to look at all the documentation for everything all the time. But that turns out it hides a lot of the actual features of elastic search. It hides the underlying complexity. So other engineers will look at this and say, oh, I'm just going to make this active record-like thing. But it's like maybe you're missing out on a lot of the stuff you could be doing there. When we have an open source plus our internal application queries, it can be a little fuzzy deciding what should go in the open source versus what should go in the app. Obviously, all of our business logic should be in the app. But if we're using this particular style of query for text relevance or that particular style, is that worth having a convenience method for in the open source package? And then early on, we had a couple of cases where it's like, okay, we want to make this feature, but then the gem doesn't support that yet. So your open source library ideally should not be blocking any future development. So to fix that part at least, we had to make it more flexible and easier to customize within our application. And then a little bit later, we made it more flexible and extensible so that we could optimize those queries in our application. And then later, we made it easier to customize and more flexible and more extensible. And that turned out to be a huge win, actually. You can really see the process of editing and refining to get a flexible, extensible solution because ultimately it needs less code. And it makes clear what the bottom level versus mid-level building blocks are. So I actually graphed this for the stretchy repo. I took each of our master commits and looked at how many lines of code are in it. So we've done one or two rewrites. The first one, I'm not going to call it bad, but it certainly added a lot of code. And then over time, we've been able to just keep cutting that down. And at this point, it's back to being a pretty small gem. So the TLDR for that, by making this open source gem in this nice green field, we got to learn the cool new visitor pattern and see how do you do that in Ruby. For me, it was a great excuse to dive in and read a bunch of code from A-REL and Mongoid and other, like, more mature frameworks. By having the core interface be open source, that allowed us to keep that very simple and then keep our complex business logic in the app where it belongs. And it's helped us keep a good separation between elastic search queries and the internals of, like, our models and our controllers. So to wrap up, this whole thing, open sourcing has been pretty cool for us, even if, you know, it hasn't made us rock stars, we don't get, you know, a learjet to fly us to RailsConf because of how awesome we are. Even if this guy is your only watcher on GitHub, you can still get a lot of technical benefits for your organization. And even if you don't get free labor, it can help you internally with your team. It can add a little bit of friction, which can be a good thing. Can be a bad thing occasionally, but we found the trade-offs to be worth it most of the time. Thinking outside of Ruby on Rails and sort of leaving behind the conventions and crutches that gives you can make your code simpler and clearer, easier to understand. Developing open source can give you a good excuse to learn more about computers, learn more about Ruby and RubyGems and Bundler and all the other systems for how you get that back into your application. And with Greenfield Blank Slate Projects, it's a great opportunity to be a little bit more thoughtful about the code that you're writing. So I told you we'd save time for Q&A, so here we are. Any questions for me or about these open source projects or about Hire? Yeah. So the question was, Elasticsearch changed versions from 2.x to 5 because reasons. They completely changed how the query DSL works. There used to be this big separation between queries which handle relevance and then filters, which just affect your result set, but don't affect relevance. So I think it took us about three or four months to get that upgraded internally. So as part of that effort, we upgraded the open source gem. That was surprisingly easy because we have this stack and tree pattern. We basically just had to change, okay, in the tree, instead of doing this, we do that. So I can point you to the pull request for that, but I was actually shocked how simple that was. All right. Well, thank you all very much for coming. I'm Agias on Twitter and I've got the slides up on my website. Use Hired. If you're looking for a new opportunity or if you're a company looking for candidates, get online. Give it a shot. Or come work for us. It's a pretty cool company. So, all right. Thanks, everybody. Thanks.
 We got a lot to cover here, so I'm going to go ahead and get started.  This is open sourcing, real talk.  So who am I?  I'm Andrew Evans.  I work for Hired.  I've been there a little over two years.  I've worked for all kinds of different startups for like nine, ten years.  I've been working with Ruby on Rails since about 1.8 or so.  If anybody remembers some fun Twitter-related arguments back then, good times.  And I've been writing bugs in PHP or QBasic or all sorts of things for longer than I care  to remember.  So what is Hired?  Hired is the best way to get a tech job.  As a candidate, you make a profile.  We put you live in our two-sided marketplace.  And then companies ask you to interview with them, and they have to give you salary, equity,  everything else up front, and you decide if it's worth responding and having that conversation.  As a company, this is the best and fastest way to get great candidates into your pipeline.  But this is not a pitch talk, so I'm not going to focus on that.  Instead, I'm going to talk about open sourcing.  This is something that Hired's been doing basically since the beginning.  We have 46 public repos on GitHub.  A lot of those are forks where we've needed to make some modifications or thought about  modifying things.  But some of those are open source projects that maybe you can use.  These are just a couple.  We have a PubSub layer over Sidekick.  We have a query builder for Elasticsearch.  We have Fortitude, which is our CSS framework, which is built to scale.  It's built for teams.  We even have some smaller ones.  We have a RackMiddleware that will log stats from the Puma web server.  So if you're using that, it can be helpful for gathering more info.  And we have a tiny thing, which will change the color of the lights in your office.  If you're using something like a Phillips Hue, we turn our lights red when the master  build fails.  So it's a good incentive to get on Slack and figure out what happened.  I'm just going to dive deep into these two so we can talk a little bit about what producing  these projects were like.  I'm going to cover whether it would be worthwhile or not for you to do any open source work,  extracting anything out of your app, and making it public.  I'm going to talk about some of the challenges that we faced in doing that.  And I'm going to talk about how it played out across our team, like how did that affect  the dynamics.  And I'm going to try and get a little technical and talk about what we learned about Ruby and  RubyGems and Rails.  Ruby will have some time for Q&A at the end.  So why should you open source things?  Well, if you've got some logic in your application, then clearly every other Rails app is doing  pretty much the same thing.  So if we all share our code, then we could all do this stuff automatically.  When you open source something, the community will fix bugs in it.  They'll add features.  They'll document your things.  It's free labor.  It's amazing.  You'll get developer street cred.  You'll get lots of respect and appreciation.  You'll get 10K followers on Twitter and hundreds of thousands of GitHub stars.  And people will open issues and say, well, there's no bugs or problems with this.  I just want to say thank you and we love you.  So literally none of that is true.  You may have bits of application code that everyone needs across basically every Rails  app, in which case what's it like working at Thoughtbot?  But for the most part, you may be doing some pretty specialized stuff.  Most of your app is business logic, so that lives in your business app.  So real talk about our projects in particular.  I pulled some data off of Best Gems.  Best Gems basically scrapes data off rubygems.org every day and tracks like total number of  downloads, average number of downloads, et cetera.  And it gives your gem a rank out of the 130,000 plus gems on rubygems.  I can see ours are not quite in the top 100 or 1,000.  We're probably not going to be on the public leaderboards anytime soon.  Our average daily downloads, I adjusted and got rid of weekends and like holidays for  that.  But basically it's a little bit over the range we'd expect for how many times a day  we deploy to Heroku.  So hopefully these aren't all just Git push Heroku.  Oh, look, we got more downloads.  I also pulled some data off GitHub.  The number of stars and followers we have is a bit more encouraging.  There's probably some people outside of our company that care about these repos.  But we're not getting people opening issues just saying how great we are.  I found some gems that are similar on the Best Gems ranking.  So one of them is a framework that lets you build bots for Google Wave.  Does anybody remember that?  That existed for a hot second.  One of them is a command line interface to the pirate bay.  I think the founders announced a few months ago that they'd been sued and thrown in jail  too many times, so that's dead.  One of them combines ActiveAdmin and Trailblazer.  Trailblazer is the object-oriented, explicit over implicit, loosely coupled, well-organized  framework that you can add on top of your Rails app.  ActiveAdmin is literally the opposite of every one of those values.  So I'm not sure what you get by combining them.  And the last one is a command line app that gives you a random Mitch Hedberg quote.  I used to use that all the time.  I mean, I still do, but I used to, too.  So what we found is if you don't market your gem, then people don't really just find it  organically.  We haven't done a lot of promotion for these.  I don't think we post about them on our blog too often.  This may be the first talk where we've gone deep into our open-source stuff at RailsConf.  So we haven't gotten a lot of people that just found this out of the blue and said this  is the greatest thing.  Was it still worth it for us to open-source this code from our application?  I'll dive in a little bit and we'll see.  So Reactor is our enhanced interface to Sidekick.  This is how we do a lot of our background processing at Hired.  Sidekick, if you don't know, is the premier background processing library for Ruby and  Rails application that was made by Mike Purhem, that guy.  Mike is awesome.  Thank you very much.  It's been a wonderful system for us at Hired and it's been rock solid.  So the thing that we were looking at when considering Reactor was when you have, say,  a candidate updating their profile, you may want to do a couple of different things in  the background.  You might want to notify the talent advocate that's working with them.  You might want to bust a cache or update a saved search or something like that.  So with Sidekick, you have worker classes for these things and you perform a sync and  those things will happen in the background outside of your web request.  But if you're doing a whole bunch of things, then you may have a whole bunch of different  workers that you have to tell to do the thing.  So you face a choice.  Do you want to copy and paste all of those calls everywhere that you're updating a candidate's  profile?  Or you can have one worker that fires off and then that calls other workers, which might  then call other workers.  For the most part, this isn't a huge deal.  I think it's a recommended practice.  But if you're like me and you tend to forget things, then you might throw a cycle into  your workers and they'll spin up 100 or 10,000 jobs.  However many activate until you can shut down Sidekick and figure out what happened.  So basically, with that pattern, you end up with this big graph of things happening throughout  Sidekick.  You have to kind of model that in your head and it can make it a little difficult to track  down where exactly specific jobs are coming from.  So Reactor is a publish and subscribe system.  Basically you publish an event.  That event has a name, which is just a symbol.  You can pass it any arbitrary data.  There's also an extension so that you can have active record objects publish an event.  And then those objects are serialized and deserialized and available within your event  code.  So this will kick off a single Sidekick worker, which then locates all these subscribers to  that particular event name.  And it will enqueue a job for each of those.  We wrote a convenience method for our controllers called action event.  This will publish an event from the current user and it will merge in parameters or any  other data that we think would be useful for tracing, tracking, analytics, et cetera.  So what do subscribers look like?  You include this subscriberable module.  That gives you the class method on event.  So you say on event, you give it the event name and then you give it a block of what  to do when that event happens.  So in our first example, we pull out the actor, which is whatever published the event.  We get their ID and then we can bust a cache related to that particular record.  You can also set a wild card handler.  So this thing will respond to all events.  And we use that for logging site activity to Postgres or to anywhere else that we want  to have that analytics data available.  There's more options to it.  Instead of passing a block, you can pass it an event name.  If your objects and functionality are a little more complicated and you want to unit test  them, this can make it a little bit easier.  You can also pass options to it like delays and perform later.  So for example, once an interview request happens from this subscriber, then two days  later we'll send an update.  And you can do some logic in that, like figure out if it's canceled and then abort.  We also have a system for creating model-based subscribers.  So in this case, we create a subscriber called admin notifier.  This is going to live in our Postgres database.  And we define the on fire code to say what happens when this event fires.  The neat thing about this is you can have it subscribe to multiple different events  from the database.  So if I'm an admin and I want to be notified when these three different things happen,  then I can just do that through our admin interface and not have to deploy new code  to handle that.  These can be a little bit magical.  So their use has fallen off a little bit.  Finally, you can have records publish events based on the data in those records.  So for example, if we have an interview that has a start at and an end at time, then we  can tell it to publish events at those times.  We can even pass it some conditions and some other things similar to validations and callbacks.  And the nice thing is if the model gets updated, the start at or end at changes, then it will  automatically reschedule that for us.  So did it help that we took this code and instead of just building it in our application,  we put it in an open source repo?  Well when you do that, you get this nice green field project.  And it actually changes your desktop wallpaper to the old Windows XP wallpaper.  Green fields are great.  You don't have to worry about your legacy code.  You can just start developing fresh.  And it keeps your code separate outside of Kansas out of your application.  So that produces a little bit of friction to updating the library, which it turns out  can be a really good thing.  The code for reactor is kind of out of sight, out of mind for day-to-day feature development  for us.  So it discourages making small arbitrary changes.  If I want to say, like, oh, this should be calling instance methods, not class methods,  or this or that, I'm less likely to take that on because I have to context switch and then  publish to RubyGems and go through this slightly higher friction process.  Because it turns it into a lower churn section of code, it keeps the pattern and convention  clear.  This action event is used all over our application, and basically it stays the same everywhere  and it has just kept working.  We had to make some updates for Rails 5, and I'm about to make some updates for Sidekick  5, but for the most part, we don't have to think about it too much.  Also, when you have this in a public repository instead of your private application repository,  that's going on your permanent record on GitHub.  That might add just a little bit more pressure to do it right.  You might be a little more attentive to things like documenting all of your methods and making  sure that your test coverage is 100% and that your code climate score isn't going down.  Finally, when you're publishing through RubyGems specifically, you have to be a little more  thoughtful about your versioning.  You really have to think about, okay, is this going to break anything?  Is this backwards incompatible?  Or is this just a feature or a patch release that isn't going to break anything existing  but maybe provide a new feature?  Also, when you're building outside your application and you're not thinking so much about features,  it gives you a chance to think more clearly about the pattern of what you're building.  So instead of thinking about what do I need to do this feature, you think about what is  the minimum information that this open source library needs to do its job.  You also have to think about how RubyGems and Rails and Bundler and these other tools  work.  One thing I stumbled on to pretty early is that requiring Rails is harder than it sounds.  I looked through things like device and paperclip and Sidekick and they all require Rails in  totally different ways, which I found amusing.  Some of these libraries require Rails for all of their functionality and some of them  have a few Rails-specific extensions.  That caused me to dive a little deeper into, okay, well, what is a rarity?  Is that something I need?  Where do I put the require statements?  How do I manage the dependencies for that?  And then it makes you think about whether you want to use Rails-specific extensions inside  your gem that's living outside your application.  So do we want to use ActiveSupportsConcerns pattern, which is a pretty nice way of extending  objects or do we want to use the basic Ruby include and extend patterns?  And then you think a little bit more about the overall design pattern.  So when we're thinking about React or we're thinking, oh, I just want this action event  method in my code so I can just put events all over the place.  But when you're designing a gem from the ground up, it's like, well, okay, what's the name  for that pattern?  So when you go through design pattern, blog post, or the Gang of Forerbook or whatever,  you say, oh, this is a message bus.  So we're going to make channels that events get passed along.  These events are going to have arbitrary data.  Some people get this confused with event delegation, particularly if you've done a lot of jQuery.  You can have multiple listeners for particular events there.  But those actually form a chain.  And any of the listeners in that chain can abort the entire process.  With the message bus, all of the subscribers get the message regardless of what each of  them are doing individually.  So if you wanted to see some examples of subscribers and message buses written in Rails, the shiny  new action cable has a pretty well put together, well documented example of this.  I have a link here to the subscriber map.  Or if you want something a bit more core, active support notifications has been in Rails  for quite a while.  If you're wondering how all of your database statements get from the application code to  your development console, this is what's doing it.  And this has a link to subscribers in the class that's responsible for organizing them  and mapping them.  So TLDR, publishing the reactor gem, didn't make us famous, didn't really help establish  any of those selfish goals that we put out.  But it did help our team establish a convention.  It has quietly kept working for years.  And it's allowed us to write that code in a way that's smaller and more loosely coupled  and better organized.  It's also a good excuse when anyone wants to work on it to go in and look at the pattern,  look at the examples, and see how does Message Bus work, what's cool about this.  The other one I wanted to talk about was Stretchy.  This is a composable query builder for Elasticsearch.  Elasticsearch is a pretty amazing technology for full text search.  It's based on Lucene.  It's similar to Solar.  Basically, when you want something more advanced than a relational database I like statement,  then this is a good place to go.  And yes, we spent an inordinate amount of time thinking about the name for this because  Nogem is going to take off and become popular unless you have a really clever pun as the  name for it.  So I wrote the first version of this.  The goals that I was thinking of were I wanted to have an active record-ish query syntax  that we can use for Elasticsearch.  I wanted to have these immutable query objects which make handling and caching responses really  easy to manage.  I wanted this to be just a query builder for indexing our models to Elasticsearch.  We already have reactor so we can index on specific events.  We have serializers.  So I figured we'd just keep our Elasticsearch gem out of all of that business.  I wanted to make this flexible enough for pretty complicated Elasticsearch queries.  Because this is just a query builder, it's taking in a bunch of method calls and then  it's producing some JSON that we're going to send to Elasticsearch.  That means we don't really need any dependencies, which is pretty nice.  We don't have to require all of Rails for this one thing.  And the end goal of all of that is we have these composable scopes that we can use in  the same way that we're familiar with from ActiveRecord.  So here's an example of what that looks like.  This is a query that should get us back Lynn Conway.  She basically invented modern microprocessors.  So it's a good person to find in our candidate database.  You can see here we have the where method.  Pretty similar to ActiveRecord.  You can throw some fairly arbitrary parameters at it and it will figure out what the right  query is to apply those.  But then there's some more Elasticsearch specific stuff.  This match query, for example, is like the most basic full text matching relevant sorting  query in Elasticsearch.  And then we have these boost methods.  That gets into what's called a function score query.  It lets you tweak how your search results are going to be ranked.  When you look at the query that's generated, it's this giant message JSON thing.  When I first started working with Elasticsearch, I tried to figure out how to make these giant  JSON documents composable so we could mix and match them in different ways.  Turned out to be really hard, which is what inspired this whole ActiveRecord-like approach.  So some other example methods.  I won't get too deep into these, but you tell it like what index you're going to search,  what type you want to get back.  And then you can use where for filters.  You can use match for full text matching.  You can or multiple filters together by using the should clause.  This comes straight from the Elasticsearch DSL.  And then you can add boost to tweak your relevance algorithms.  And finally, if what Stretchy has isn't enough, then you can go lower level.  You can say this is just a query and then throw some raw JSON at it.  And it will figure out where that needs to go in your final query that's going to be  sent over.  Elasticsearch has a whole lot of stuff in it.  It has about 7 million tools for doing different kinds of searches and figuring out relevance  and producing the set of results that you want.  So it became obvious pretty early we couldn't support all of that.  So the goal was just to get to a system where we could do what we needed to in the app and  then hopefully expand that as we got more and more features and hired.  So how do I get started making an active record chainable interface?  I pulled out this 2010 blog post that I've had bookmarked forever, RailsTips.org.  Thanks, John Unimaker.  This stuff still works.  It's still pretty much the same in Ruby 2.3.  Hopefully you have a class that class has some data, maybe an array of nodes, let's say.  And each of your chainable methods returns a new instance of that class with some new  stuff added to it.  So when you get down, you're like, okay, we're building up this array and then we're  going to compile it to this JSON query.  What does that look like?  Is there a name for that?  How does it work?  First out, it's called the visitor pattern.  I was looking around at how other repos do this.  I looked at active record and oh, that uses A-REL.  And A-REL has a visitor pattern.  So that has a collector class and then it has a tree manager for figuring out what was  collected and compiling it.  And then it has multiple different visitor classes to take that tree of abstract nodes  and compile it into a query for, say, MySQL or Postgres or Oracle, I guess, if you're  into that kind of thing.  This is an example of what that tree of abstract nodes is going to look like.  You have a select node and then that can have multiple child nodes, like what are you selecting,  where are you getting it from, and then any filters that are going to be applied to your  query.  And then those filters can have child nodes and those have child nodes.  So the visitor will traverse all of those and compile it into the final string.  So Stretchy didn't need that full stack.  There's really only one elastic search.  We decided because time, we only wanted to support the latest version.  So we can basically combine the visiting and compile steps.  But the pattern was still pretty much the same.  Some challenges we ran into in having this query builder live outside of our repo and  kind of its own thing.  It was a little confusing to the rest of the team initially.  If you look at other gems for integrating elastic search into your Rails app, they do  a lot more.  They have search Rails and search kick and chewy.  They all tie into active model.  They have some other miscellaneous helpers.  And Stretchy doesn't do any of that.  So other developers were like, how do I do this or that with Stretchy?  And it's like, well, we don't actually do that.  And it turns out when you have this API-like active record, the whole point is to make  it easier so you don't have to look at all the documentation for everything all the time.  But that turns out it hides a lot of the actual features of elastic search.  It hides the underlying complexity.  So other engineers will look at this and say, oh, I'm just going to make this active record-like  thing.  But it's like maybe you're missing out on a lot of the stuff you could be doing there.  When we have an open source plus our internal application queries, it can be a little fuzzy  deciding what should go in the open source versus what should go in the app.  Obviously, all of our business logic should be in the app.  But if we're using this particular style of query for text relevance or that particular  style, is that worth having a convenience method for in the open source package?  And then early on, we had a couple of cases where it's like, okay, we want to make this  feature, but then the gem doesn't support that yet.  So your open source library ideally should not be blocking any future development.  So to fix that part at least, we had to make it more flexible and easier to customize within  our application.  And then a little bit later, we made it more flexible and extensible so that we could optimize  those queries in our application.  And then later, we made it easier to customize and more flexible and more extensible.  And that turned out to be a huge win, actually.  You can really see the process of editing and refining to get a flexible, extensible  solution because ultimately it needs less code.  And it makes clear what the bottom level versus mid-level building blocks are.  So I actually graphed this for the stretchy repo.  I took each of our master commits and looked at how many lines of code are in it.  So we've done one or two rewrites.  The first one, I'm not going to call it bad, but it certainly added a lot of code.  And then over time, we've been able to just keep cutting that down.  And at this point, it's back to being a pretty small gem.  So the TLDR for that, by making this open source gem in this nice green field, we got  to learn the cool new visitor pattern and see how do you do that in Ruby.  For me, it was a great excuse to dive in and read a bunch of code from A-REL and Mongoid  and other, like, more mature frameworks.  By having the core interface be open source, that allowed us to keep that very simple and  then keep our complex business logic in the app where it belongs.  And it's helped us keep a good separation between elastic search queries and the internals  of, like, our models and our controllers.  So to wrap up, this whole thing, open sourcing has been pretty cool for us, even if, you  know, it hasn't made us rock stars, we don't get, you know, a learjet to fly us to Rails Conf because of how awesome we are.  Even if this guy is your only watcher on GitHub, you can still get a lot of technical  benefits for your organization.  And even if you don't get free labor, it can help you internally with your team.  It can add a little bit of friction, which can be a good thing.  Can be a bad thing occasionally, but we found the trade-offs to be worth it most of the  time.  Thinking outside of Ruby on Rails and sort of leaving behind the conventions and crutches  that gives you can make your code simpler and clearer, easier to understand.  Developing open source can give you a good excuse to learn more about computers, learn  more about Ruby and RubyGems and Bundler and all the other systems for how you get that  back into your application.  And with Greenfield Blank Slate Projects, it's a great opportunity to be a little bit  more thoughtful about the code that you're writing.  So I told you we'd save time for Q&A, so here we are.  Any questions for me or about these open source projects or about Hire?  Yeah.  So the question was, Elasticsearch changed versions from 2.x to 5 because reasons.  They completely changed how the query DSL works.  There used to be this big separation between queries which handle relevance and then filters,  which just affect your result set, but don't affect relevance.  So I think it took us about three or four months to get that upgraded internally.  So as part of that effort, we upgraded the open source gem.  That was surprisingly easy because we have this stack and tree pattern.  We basically just had to change, okay, in the tree, instead of doing this, we do that.  So I can point you to the pull request for that, but I was actually shocked how simple  that was.  All right.  Well, thank you all very much for coming.  I'm Agias on Twitter and I've got the slides up on my website.  Use Hired.  If you're looking for a new opportunity or if you're a company looking for candidates,  get online.  Give it a shot.  Or come work for us.  It's a pretty cool company.  So, all right.  Thanks, everybody.  Thanks.
Open_Sourcing_Real_Talk_2_0_47.png Open_Sourcing_Real_Talk_2_48_311.png Open_Sourcing_Real_Talk_2_312_479.png Open_Sourcing_Real_Talk_2_480_1343.png Open_Sourcing_Real_Talk_2_1344_2231.png Open_Sourcing_Real_Talk_2_2232_4847.png Open_Sourcing_Real_Talk_2_4848_6539.png Open_Sourcing_Real_Talk_2_6540_9839.png Open_Sourcing_Real_Talk_2_9840_10307.png Open_Sourcing_Real_Talk_2_10308_10511.png Open_Sourcing_Real_Talk_2_10512_11015.png Open_Sourcing_Real_Talk_2_11016_11435.png Open_Sourcing_Real_Talk_2_11436_13403.png Open_Sourcing_Real_Talk_2_13404_13835.png Open_Sourcing_Real_Talk_2_13836_19367.png Open_Sourcing_Real_Talk_2_19368_19811.png Open_Sourcing_Real_Talk_2_19812_19979.png Open_Sourcing_Real_Talk_2_19980_27959.png Open_Sourcing_Real_Talk_2_27960_29123.png Open_Sourcing_Real_Talk_2_29124_29615.png Open_Sourcing_Real_Talk_2_29616_29927.png Open_Sourcing_Real_Talk_2_29928_42767.png Open_Sourcing_Real_Talk_2_42768_43343.png Open_Sourcing_Real_Talk_2_43344_45251.png Open_Sourcing_Real_Talk_2_45252_45911.png Open_Sourcing_Real_Talk_2_45912_47519.png Open_Sourcing_Real_Talk_2_47520_47567.png Open_Sourcing_Real_Talk_2_47568_47747.png Open_Sourcing_Real_Talk_2_47748_47903.png Open_Sourcing_Real_Talk_2_47904_48227.png Open_Sourcing_Real_Talk_2_48228_48407.png Open_Sourcing_Real_Talk_2_48408_48779.png Open_Sourcing_Real_Talk_2_48780_48899.png Open_Sourcing_Real_Talk_2_48900_49031.png Open_Sourcing_Real_Talk_2_49032_49127.png Open_Sourcing_Real_Talk_2_49128_49367.png Open_Sourcing_Real_Talk_2_49368_49583.png Open_Sourcing_Real_Talk_2_49584_49655.png Open_Sourcing_Real_Talk_2_49656_49775.png Open_Sourcing_Real_Talk_2_49776_49859.png Open_Sourcing_Real_Talk_2_49860_49907.png Open_Sourcing_Real_Talk_2_49908_49955.png Open_Sourcing_Real_Talk_2_49956_50003.png Open_Sourcing_Real_Talk_2_50004_50051.png