Building maintainable command-line tools with MRuby
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Hodel, Eric
mruby and mruby-cli makes it possible to ship single binary command-line tools that can be used without setup effort, but how difficult is development in MRuby? In this talk we'll explore how to work with MRuby and mruby-cli. We'll cover the differences in development patterns when using MRuby, the maturity of the MRuby ecosystem, and development patterns when using mruby-cli to build a command-line tool.
 Hello and welcome to Building Maintainable Command Line Tools with mruby. I am Eric Hodle and this talk about is working with mruby and mruby CLI. Without me I am a mruby committer, mruby jems committer, rake committer and the author of many gems. I am also on vacation from most of my open source work. Instead of working on mruby in my spare time I have been spending a lot of time making shelves like these. The far one there is still a work in progress, still trying to put on a varnish to fill those gaps in so it is smooth. And I work for FASLI, real time content delivery network. This is our amazing website, FASLI.com with Gordo, one of the dogs of FASLI. FASLI supports many open source projects including hosting the mruby lang website, mruby version downloads and supports mruby gems. Last month FASLI served 425 terabytes of data and 8 billion requests supporting mruby gems for free. And I would also like to highlight the work of David Radcliffe and the rest of the mruby gem team for speeding up gem downloads especially for users outside of the United States and they should be making a blog post about that work real soon now. And of course you can follow FASLI on Twitter. So part one, my exploration of mruby and mruby CLI started with a web API wrapper. We're building a new product of FASLI with new APIs and I wanted to, I wanted an easy way to interact with the APIs so that I could demonstrate that they were usable and that they were interacting the way that we, that was, that made sense. And why not choose mruby to write the wrapper? Well when we use mruby we have to install the correct version depending on what mruby features you want to use. Then you need to install the gems that you're going to depend on and then you have to keep both the mruby version and the gems up to date for as long as you're supporting the tool. And mruby is very similar to mruby so we can start with a prototype and mruby, port that to mruby and then continue our maintenance in mruby. And what about mruby that makes it more suitable than mruby? mruby is a lightweight ruby implementation. You only need to use as much of the features of ruby as you need which keeps the language and the size smaller. It's also designed for embedding. You can run, you can run ruby on low resource hardware or embed it into other software without all the hassles of embedding mruby. Like mruby, mruby has libraries and it calls them gems. mruby ships with a limited standard library so you can use a gem to get an array or use a gem to get an array with all of the features of mruby. There's also standalone mruby c libraries, some of which are c extensions that wrap c libraries like the mruby gem. At past ruby kikes there have been many other talks on mruby if you want to see other applications and you can find recordings of these on the ruby kikey website. Especially interesting one is the one at the top making high tech seat in mruby. It's about bidet built with lego mind storms. And you can find recordings of all these talks on the ruby kikey websites. There's also two talks on mruby at this year's ruby kikes. Both are in ballroom A. One is in the first session after lunch today on evaluating ruby without ruby. And one is the very first session tomorrow about mruby on small devices. Since mruby is compiled unlike c ruby, how do you use it? The first thing you do is you edit your build config rb. Like c ruby where you can use any gem at any time, in mruby you must pick in advance. In the build config you pick which mruby gems you want to use. You also choose which compiler to use. mruby cli uses this for cross compiling to all the different platforms. And you can also specify library include paths in case you need to link with other c libraries. If you're writing in mruby gem instead of embedding mruby you'll fill out an mruby gem specification. mruby cli applications use the mruby gem specification for specifying the executable name and the test files. The mruby gem dot rake is where this mruby gem specification lives. Like a ruby gem's gem specification it contains the gem metadata, the dependencies and the list of binaries. It includes information like the name, the version, license, test files and so on. You can also specify the gems you depend on and the binaries that get created. And after you have these files configured you run rake to build mruby and all the executables you need. And once rake is complete you'll see several executables in the bin directory. If you're using mruby cli you'll see the name of the command line tool you specified in the gem specification. You'll also have mruby and mrb executables that have all the gem dependencies you need compiled in. And you can use these to run one-off tests. And you'll have an mrb test if you have enabled testing for this build. The goal of mruby cli is to allow rubyists to build single file executables in a familiar environment. Last year Terence Lee and Zachary Scott, David talked about mruby cli that showed the basics of the tool and how it worked. And the very basic description of mruby cli is that you can write ruby and release software on linux, on os10 and on windows. The typical development loop of using mruby cli is to start by writing some mruby. You compile the code that you've written and you run the tests and go back to the start if your tests don't pass. And once you have enough of your features built and tests passing then you can release the executables. And when I first heard about mruby and mruby cli, I thought it sounded pretty easy, but I was sure there would be some challenges involved when using a different ruby implementation. The first one is that mruby is not the same as cruby. The implementation doesn't support doing everything that you could do in cruby like I'm familiar with. The first problem I had was the modular standard library. mruby's core library has classes like string and array and hash, but they only support the minimum number of methods. For the full cruby experience, you have to add extension gems like mruby array ext. And it's very easy to forget to do this if you're developing your own mruby gem and you forget to add these extension gem dependencies. And without them, the gem won't work in another project. So it might work in one spot and not another. mruby also has, mruby is also a smaller language outside the standard library. There are no predefined globals such as load path since everything is compiled in, nor any of the globals inherited from Perl or shell scripts. So some libraries you might want to port from mruby may depend on these and you'll have to work around them. I also had a problem with here document support when porting a library from mruby and I haven't yet reproduced it on its own, so I need to file a bug for it. And while you can use keyword arguments to call methods, you can't define methods that automatically check keyword arguments like you can in mruby. If you haven't explored keyword arguments yet, there's a talk today in the second session after lunch in ballroom C on keyword arguments to check that out. Also some of the G sub replacements don't work in mruby, so you may need to change your regular expressions. Back traces in mruby also seem to be less useful than in mruby. I feel like I did more exploring in the C source to determine why an exception was raised. I'm not sure that's due to unfamiliarity with mruby or due to other reasons. There are many pure ruby libraries that are already written in CRuby that you'll want to use in mruby to make your life easier. And overall this is a pretty straightforward process provided you beware there are restricted syntax. Unfortunately though, porting the tests for these is much harder due to differences in the way tests run and we'll cover that in the future in this talk. The existing gems for mruby feel a lot like working with libraries from the ruby one six days. And I think this is because there just haven't been enough people working in mruby to build up the libraries that apply well to many different types of problems. So an mruby library often has a very narrow focus, so you'll either have to do a bunch of hunting to find the library that fits your use case or you'll need to submit patches to get the library to work for you. So far I've submitted some patches to all of these libraries. For some libraries I've sent multiple patches. They're usually pretty small which is nice. And the last challenge I'll cover for mruby is getting the build information out of mruby. Since mruby is compiled it needs to link in C libraries to compile successfully. With mruby CLI you're also cross compiling these C libraries to many different platforms so you need to use the correct C libraries. Unfortunately it's hard to extract the cross compiler information out of the mruby build system without loading all of its rake files which is pretty slow. And I think improving this will make mruby's build system more flexible but I haven't yet tried to or haven't done any work on this yet. mruby CLI like mruby was a new tool for me so it took some time for me to learn to use it the right way and here's some of the larger challenges that I had with it. The first one was Docker. The mruby CLI build system is built to top Docker and one of the really great things about this is it's ready to cross compile to Linux, the OS 10 and to Windows without any extra setup. Unfortunately Docker is kind of clumsy the way mruby CLI uses it. Typically Docker is used to run a long lived service but mruby CLI only needs to keep the container running just enough to finish compiling or to run the tests. But I think out of all the options Docker is the best one available because there really is no extra work to get started with building command line tools. In order to start a build of your command line tool you run Docker compose run compile and this is far too long to type even if you use your shell history well. When there's a problem with the compile it's hard to debug from the outside. You have to start a shell inside of Docker then figure out which commands you were running and then run those to reproduce the problem before you can start to fix it. And when you're working with this Docker shell you don't have your familiar development environment and there's no shell history from the last time you were doing this so it's harder than usual to get things fixed. And the biggest challenge I've had with mruby CLI was its build system. Most of these problems came from the way the tasks were organized, the tasks for building your executable and the tasks for building mruby were all mixed together into a single rate command. And this made it hard for her for you to add your own tasks and then have them run in the right spot. For example before compiling the mruby curl extension you need to have the curl CLI where installed for every platform you're cross compiling to and by default that's very difficult. And due to these difficulties I made some improvements to the build tasks. The first was to build mruby separately from all the other work that must be performed inside Docker. This means I can cross compile the CLI before time. The other improvement was to have a rake task that exists outside Docker and starts Docker for me. This allows me to perform extra tasks that don't need Docker to run and are easier for me to debug. So the new build system runs rake three times. First the outer rake tasks run and then once they're done they start up the inner rake tasks inside of Docker. The inner rake tasks perform things like cross compilation or any other setup that can only occur inside Docker. And when these are done mruby's rake tasks are run which builds your command line tool. Since the outer tasks don't need Docker I use them to do things like download and unpack mruby to get it ready to compile. There's also some global test setup tasks for reason I'll cover later. And finally there are release tasks to publish your finished gem executable. The inner tasks that run inside Docker cross compile the CLI you have configured and then invoke rake again to compile mruby and run the tests. There's some additional benefits to my changes to the build system. It's now much easier to add hooks for custom tasks since there are well defined tasks for different phases of the build. I also made rake test faster by only compiling for the host platform when running tests. There's no need to build every platform when you can only run tests on the host Docker platform. It's also pretty likely you'll need to use CLI as a dependency of an mruby gem in your project and there's two options. You can include them in your command line tool. The first is static linking and this embeds the library inside your command line tool and keeps you with a single file executable. The advantage is you don't have to provide any separate instructions for installing the correct CLI for your tool. The downside though is that you need to re-release your command if there are any vulnerabilities in those CLI. This also makes your executable a little larger. The second option is to dynamically link your CLI. The advantage here is that vulnerability management is much easier. The user should be upgrading their own libraries for security vulnerabilities so there's less for you to do. The disadvantage is this adds an external dependency so you're no longer truly a single file. The final challenge I had with mruby CLI was cross-compiling libraries. mruby CLI can cross-compile your executable for up to six different platforms and this means your CLI also need to be compiled for all of those six different platforms. For example, for libcurl you can't just install the libcurl development package inside Docker and be done since this library doesn't work on OS X or Windows. Instead you have to download the source, unpack it, cross-compile for each platform you're going to be releasing on, and then set up mruby's compiler to use those libraries that you've built. To help myself out, I wrote a tool to automate cross-compiling dependencies. You can configure the library to cross-compile with a few values. The first is the name, curl here. The release name is used to create the source directory since libraries don't necessarily use consistent naming styles. The URL is where the source is downloaded from, and then optionally you can set extra configure flags. For example, I don't yet have open SSL cross-compiled so I disabled it for now. In part two, I'll discuss the origins and the structure of the command line tool. So my exploration of mruby CLI started with a tool that our customer support team could use to configure a new product we were developing. Our service was and still is API only so there's not yet any web UI for anyone to use. The service has a complicated setup so I wanted to automate common use cases to make it easy to avoid errors. I also wanted to have a tool that customer support could use and also send to users to reduce the burden of supporting this new product. The solution to these problems is to write a tool that could automate the setup, validate any inputs, and reduce the number of errors the users would encounter. It would have the benefit of also giving design feedback on the API and we could use that to make further improvements. Before exploring mruby CLI, I started with some prototypes in CRuby. The first was a single file script that primarily served as documentation. Script was very easy for me to write and customer support could modify it for their needs but it wasn't flexible enough to handle all the ways they needed to interact with the API. As the needs of customer support grew, I split this single script into multiple scripts. Each script had its own purpose, for example, listing options, performing initial setup, or enabling or disabling features. This was more flexible and I made the point of making the scripts well documented so it would be easy to modify for writing future scripts. And finally, I extracted a couple classes from the scripts for reuse. This included a basic implementation of JSON API, set as our API transport specification, and one for argument parsing using Ruby's option parser as we have many commands using the same arguments. So what lessons did I learn from the prototypes? The first was that documentation in the tool was very important. We had not yet budgeted time for API documentation of the service, so documentation via the command line tool was very useful. Roughly half the lines in the script ended up being comments. This also meant our support staff could figure out what the tool was doing and suggest changes that would improve their lives. Keeping the script simple made everything more understandable. None of the scripts had any external dependencies, so they were easy to install just by checking out the repository they lived in. Most scripts were small and had a single well-defined purpose. And it was okay to have one or two larger scripts that automated an important action like initial setup. Ultimately, though, they were hard to maintain. Our development environments install an old version of Ruby without keyword arguments by default. So we needed to install a modern Ruby version separately, and customer support also would add or edit some of the existing tools to make their lives easier, but then they would end up with multiple independent versions that were hard for the both of us to support. I decided to investigate a switch to M-Ruby-CLI to address the maintenance burden while trying to keep the maintainability high. I decided to model my command line tool off of the pattern that RubyGems uses. It has a set of subcommands with one subclass per subcommand, and they both use option parser to parse arguments. Each has a common setup method that picks which options are used for the command, and then after parsing arguments, the execute method runs the command and handles any errors. For example, if we got a command that lists conditions, the setup method would require a service and a version that the conditions are attached to, and then the execute method would perform the API operations to fetch the conditions from the service and then display them. I decided to port M-Ruby-OptParse from the C-Ruby standard library for option parsing. I chose this because it was familiar to me and to other Rubyists, and it has good documentation. OptParse performs automatic type checking and argument completion for you, which means less code to write to check these by hand. OptParse even offers shell completion as a standard feature, which is very nice, and all of these would make parsing arguments easy and straightforward. For API requests, I chose M-Ruby-Curl. I evaluated several other HTTP libraries for M-Ruby, and this one seems to be the most mature, mostly because it uses Curl. It supports both HTTP, which we use in our development environment, and HTTPS for accessing our production systems. The library also supports persistent connections, which is especially important for HTTPS requests. The only downside to is the lib-curl dependency, which must be cross-compiled and linked into the tool. For testing, there are two M-Ruby gems that seem to be the most popular. The M-Ruby-M-Test gem provides a library for assertions, while the M-Ruby-Test gem provides the tool that runs the test cases. While using these gems is pretty easy, setting up the test cases can be pretty painful. Testing gave me the most frustration when building the command line tool. It's the most important piece of writing maintainable software, but due to the unfamiliar environment, there were some adjustments I had to make in order to understand testing with M-Ruby-CLI. The M-Ruby-M-Test gem is very much like the mini-test library that ships with C-Ruby. You make a subclass of the test class, you write test methods the same as mini-test, set up and tear down methods are used for common test setup in the same test case. But you have to remember to start the tests with mtestunit.new.run at the bottom of your test case, since M-Ruby doesn't have at exit without an extra gem. Since M-Ruby is compiled, you can't just run the test directly. Instead, you use the M-Ruby-Test gem, which provides the test runner. When the tests run, a new M-Ruby VM is created for each test file. This gives complete isolation in between tests. The design of a test runner has some complications to it, though. The first is that creating a global test setup or global test helper requires setting test preload in your M-Ruby gem specification. Without this, the helper won't be available to any of your tests, because M-Ruby just thinks it's a separate test file that it's supposed to run. I'm still trying to get this to work, because I only discovered it recently. The second complication was Docker. M-Ruby CLI uses Docker compose, but most of the documentation is written for the Docker command. So it took some time for me to figure out how to set up DNS records so that tests could use other services on my local machine. I would have preferred to mock all the interactions, but without an easy test helper and a mocking library. It was just easier for me to use the development services directly. The final complication was that M-Ruby CLI was slow to start because it compiled all platforms before starting the tests. This made the feedback loop very long, even though you might be making a very small change. To work around the global test setup, I chose to perform some of the tests set up before starting any of the tests. I used the improved build structure to do the setup outside of Docker before starting any of the tests, because I have better and more capable libraries there. Since I only learned about test preload recently, I'm passing the data needed for global setup into the tests using environment variables that each test can look up when it needs them. And to improve startup, test startup speed, I adjusted the rake tasks to only compile for the host platform, which is what the tests use. This meant I only had to compile one platform instead of seven. M-Ruby CLI also supports bin tests, which are integration tests. Since I prefer strong unit tests, I haven't explored these much yet. For an integration test, you run your command, and then it captures the output, and then you can assert what the output looks like. And despite all these difficulties, though, how would you choose to use M-Ruby CLI? And why not use Go or one of these other tools that has the same capability? I'm primarily a Rubyist, and I work with a team of Rubyists. And while we aren't opposed to learning new things, M-Ruby keeps us in our comfort zone. For example, when I had problems with the test setup, I could use C-Ruby that I was more comfortable with to get the job done. And remember that I want to have a tool that I can deliver to fastly customers. So the ability to work where I'm most comfortable means that it's easier to ship quality software. By limiting the number of things I need to learn at once, my stress level is lower and my ability to deliver is higher. And now we approach the end. What would I like to see that would make M-Ruby and M-Ruby CLI easier to use? For M-Ruby CLI, I'm going to contribute back the improvements to the build system I've made. I've already had some discussion with the maintainers about the improvements I've made and what benefits they bring, and they've seen positive on them so far. So documentation for M-Ruby CLI could also use some improvement, as it isn't necessarily obvious where to start with a new command line tool or what complications you might run into or which libraries you might want to check out. I'd also like to make upgrades to new versions of M-Ruby CLI easier, but I'm not yet sure how that will work yet. Accurate backtraces would be very helpful. Right now you have to pay extra close attention to figure out which line M-Ruby meant the error occurred on. I also want to separate the loading of the build configuration from generating the rake tasks. This will make it easier to integrate M-Ruby's build system with M-Ruby CLI's build system. And finally, I'd like to see keyword arguments and methods. M-Ruby's CAPI has a fancy implementation of argument checking, so I'm unsure how this will work with keyword arguments. In C-Ruby, I'd like to see the M-Ruby get args ported over from M-Ruby. This is a function of the CAPI that extracts method arguments when a Ruby method calls a C function. In M-Ruby, the method allows you to specify the types of the arguments, so you get automatic type checks and automatic conversion. The equivalent method in C-Ruby is RB scan args, but it's much less expressive since it only checks a number of arguments, so you have to do all of your type conversion by hand. Also, finding the test preload was difficult, so I like it to make it automatic, so it's easier to write tests. Also M-Ruby test runs all the tests and doesn't have an option to run just one test, so or subset of tests. And without this, it's much harder to make large changes since you have to go and filter your tests by hand to figure out which test values are important. Thank you. Are there any questions? Yeah, so this was, so the question was, what is passing data out of Bandmeat here for the testing? And for that, since you can't pass data across Docker without it going through a file, I had to go and do some setup, write a file, and then load that file inside of M-Ruby to get the test setup data that I needed. And so I did that through argument, or through environment variables. So the question was, how robust is support for linking dynamic libraries? Like, do you, can you specify paths and all that? And yes, you can specify all those paths with, there's a, the support for all those through the M-Ruby gem specification stuff. So you can say, like for example, for all of the cross-compiling work I did, it'll build it into per platform directories, and then I can say, go look at x86, Linux, this directory, and that's where curl will be. So that, yeah, that's all very easy. Can the user pass something in if it's not in the expected place? I think you have to edit the build config. I don't know if you can override with CC, like the CC or LD flags or LD library path or so forth. Did I notice much memory usage difference between CRuby and M-Ruby? I did not measure that because the CRuby prototype that only runs for a few seconds and does very little, although I did do a test. One of my M-Ruby curl patches was to add persistent connection support by reusing the curl connection, and I believe I ran it for a couple days, just repeatedly checking a local server, and I don't believe the memory usage grew over a couple megabytes. So it's very, it's very nice. How long did it take to iterate from the first solution to the last one? So I'm familiar with building command line tools in CRuby, so a lot of the difficulty was figuring out how to use M-Ruby and how does this piece work and how does that piece work. In terms of writing the implementation in M-Ruby, it was very straightforward after I was like, okay, so I figured out how to use curl, how to get opt-pars going, and then everything else after that was very straightforward. I probably spent three or four days writing the implementation of the command line tool, or at least write one command was maybe a couple days. The question was what does an upgrade path look like for a customer, and how do you mean exactly, like upgrading? So how would you release a new version of the command line tool? So since it's a single file, you can upload the binary and say download this. You can also do things like maybe releasing on Linux, you can wrap it up in a package, which would be great if you had, if you're using dynamic linking, because you would say I need curl and open SSL and whatever else. So it will depend like, do you want to, it'll also depend on things like how exactly you want to release it. The easiest one is just, hey, here download my new version. I suppose you could integrate a thing like Sparkle that OS X uses, but I haven't explored any of that yet. Command line tool. So what are some notable command line tools built with Mrooby CLI? There was a talk at RubyKaigi about Hakonewa, which was a tool for using containers, setting up containers, I believe. I don't have any other examples off the top of my head. Maybe Terrence might know. So Terrence says there is a lot of stuff in Japan less in the US, and one of the more notable ones is a system monitoring tool. I believe there was a RubyKaigi talk on that one. I don't, it might be that, might be, yeah. What did I do to cross compile lib curl? So the only thing that I had to do was extract which compiler, compiler configuration, Mrooby CLI provided that by default. So I extracted those compiler information and then plugged those into the call to configure that, the configure script that came with the lib curl source. I can show you after the talk if you'd like to see. So how different is Mrooby from CRuby and how different is CRuby from Ruby? So CRuby is synonymous with MRI. So yeah. So I originally gave this talk at RubyKaigi, so I use the, they're familiar with CRuby there. And Mrooby and CRuby are syntax-wise. They're very similar. There's a few things that are bugs probably. There's a ISO specification for the Ruby language and it's, Mrooby is built off of that specification. Like I said, there's a few things that are missing like automatic argument checking for keyword arguments. You can just do star star args and that's about it. Global variables are different. Some libraries may not support a thing you need. One of my patches was to on a Geruma where it didn't give, it didn't expose optional captures properly so it had empty strings instead of nil. So there's a few things you're going to run into that are like, oh, somebody didn't fill out all of the features yet. But overall it's going to be very familiar. So how did I handle the non-availability of require? So part of it is you just list your Mrooby gems in your specification and those get compiled in automatically. There are some metaprogramming tricks you can do with require in CRuby that you can't do in Mrooby like the Ruby, sorry, the Ruby gems command library will opportunistically load just the sub-command that you need. But since it's all compiled in and it's all loaded at once, you don't need to worry about that but it requires you to adjust your thinking about some of the metaprogramming things you do like this thing loaded and it automatically gets registered while you have to just register it manually. So as far as regular programming, it's fine if you want to do some of those metaprogramming tricks that sit on top of require doing some stuff and having callbacks like this subclass was created. You're kind of like, well, I don't need to worry about that anymore because it's just going to be there and I'll just have a thing that pre-registers it manually. How many platforms did I build it for and why do all these cross-compilation? So Fastly's customers operate on many different operating systems and Mrooby CLI provides us out of the box. I disabled Windows support just because I wanted to save time and I'd come back to it later. So right now it's building for four platforms, Linux and OS X 64 and 32-bit. There's really no reason to turn it off unless I wanted to test it on OS X and the Docker platform and I had a development environment that used Linux. So it was just easy to be like, I'll just leave those on and then I can make sure that I don't have any libcurl bug that didn't compile something correctly or whatever. But yeah, it's kind of like Mrooby CLI. That's all that stuff up for you so there's no reason to disable it unless you want to save some time. So the question was, based on all the things I want to add to Mrooby, how easy is it to contribute to Mrooby compared to Crooby? Because even Mrooby's internals are broken up into a bunch of little gems like Mrooby array extension and so on. So this makes it much easier to figure out where you want to make a change. So if I wanted to fix a bug that was in some array method, then I would be able to go and find the spot to make the change much easier, which means that it's less of a burden of like, how do I make this change and where does it go with Crooby? It's one big project with all the files mixed together so it's much harder to figure out how to make the change. So I think I need to make this change, where does it go is much easier to do. So Terrence pointed out that Mrooby is all on GitHub with a pull request workflow so there's not the complication of Crooby's hybrid Git subversion issue tracking. The community is smaller but reasonably active and some other stuff that I forgot. I believe there are no more hands so thank you for your time.
 Hello and welcome to Building Maintainable Command Line Tools with mruby.  I am Eric Hodle and this talk about is working with mruby and mruby CLI.  Without me I am a mruby committer, mruby jems committer, rake committer and the author  of many gems.  I am also on vacation from most of my open source work.  Instead of working on mruby in my spare time I have been spending a lot of time making  shelves like these.  The far one there is still a work in progress, still trying to put on a varnish to fill those  gaps in so it is smooth.  And I work for FASLI, real time content delivery network.  This is our amazing website, FASLI.com with Gordo, one of the dogs of FASLI.  FASLI supports many open source projects including hosting the mruby lang website, mruby version  downloads and supports mruby gems.  Last month FASLI served 425 terabytes of data and 8 billion requests supporting mruby gems  for free.  And I would also like to highlight the work of David Radcliffe and the rest of the mruby  gem team for speeding up gem downloads especially for users outside of the United States and  they should be making a blog post about that work real soon now.  And of course you can follow FASLI on Twitter.  So part one, my exploration of mruby and mruby CLI started with a web API wrapper.  We're building a new product of FASLI with new APIs and I wanted to, I wanted an easy  way to interact with the APIs so that I could demonstrate that they were usable and that  they were interacting the way that we, that was, that made sense.  And why not choose mruby to write the wrapper?  Well when we use mruby we have to install the correct version depending on what mruby  features you want to use.  Then you need to install the gems that you're going to depend on and then you have to keep  both the mruby version and the gems up to date for as long as you're supporting the  tool.  And mruby is very similar to mruby so we can start with a prototype and mruby, port  that to mruby and then continue our maintenance in mruby.  And what about mruby that makes it more suitable than mruby?  mruby is a lightweight ruby implementation.  You only need to use as much of the features of ruby as you need which keeps the language  and the size smaller.  It's also designed for embedding.  You can run, you can run ruby on low resource hardware or embed it into other software without  all the hassles of embedding mruby.  Like mruby, mruby has libraries and it calls them gems.  mruby ships with a limited standard library so you can use a gem to get an array or use  a gem to get an array with all of the features of mruby.  There's also standalone mruby c libraries, some of which are c extensions that wrap c  libraries like the mruby gem.  At past ruby kikes there have been many other talks on mruby if you want to see other applications  and you can find recordings of these on the ruby kikey website.  Especially interesting one is the one at the top making high tech seat in mruby.  It's about bidet built with lego mind storms.  And you can find recordings of all these talks on the ruby kikey websites.  There's also two talks on mruby at this year's ruby kikes.  Both are in ballroom A. One is in the first session after lunch today on evaluating ruby  without ruby.  And one is the very first session tomorrow about mruby on small devices.  Since mruby is compiled unlike c ruby, how do you use it?  The first thing you do is you edit your build config rb.  Like c ruby where you can use any gem at any time, in mruby you must pick in advance.  In the build config you pick which mruby gems you want to use.  You also choose which compiler to use.  mruby cli uses this for cross compiling to all the different platforms.  And you can also specify library include paths in case you need to link with other c libraries.  If you're writing in mruby gem instead of embedding mruby you'll fill out an mruby gem  specification.  mruby cli applications use the mruby gem specification for specifying the executable  name and the test files.  The mruby gem dot rake is where this mruby gem specification lives.  Like a ruby gem's gem specification it contains the gem metadata, the dependencies and the  list of binaries.  It includes information like the name, the version, license, test files and so on.  You can also specify the gems you depend on and the binaries that get created.  And after you have these files configured you run rake to build mruby and all the executables  you need.  And once rake is complete you'll see several executables in the bin directory.  If you're using mruby cli you'll see the name of the command line tool you specified in  the gem specification.  You'll also have mruby and mrb executables that have all the gem dependencies you need  compiled in.  And you can use these to run one-off tests.  And you'll have an mrb test if you have enabled testing for this build.  The goal of mruby cli is to allow rubyists to build single file executables in a familiar  environment.  Last year Terence Lee and Zachary Scott, David talked about mruby cli that showed the basics  of the tool and how it worked.  And the very basic description of mruby cli is that you can write ruby and release software  on linux, on os10 and on windows.  The typical development loop of using mruby cli is to start by writing some mruby.  You compile the code that you've written and you run the tests and go back to the start  if your tests don't pass.  And once you have enough of your features built and tests passing then you can release  the executables.  And when I first heard about mruby and mruby cli, I thought it sounded pretty easy, but  I was sure there would be some challenges involved when using a different ruby implementation.  The first one is that mruby is not the same as cruby.  The implementation doesn't support doing everything that you could do in cruby like  I'm familiar with.  The first problem I had was the modular standard library.  mruby's core library has classes like string and array and hash, but they only support  the minimum number of methods.  For the full cruby experience, you have to add extension gems like mruby array ext.  And it's very easy to forget to do this if you're developing your own mruby gem and  you forget to add these extension gem dependencies.  And without them, the gem won't work in another project.  So it might work in one spot and not another.  mruby also has, mruby is also a smaller language outside the standard library.  There are no predefined globals such as load path since everything is compiled in, nor  any of the globals inherited from Perl or shell scripts.  So some libraries you might want to port from mruby may depend on these and you'll have  to work around them.  I also had a problem with here document support when porting a library from mruby and I haven't  yet reproduced it on its own, so I need to file a bug for it.  And while you can use keyword arguments to call methods, you can't define methods that  automatically check keyword arguments like you can in mruby.  If you haven't explored keyword arguments yet, there's a talk today in the second session  after lunch in ballroom C on keyword arguments to check that out.  Also some of the G sub replacements don't work in mruby, so you may need to change your  regular expressions.  Back traces in mruby also seem to be less useful than in mruby.  I feel like I did more exploring in the C source to determine why an exception was raised.  I'm not sure that's due to unfamiliarity with mruby or due to other reasons.  There are many pure ruby libraries that are already written in CRuby that you'll want  to use in mruby to make your life easier.  And overall this is a pretty straightforward process provided you beware there are restricted  syntax.  Unfortunately though, porting the tests for these is much harder due to differences in  the way tests run and we'll cover that in the future in this talk.  The existing gems for mruby feel a lot like working with libraries from the ruby one six  days.  And I think this is because there just haven't been enough people working in mruby to build  up the libraries that apply well to many different types of problems.  So an mruby library often has a very narrow focus, so you'll either have to do a bunch  of hunting to find the library that fits your use case or you'll need to submit patches  to get the library to work for you.  So far I've submitted some patches to all of these libraries.  For some libraries I've sent multiple patches.  They're usually pretty small which is nice.  And the last challenge I'll cover for mruby is getting the build information out of mruby.  Since mruby is compiled it needs to link in C libraries to compile successfully.  With mruby CLI you're also cross compiling these C libraries to many different platforms  so you need to use the correct C libraries.  Unfortunately it's hard to extract the cross compiler information out of the mruby build  system without loading all of its rake files which is pretty slow.  And I think improving this will make mruby's build system more flexible but I haven't yet  tried to or haven't done any work on this yet.  mruby CLI like mruby was a new tool for me so it took some time for me to learn to use  it the right way and here's some of the larger challenges that I had with it.  The first one was Docker.  The mruby CLI build system is built to top Docker and one of the really great things  about this is it's ready to cross compile to Linux, the OS 10 and to Windows without  any extra setup.  Unfortunately Docker is kind of clumsy the way mruby CLI uses it.  Typically Docker is used to run a long lived service but mruby CLI only needs to keep the  container running just enough to finish compiling or to run the tests.  But I think out of all the options Docker is the best one available because there really  is no extra work to get started with building command line tools.  In order to start a build of your command line tool you run Docker compose run compile  and this is far too long to type even if you use your shell history well.  When there's a problem with the compile it's hard to debug from the outside.  You have to start a shell inside of Docker then figure out which commands you were running  and then run those to reproduce the problem before you can start to fix it.  And when you're working with this Docker shell you don't have your familiar development environment  and there's no shell history from the last time you were doing this so it's harder than  usual to get things fixed.  And the biggest challenge I've had with mruby CLI was its build system.  Most of these problems came from the way the tasks were organized, the tasks for building  your executable and the tasks for building mruby were all mixed together into a single  rate command.  And this made it hard for her for you to add your own tasks and then have them run in the  right spot.  For example before compiling the mruby curl extension you need to have the curl CLI where  installed for every platform you're cross compiling to and by default that's very difficult.  And due to these difficulties I made some improvements to the build tasks.  The first was to build mruby separately from all the other work that must be performed  inside Docker.  This means I can cross compile the CLI before time.  The other improvement was to have a rake task that exists outside Docker and starts Docker  for me.  This allows me to perform extra tasks that don't need Docker to run and are easier for  me to debug.  So the new build system runs rake three times.  First the outer rake tasks run and then once they're done they start up the inner rake  tasks inside of Docker.  The inner rake tasks perform things like cross compilation or any other setup that can only  occur inside Docker.  And when these are done mruby's rake tasks are run which builds your command line tool.  Since the outer tasks don't need Docker I use them to do things like download and unpack  mruby to get it ready to compile.  There's also some global test setup tasks for reason I'll cover later.  And finally there are release tasks to publish your finished gem executable.  The inner tasks that run inside Docker cross compile the CLI you have configured and then  invoke rake again to compile mruby and run the tests.  There's some additional benefits to my changes to the build system.  It's now much easier to add hooks for custom tasks since there are well defined tasks  for different phases of the build.  I also made rake test faster by only compiling for the host platform when running tests.  There's no need to build every platform when you can only run tests on the host Docker  platform.  It's also pretty likely you'll need to use CLI as a dependency of an mruby gem in your  project and there's two options.  You can include them in your command line tool.  The first is static linking and this embeds the library inside your command line tool  and keeps you with a single file executable.  The advantage is you don't have to provide any separate instructions for installing the  correct CLI for your tool.  The downside though is that you need to re-release your command if there are any vulnerabilities  in those CLI.  This also makes your executable a little larger.  The second option is to dynamically link your CLI.  The advantage here is that vulnerability management is much easier.  The user should be upgrading their own libraries for security vulnerabilities so there's less  for you to do.  The disadvantage is this adds an external dependency so you're no longer truly a single  file.  The final challenge I had with mruby CLI was cross-compiling libraries.  mruby CLI can cross-compile your executable for up to six different platforms and this  means your CLI also need to be compiled for all of those six different platforms.  For example, for libcurl you can't just install the libcurl development package inside Docker  and be done since this library doesn't work on OS X or Windows.  Instead you have to download the source, unpack it, cross-compile for each platform you're  going to be releasing on, and then set up mruby's compiler to use those libraries that  you've built.  To help myself out, I wrote a tool to automate cross-compiling dependencies.  You can configure the library to cross-compile with a few values.  The first is the name, curl here.  The release name is used to create the source directory since libraries don't necessarily  use consistent naming styles.  The URL is where the source is downloaded from, and then optionally you can set extra  configure flags.  For example, I don't yet have open SSL cross-compiled so I disabled it for now.  In part two, I'll discuss the origins and the structure of the command line tool.  So my exploration of mruby CLI started with a tool that our customer support team could  use to configure a new product we were developing.  Our service was and still is API only so there's not yet any web UI for anyone to use.  The service has a complicated setup so I wanted to automate common use cases to make it easy  to avoid errors.  I also wanted to have a tool that customer support could use and also send to users to  reduce the burden of supporting this new product.  The solution to these problems is to write a tool that could automate the setup, validate  any inputs, and reduce the number of errors the users would encounter.  It would have the benefit of also giving design feedback on the API and we could use that  to make further improvements.  Before exploring mruby CLI, I started with some prototypes in CRuby.  The first was a single file script that primarily served as documentation.  Script was very easy for me to write and customer support could modify it for their needs but  it wasn't flexible enough to handle all the ways they needed to interact with the API.  As the needs of customer support grew, I split this single script into multiple scripts.  Each script had its own purpose, for example, listing options, performing initial setup,  or enabling or disabling features.  This was more flexible and I made the point of making the scripts well documented so it  would be easy to modify for writing future scripts.  And finally, I extracted a couple classes from the scripts for reuse.  This included a basic implementation of JSON API, set as our API transport specification,  and one for argument parsing using Ruby's option parser as we have many commands using  the same arguments.  So what lessons did I learn from the prototypes?  The first was that documentation in the tool was very important.  We had not yet budgeted time for API documentation of the service, so documentation via the  command line tool was very useful.  Roughly half the lines in the script ended up being comments.  This also meant our support staff could figure out what the tool was doing and suggest changes  that would improve their lives.  Keeping the script simple made everything more understandable.  None of the scripts had any external dependencies, so they were easy to install just by checking  out the repository they lived in.  Most scripts were small and had a single well-defined purpose.  And it was okay to have one or two larger scripts that automated an important action  like initial setup.  Ultimately, though, they were hard to maintain.  Our development environments install an old version of Ruby without keyword arguments  by default.  So we needed to install a modern Ruby version separately, and customer support also would  add or edit some of the existing tools to make their lives easier, but then they would  end up with multiple independent versions that were hard for the both of us to support.  I decided to investigate a switch to M-Ruby-CLI to address the maintenance burden while trying  to keep the maintainability high.  I decided to model my command line tool off of the pattern that RubyGems uses.  It has a set of subcommands with one subclass per subcommand, and they both use option  parser to parse arguments.  Each has a common setup method that picks which options are used for the command, and  then after parsing arguments, the execute method runs the command and handles any errors.  For example, if we got a command that lists conditions, the setup method would require  a service and a version that the conditions are attached to, and then the execute method  would perform the API operations to fetch the conditions from the service and then display  them.  I decided to port M-Ruby-OptParse from the C-Ruby standard library for option parsing.  I chose this because it was familiar to me and to other Rubyists, and it has good documentation.  OptParse performs automatic type checking and argument completion for you, which means  less code to write to check these by hand.  OptParse even offers shell completion as a standard feature, which is very nice, and  all of these would make parsing arguments easy and straightforward.  For API requests, I chose M-Ruby-Curl.  I evaluated several other HTTP libraries for M-Ruby, and this one seems to be the most  mature, mostly because it uses Curl.  It supports both HTTP, which we use in our development environment, and HTTPS for accessing  our production systems.  The library also supports persistent connections, which is especially important for HTTPS requests.  The only downside to is the lib-curl dependency, which must be cross-compiled and linked into  the tool.  For testing, there are two M-Ruby gems that seem to be the most popular.  The M-Ruby-M-Test gem provides a library for assertions, while the M-Ruby-Test gem provides  the tool that runs the test cases.  While using these gems is pretty easy, setting up the test cases can be pretty painful.  Testing gave me the most frustration when building the command line tool.  It's the most important piece of writing maintainable software, but due to the unfamiliar environment,  there were some adjustments I had to make in order to understand testing with M-Ruby-CLI.  The M-Ruby-M-Test gem is very much like the mini-test library that ships with C-Ruby.  You make a subclass of the test class, you write test methods the same as mini-test,  set up and tear down methods are used for common test setup in the same test case.  But you have to remember to start the tests with mtestunit.new.run at the bottom of your  test case, since M-Ruby doesn't have at exit without an extra gem.  Since M-Ruby is compiled, you can't just run the test directly.  Instead, you use the M-Ruby-Test gem, which provides the test runner.  When the tests run, a new M-Ruby VM is created for each test file.  This gives complete isolation in between tests.  The design of a test runner has some complications to it, though.  The first is that creating a global test setup or global test helper requires setting test  preload in your M-Ruby gem specification.  Without this, the helper won't be available to any of your tests, because M-Ruby just  thinks it's a separate test file that it's supposed to run.  I'm still trying to get this to work, because I only discovered it recently.  The second complication was Docker.  M-Ruby CLI uses Docker compose, but most of the documentation is written for the Docker  command.  So it took some time for me to figure out how to set up DNS records so that tests could  use other services on my local machine.  I would have preferred to mock all the interactions, but without an easy test helper and a mocking  library.  It was just easier for me to use the development services directly.  The final complication was that M-Ruby CLI was slow to start because it compiled all  platforms before starting the tests.  This made the feedback loop very long, even though you might be making a very small change.  To work around the global test setup, I chose to perform some of the tests set up before  starting any of the tests.  I used the improved build structure to do the setup outside of Docker before starting  any of the tests, because I have better and more capable libraries there.  Since I only learned about test preload recently, I'm passing the data needed for global setup  into the tests using environment variables that each test can look up when it needs them.  And to improve startup, test startup speed, I adjusted the rake tasks to only compile for  the host platform, which is what the tests use.  This meant I only had to compile one platform instead of seven.  M-Ruby CLI also supports bin tests, which are integration tests.  Since I prefer strong unit tests, I haven't explored these much yet.  For an integration test, you run your command, and then it captures the output, and then  you can assert what the output looks like.  And despite all these difficulties, though, how would you choose to use M-Ruby CLI?  And why not use Go or one of these other tools that has the same capability?  I'm primarily a Rubyist, and I work with a team of Rubyists.  And while we aren't opposed to learning new things, M-Ruby keeps us in our comfort zone.  For example, when I had problems with the test setup, I could use C-Ruby that I was  more comfortable with to get the job done.  And remember that I want to have a tool that I can deliver to fastly customers.  So the ability to work where I'm most comfortable means that it's easier to ship quality software.  By limiting the number of things I need to learn at once, my stress level is lower and  my ability to deliver is higher.  And now we approach the end.  What would I like to see that would make M-Ruby and M-Ruby CLI easier to use?  For M-Ruby CLI, I'm going to contribute back the improvements to the build system I've  made.  I've already had some discussion with the maintainers about the improvements I've made  and what benefits they bring, and they've seen positive on them so far.  So documentation for M-Ruby CLI could also use some improvement, as it isn't necessarily  obvious where to start with a new command line tool or what complications you might  run into or which libraries you might want to check out.  I'd also like to make upgrades to new versions of M-Ruby CLI easier, but I'm not yet sure  how that will work yet.  Accurate backtraces would be very helpful.  Right now you have to pay extra close attention to figure out which line M-Ruby meant the  error occurred on.  I also want to separate the loading of the build configuration from generating the rake  tasks.  This will make it easier to integrate M-Ruby's build system with M-Ruby CLI's build system.  And finally, I'd like to see keyword arguments and methods.  M-Ruby's CAPI has a fancy implementation of argument checking, so I'm unsure how this  will work with keyword arguments.  In C-Ruby, I'd like to see the M-Ruby get args ported over from M-Ruby.  This is a function of the CAPI that extracts method arguments when a Ruby method calls  a C function.  In M-Ruby, the method allows you to specify the types of the arguments, so you get automatic  type checks and automatic conversion.  The equivalent method in C-Ruby is RB scan args, but it's much less expressive since  it only checks a number of arguments, so you have to do all of your type conversion by  hand.  Also, finding the test preload was difficult, so I like it to make it automatic, so it's  easier to write tests.  Also M-Ruby test runs all the tests and doesn't have an option to run just one test, so or  subset of tests.  And without this, it's much harder to make large changes since you have to go and filter  your tests by hand to figure out which test values are important.  Thank you.  Are there any questions?  Yeah, so this was, so the question was, what is passing data out of Bandmeat here for the  testing?  And for that, since you can't pass data across Docker without it going through a file, I had  to go and do some setup, write a file, and then load that file inside of M-Ruby to get  the test setup data that I needed.  And so I did that through argument, or through environment variables.  So the question was, how robust is support for linking dynamic libraries?  Like, do you, can you specify paths and all that?  And yes, you can specify all those paths with, there's a, the support for all those through  the M-Ruby gem specification stuff.  So you can say, like for example, for all of the cross-compiling work I did, it'll build  it into per platform directories, and then I can say, go look at x86, Linux, this directory,  and that's where curl will be.  So that, yeah, that's all very easy.  Can the user pass something in if it's not in the expected place?  I think you have to edit the build config.  I don't know if you can override with CC, like the CC or LD flags or LD library path  or so forth.  Did I notice much memory usage difference between CRuby and M-Ruby?  I did not measure that because the CRuby prototype that only runs for a few seconds  and does very little, although I did do a test.  One of my M-Ruby curl patches was to add persistent connection support by reusing the  curl connection, and I believe I ran it for a couple days, just repeatedly checking a  local server, and I don't believe the memory usage grew over a couple megabytes.  So it's very, it's very nice.  How long did it take to iterate from the first solution to the last one?  So I'm familiar with building command line tools in CRuby, so a lot of the difficulty  was figuring out how to use M-Ruby and how does this piece work and how does that piece  work.  In terms of writing the implementation in M-Ruby, it was very straightforward after  I was like, okay, so I figured out how to use curl, how to get opt-pars going, and then  everything else after that was very straightforward.  I probably spent three or four days writing the implementation of the command line tool,  or at least write one command was maybe a couple days.  The question was what does an upgrade path look like for a customer, and how do you mean  exactly, like upgrading?  So how would you release a new version of the command line tool?  So since it's a single file, you can upload the binary and say download this.  You can also do things like maybe releasing on Linux, you can wrap it up in a package,  which would be great if you had, if you're using dynamic linking, because you would say  I need curl and open SSL and whatever else.  So it will depend like, do you want to, it'll also depend on things like how exactly you  want to release it.  The easiest one is just, hey, here download my new version.  I suppose you could integrate a thing like Sparkle that OS X uses, but I haven't explored  any of that yet.  Command line tool.  So what are some notable command line tools built with Mrooby CLI?  There was a talk at RubyKaigi about Hakonewa, which was a tool for using containers, setting  up containers, I believe.  I don't have any other examples off the top of my head.  Maybe Terrence might know.  So Terrence says there is a lot of stuff in Japan less in the US, and one of the more  notable ones is a system monitoring tool.  I believe there was a RubyKaigi talk on that one.  I don't, it might be that, might be, yeah.  What did I do to cross compile lib curl?  So the only thing that I had to do was extract which compiler, compiler configuration, Mrooby  CLI provided that by default.  So I extracted those compiler information and then plugged those into the call to configure  that, the configure script that came with the lib curl source.  I can show you after the talk if you'd like to see.  So how different is Mrooby from CRuby and how different is CRuby from Ruby?  So CRuby is synonymous with MRI.  So yeah.  So I originally gave this talk at RubyKaigi, so I use the, they're familiar with CRuby  there.  And Mrooby and CRuby are syntax-wise.  They're very similar.  There's a few things that are bugs probably.  There's a ISO specification for the Ruby language and it's, Mrooby is built off of that specification.  Like I said, there's a few things that are missing like automatic argument checking for  keyword arguments.  You can just do star star args and that's about it.  Global variables are different.  Some libraries may not support a thing you need.  One of my patches was to on a Geruma where it didn't give, it didn't expose optional  captures properly so it had empty strings instead of nil.  So there's a few things you're going to run into that are like, oh, somebody didn't fill  out all of the features yet.  But overall it's going to be very familiar.  So how did I handle the non-availability of require?  So part of it is you just list your Mrooby gems in your specification and those get  compiled in automatically.  There are some metaprogramming tricks you can do with require in CRuby that you can't do  in Mrooby like the Ruby, sorry, the Ruby gems command library will opportunistically load  just the sub-command that you need.  But since it's all compiled in and it's all loaded at once, you don't need to worry about  that but it requires you to adjust your thinking about some of the metaprogramming things you  do like this thing loaded and it automatically gets registered while you have to just register  it manually.  So as far as regular programming, it's fine if you want to do some of those metaprogramming  tricks that sit on top of require doing some stuff and having callbacks like this subclass  was created.  You're kind of like, well, I don't need to worry about that anymore because it's just  going to be there and I'll just have a thing that pre-registers it manually.  How many platforms did I build it for and why do all these cross-compilation?  So Fastly's customers operate on many different operating systems and Mrooby CLI provides  us out of the box.  I disabled Windows support just because I wanted to save time and I'd come back to it  later.  So right now it's building for four platforms, Linux and OS X 64 and 32-bit.  There's really no reason to turn it off unless I wanted to test it on OS X and the Docker  platform and I had a development environment that used Linux.  So it was just easy to be like, I'll just leave those on and then I can make sure that  I don't have any libcurl bug that didn't compile something correctly or whatever.  But yeah, it's kind of like Mrooby CLI.  That's all that stuff up for you so there's no reason to disable it unless you want to  save some time.  So the question was, based on all the things I want to add to Mrooby, how easy is it to  contribute to Mrooby compared to Crooby?  Because even Mrooby's internals are broken up into a bunch of little gems like Mrooby  array extension and so on.  So this makes it much easier to figure out where you want to make a change.  So if I wanted to fix a bug that was in some array method, then I would be able to go and  find the spot to make the change much easier, which means that it's less of a burden of  like, how do I make this change and where does it go with Crooby?  It's one big project with all the files mixed together so it's much harder to figure out  how to make the change.  So I think I need to make this change, where does it go is much easier to do.  So Terrence pointed out that Mrooby is all on GitHub with a pull request workflow so  there's not the complication of Crooby's hybrid Git subversion issue tracking.  The community is smaller but reasonably active and some other stuff that I forgot.  I believe there are no more hands so thank you for your time.
Building_maintainable_command-line_0_59.png Building_maintainable_command-line_60_2363.png Building_maintainable_command-line_2364_2435.png Building_maintainable_command-line_2436_3731.png Building_maintainable_command-line_3732_5771.png Building_maintainable_command-line_5772_5891.png Building_maintainable_command-line_5892_8387.png Building_maintainable_command-line_8388_8579.png Building_maintainable_command-line_8580_9695.png Building_maintainable_command-line_9696_18311.png Building_maintainable_command-line_18312_18851.png Building_maintainable_command-line_18852_24467.png Building_maintainable_command-line_24468_24611.png Building_maintainable_command-line_24612_25907.png Building_maintainable_command-line_25908_26015.png Building_maintainable_command-line_26016_27923.png Building_maintainable_command-line_27924_30995.png Building_maintainable_command-line_30996_35459.png Building_maintainable_command-line_35460_36479.png Building_maintainable_command-line_36480_36683.png Building_maintainable_command-line_36684_40835.png Building_maintainable_command-line_40836_41987.png Building_maintainable_command-line_41988_44351.png Building_maintainable_command-line_44352_44555.png Building_maintainable_command-line_44556_44891.png Building_maintainable_command-line_44892_45047.png Building_maintainable_command-line_45048_45347.png Building_maintainable_command-line_45348_46019.png Building_maintainable_command-line_46020_46115.png Building_maintainable_command-line_46116_46499.png Building_maintainable_command-line_46500_47159.png Building_maintainable_command-line_47160_47639.png Building_maintainable_command-line_47640_48095.png Building_maintainable_command-line_48096_49307.png Building_maintainable_command-line_49308_49427.png Building_maintainable_command-line_49428_49535.png Building_maintainable_command-line_49536_49739.png Building_maintainable_command-line_49740_50087.png Building_maintainable_command-line_50088_50411.png Building_maintainable_command-line_50412_51035.png Building_maintainable_command-line_51036_51983.png Building_maintainable_command-line_51984_52247.png Building_maintainable_command-line_52248_52619.png Building_maintainable_command-line_52620_52703.png Building_maintainable_command-line_52704_52979.png Building_maintainable_command-line_52980_53123.png Building_maintainable_command-line_53124_53207.png Building_maintainable_command-line_53208_53339.png Building_maintainable_command-line_53340_53771.png Building_maintainable_command-line_53772_53999.png Building_maintainable_command-line_54000_54335.png Building_maintainable_command-line_54336_54467.png Building_maintainable_command-line_54468_54527.png Building_maintainable_command-line_54528_54719.png Building_maintainable_command-line_54720_54959.png Building_maintainable_command-line_54960_54981.png