metajungle A metajungle by Jakob Henriksson Tue, 14 Feb 2017 14:20:45 +0000 Tue, 14 Feb 2017 14:20:45 +0000 Jekyll v2.5.3 Poor man's service monitoring using HAProxy <p>There are many tools and services available to monitor your web services. <a href="" title="Pingdom">Pingdom</a>, for example, is a popular service, but you can also write your own custom shell scripts to ensure your services are up and healthy. </p> <p>A compromise between a service you can pay for and a completely custom solution is to rely on existing tools to do the job for you. For example, why not let a load balancer with support for health checks monitor your web services? <a href="" title="HAProxy">HAProxy</a> is a great load balancer with a built-in UI for displaying the status of the services you are (supposedly) load balancing between. Of course in our case we are not load balancing between anything, because we are not sending requests to the service instances. We just want to know if the services are up or not. Doesn’t this look great? </p> <p><img src="/static/images/posts/poor-mans-service-monitoring/monitoring-using-haproxy.png" title="Monitoring using HAProxy" /></p> <p>At least this gives you something to look at with clear color-coded indications if something is wrong with your services. </p> <p>However, you may want to be notified if something goes down, so how do you accomplish that? Luckily, HAProxy can export this list as CSV using an HTTP endpoint (<code>/;csv</code>). So if you wanted to you could just <code>curl</code> that endpoint, <code>grep</code> the result for “DOWN” and send a push notification to your phone for each match using something like <a href="/2015/10/push-notifications-for-developers/">Pushover</a>. </p> <p>Here is an example HAProxy configuration for this setup (file: <code>/etc/haproxy/haproxy.cfg</code>): </p> <pre><code>global log local0 notice maxconn 100 user haproxy group haproxy defaults log global mode http option httplog option dontlognull retries 3 option redispatch frontend frontend bind *:80 mode http backend tomcat-servers mode http balance roundrobin option httpclose option forwardfor option httpchk GET /health_check/ # nodes to monitor server check fall 3 rise 2 maxconn 10 # more services here... backend docker-hosts mode http balance roundrobin option httpclose option forwardfor option httpchk GET /health_check/ # nodes to monitor server check fall 3 rise 2 maxconn 10 # more services here... listen stats *:1936 stats enable stats uri / stats hide-version </code></pre> <p>Of course, you want to install and configure HAProxy using <a href="" title="Ansible">Ansible</a>. And if you get creative you can be smart about which hosts from your Ansible inventory to include to be monitored, and how to categorize the services in HAProxy to better understand what is down when something goes wrong. </p> <p>The key to this particular configuration is the <code>httpchk</code> option which uses a custom endpoint (<code>/health_check/</code>) that needs to be implemented by the services being monitored. As long as the services returns a valid HTTP 200 response for this endpoint, everything will look good. The implementation of this endpoint can be as simple as to immediately return a response, or exercise the service is some way, for example by querying any configured databases. </p> <p>What’s your poor man’s web service monitoring setup? </p> Mon, 13 Feb 2017 22:00:00 +0000 post Ansible playbooks and mastering task execution <p>In my attempt to promote the usage of <a href="">Ansible</a>, I have often tried to succinctly explain some of its core concepts. In particular, I’ve tried to communicate a few simple features that I personally have gotten a lot of leverage out of: </p> <ol> <li>Roles</li> <li>Tags </li> <li>Host limits </li> </ol> <p>Master these simple concepts and you can do powerful things with Ansible. <!--more--></p> <p>Roles can create some confusion when learning Ansible. I’ve always talked about them as an abstraction for task reuse. But that’s not very clear. It’s easier to think of them as reusable tasks executing against hosts in a <em>context</em>. That context can provide files, templates, variables, handlers, that the role’s tasks can refer to and make use of during a play. But remember that, in the end, applying a role to a host is still all about the tasks. </p> <p>That brings us to two main ways of slicing and dicing the execution of tasks against hosts during a play. This is in particularly useful when you have large inventories, and master playbooks for entire deployments (playbooks that in turn include playbooks for specific services/applications).<sup id="fnref:ref-best-practice"><a href="#fn:ref-best-practice" class="footnote">1</a></sup> Let’s say you have a master playbook <code>site.yml</code> and can do: </p> <pre><code>$ ansible-playbook -i prod site.yml </code></pre> <p>That’s great because you can quickly deploy your entire system with a single command. It’s also easy to remember. But you rarely need to deploy your entire system. </p> <p>By mastering the usage of <em>tags</em> and <em>host limits</em>, you can precisely target the execution of specific tasks across your entire deployment. I use that all the time. Visually, this is how I think about it: </p> <p><img src="/static/images/posts/ansible/ansible-tasks.png" title="Ansible plays" /></p> <p>All hosts in the play (across the bottom in the graphics above) can have tasks executed against them. By default, all applicable tasks gets executed against all hosts in the play. However, within a given play, exactly which tasks gets executed against which hosts can be controlled using tags and host limits. </p> <p>If all tasks in your roles have tags with a sensible naming convention, it’s easy to only run tasks in a specific role.<sup id="fnref:ref-tags-naming"><a href="#fn:ref-tags-naming" class="footnote">2</a></sup> </p> <pre><code>$ ansible-playbook -i prod site.yml -t "mysql-server" </code></pre> <p>This play might only be applicable to specific servers anyway. You can easily limit the hosts to be considered with limits, like so: </p> <pre><code>$ ansible-playbook -i prod site.yml -t "mysql-server" -l "mysql-servers" </code></pre> <p>Alternatively, consider this example, re-deploying all Tomcat applications: </p> <pre><code>$ ansible-playbook -i prod site.yml -t "tomcat-war" -l "tomcat-servers" </code></pre> <p>Now you have limited the play to a specific subset of hosts, and a subset of tasks. This gives you a lot of freedom and control. Below are some more examples that I use all the time. </p> <p>Pull all Docker images across the deployment infrastructure: </p> <pre><code>$ ansible-playbook -i prod site.yml -t "docker-images-pull" -l "docker-hosts" </code></pre> <p>Deploy all Docker containers: </p> <pre><code>$ ansible-playbook -i prod site.yml -t "docker-containers" -l "docker-hosts" </code></pre> <p>Re-deploy a specific application on Tomcat: </p> <pre><code>$ ansible-playbook -i prod site.yml -t "tomcat-war" -l "front-end-applications" </code></pre> <p>Re-configure your front-end application load balancer: </p> <pre><code>$ ansible-playbook -i prod site.yml -t "haproxy" -l "front-end-applications" </code></pre> <p>Spend a few moments learning these concepts. I can guarantee you will find them useful. </p> <div class="footnotes"> <ol> <li id="fn:ref-best-practice"> <p>As recommended in Ansible’s <a href="">best practices documentation</a>. <a href="#fnref:ref-best-practice" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:ref-tags-naming"> <p>I like to tag all tasks in a role with the name of the role, and then provide additional tags with the role name as a prefix. For example, I might give the following tags to a task in the role <code>mysql-server</code> that updates the configuration: <code>mysql-server</code>, <code>mysql-server-config</code>. <a href="#fnref:ref-tags-naming" class="reversefootnote">&#8617;</a></p> </li> </ol> </div> Fri, 07 Oct 2016 20:00:00 +0000 post Programming and Consistency <p>In software development there is a tension between obsessing about a single line of code, and shipping. Focus too much on the perfect code and you will never ship a product, but ship without caring about the underlying code and its structure introduces the risk of poor product quality (bugs, inefficiencies) and building up long-term <a href="" title="Technical Debt">technical debt</a>. This tension is not specific to software development; the tension equally well applies to other kinds of creative work like writing a paper or a book. </p> <p>In the end, shipping becomes most important, because otherwise the value of all your efforts are to some extent squandered.<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup> </p> <p><em>Perfect is the enemy of good.</em> </p> <hr /> <p>Not every line or block of code has to be perfect, but there is great value in consistency. I believe <em>consistency</em> is a highly valuable trait in a programmer. What do I mean by consistency? Consistency can mean many different things, ranging from coding style to the choice of libraries and architectural designs. </p> <p>Here are a few examples of poor stylistic (syntactical) decisions: </p> <ul> <li>Sometimes indent you code with tabs and sometimes with spaces, or if using spaces, sometimes using 2 spaces and sometimes 4 </li> <li>Sometimes write <code>if(var==2){</code> and sometimes <code>if (var == 2) {</code>, or any combination in between </li> <li>If semi-colons are optional at end of statements, sometimes include them and sometimes not </li> <li>If braces are optional for single-statement blocks, sometimes include use them and sometimes not<sup id="fnref:java-blocks"><a href="#fn:java-blocks" class="footnote">2</a></sup></li> <li>Sometimes horizontally align variables and sometimes not<sup id="fnref:variable-alignment"><a href="#fn:variable-alignment" class="footnote">3</a></sup></li> </ul> <p>You get the point. But why does it matter you say? It’s not going to effect the running code. Am I just being pedantic? Perhaps, but I strongly believe in syntactical and stylistic consistency. It’s important for readability, which is important for maintainability, which in turn is important for evolving and improving the code in the long run. Readability is more than just avoiding writing “too smart” code, it’s also about style. </p> <p>Examples of consistency in library choices (unless there are specific needs) include avoiding using two different libraries for handling <a href="" title="JavaScript Object Notation">JSON</a> in the same program, or using two different libraries for making HTTP requests. Sometimes a language, or framework, comes with built-in capabilities to make HTTP requests. If the built-in library is used in some situations but an external library is used in others, it requires programmers to be familiar with the quirks of both libraries. Be pragmatic, but think twice before adding a <em>second</em> way to make HTTP requests, or for anything else. </p> <p>Examples of architectural decisions can for example be as simple as having consistent API designs with common formats and structures for handling errors and providing error messages. </p> <hr /> <p>To me consistency in programming and software development is just common sense, but I’m surprised how many programmers don’t seem to care about it. </p> <p>To be consistent, you have to be <em>detail-oriented</em>. It’s impossible to be consistent if you pay no attention to details. </p> <p>The kind of detail-orientation I’ve given examples of above are not necessarily <em>requirements</em> in deciding whether a person can, for example, design scalable and effective software architectures. However, I do believe attention to detail is somewhat of a canary in the coal mine to determine if a developer will be able to effectively develop, organize and maintain large-scale code-bases and architectures. </p> <p>The need for attention to details is not only for the code itself, but also for the documentation, the issue tracking, the tests, and the deployments and orchestration scripts. </p> <p>Douglas Rushkoff writes in <a href="" title="It's Not Just Grammar; It's Clear Thinking">The New York Times about grammar and clear thinking</a>: </p> <blockquote> <p>[A]n employee who can write properly is far more valuable and promotable than one whose ambiguous text is likely to create confusion, legal liability and embarrassment. Moreover, a thinking citizen deserves the basic skills required to make sense through language, and to parse the sense and nonsense of others.</p> </blockquote> <p>Writing has to be clear, otherwise it causes confusion. This also applies to software development and architectural design, choice of tools and libraries, and, yes, the style of a single line of code. </p> <div class="footnotes"> <ol> <li id="fn:1"> <p>However, there <em>is</em> also inherent value in the work itself because you always learn something along the way. <a href="#fnref:1" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:java-blocks"> <p>I’m looking at you Java. <a href="#fnref:java-blocks" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:variable-alignment"> <p>Actually, never align your variables horizontally <a href="#fnref:variable-alignment" class="reversefootnote">&#8617;</a></p> </li> </ol> </div> Mon, 25 Jan 2016 11:20:00 +0000 post Ansible and Amazon EC2 <p><a href="" title="Ansible">Ansible</a> is a great software orchestration tool that I’ve <a href="" title="Software Technology (2015)">enjoyed using for a while</a>. A core concept in Ansible is the <em>inventory</em>: a description of all the hosts available for deployments. The inventory is not only a list of hosts, but provides ways to group hosts into logical <em>named</em> units. The names of the groups can describe what role the hosts play in your architecture, or their geographical location or something else. A simple inventory can look like this: </p> <pre><code># file: prod [mysql-servers] [web-applications] </code></pre> <p>This inventory has one host in the <code>mysql-servers</code> group, and two hosts in the <code>web-applications</code> group. This communicates that there is one host for running a <a href="" title="MySQL">MySQL server</a> and two hosts for running a front-end web application. </p> <p>Now we can write a simple playbook to configure the MySQL hosts: </p> <pre><code>--- # file: mysql-servers.yml - hosts: mysql-servers roles: - mysql-server </code></pre> <p>Notice how we reference the logical inventory group <code>mysql-servers</code> in the playbook. We can then configure the MySQL hosts by running the Ansible playbook: </p> <pre><code>$ ansible-playbook -i prod mysql-servers.yml </code></pre> <p>This is fairly straightforward and the <em>naming</em> makes everything easy to understand. Importantly, the playbook is generic and can be reused for a different deployment environment were we also need MySQL servers. This as long as the new inventory file contains a similar organization and grouping as the first one, using the same names. </p> <h2 id="amazon-ec2">Amazon EC2</h2> <p><a href="" title="Amazon EC2">Amazon Elastic Compute Cloud (EC2)</a> is a popular platform for deploying applications and services. When managing a larger infrastructure, in particular a dynamic one, it can get difficult to manually manage the inventory file and keep track of all the different hosts. Ansible supports <a href="" title="Dynamic Inventories">dynamic inventories</a> that help with exactly this problem. In short, instead of referencing a simple text inventory file when running playbooks, you can give Ansible a script as the inventory. The task of this script is to figure out the current state of the infrastructure and return the inventory in a particular format that Ansible understands (a <a href="" title="JSON">JSON</a> object). </p> <p>Ansible provides some example scripts for common platforms, including Amazon’s EC2. This script (<code></code>) returns all your EC2 hosts using different kinds of grouping concepts, including EC2 regions, instance types, and EC2 instance tags (key-value pairs). If I create an EC2 instance with the tags <code>Name=mysql1</code> and <code>Group=mysql-servers</code>, the Ansible EC2 inventory script would return something like this: </p> <pre><code>$ --list { ... "us-east-1": [ "" ], "tag_Name_mysql1": [ "" ], "type_m3_medium": [ "" ], "ec2": [ "" ], "tag_Group_mysql_servers": [ "" ] } </code></pre> <p>Now I can reference all my instances in a particular region, or by instance type, or by my tags. But to configure my MySQL servers on EC2 using my very simple and supposedly reusable playbook from above, I need to change it to this: </p> <pre><code>--- # file: mysql-servers.yml - hosts: tag_Group_mysql_servers roles: - role: mysql-server </code></pre> <p>However, I don’t want to have a separate set of playbooks for different platforms. There are several options when it comes to resolving this issue. One solution is to define a text-based inventory that does the mapping for us, something like this: </p> <pre><code># file: prod-mapping [mysql-servers:children] tag_Group_mysql_servers </code></pre> <p>Now you can reference <code>mysql-servers</code> in your playbooks and it will include all the hosts in the <code>tag_Group_mysql_servers</code> group. However, this has to be done for all groups, so you have to do some work in managing and maintaining the mapping. </p> <p>The solution I have taken so far is to have a very simple script in front of the bundled <code></code> inventory script that allows us to augment the grouping of hosts. I’ve called this <a href="" title="Augments response from Ansible EC2 dynamic inventory with additional non-tagged host groups">script</a> <code></code> and it can be configured to create additional host groups based on the instance tags. For example, for the group <code>tag_Group_mysql_servers</code> to add the same group of hosts under the name <code>mysql-servers</code>. So, if my script is used as the inventory script, it would return the following: </p> <pre><code>$ --list { ... "us-east-1": [ "" ], "tag_Name_mysql1": [ "" ], "type_m3_medium": [ "" ], "ec2": [ "" ], "mysql-servers": [ "" ], "tag_Group_mysql_servers": [ "" ] } </code></pre> <p>This will work as long as I tag all my EC2 instances with the key <code>Group</code> and the value of the group name I want. For example, <code>Group=mysql-servers</code>, or <code>Group=web-applications</code>. This is easy to do consistently if you create all EC2 instances using the Ansible <a href="" title="Ansible ec2 module">ec2 module</a>.</p> <p>I can now run my existing playbooks against EC2: </p> <pre><code>$ ansible-playbook -i mysql-servers.yml </code></pre> <p>I’ve published the current version of the script as a <a href="" title="Augments response from Ansible EC2 dynamic inventory with additional non-tagged host groups">Gist</a>. </p> Sun, 17 Jan 2016 08:00:00 +0000 post Software Technology (2015) <p>The past 12 months or so have been intersting for me from a software technology perspective. I’ve been trying to balance learning and introducing new technologies into the software stack for production systems I work on, while at the same time staying <em>conservative</em> enough to not introduce too many complexities and minimizing day-to-day operational issues. </p> <p>As documented in <a href="" title="Scaling Pinterest">Scaling Pinterest</a>, via <a href="" title="High Scalability: Scaling Pinterest –"></a>, choices should often fall on well known, liked, battle-tested, and performant tools. There is a good case to be made to adopt technologies that have had at least 3-5 years to mature and self-select in the marketplace, especially if they are to be used in production systems with requirements on stability and uptime. </p> <p>But it’s not only technology choices that affect a system’s <em>runtime</em> that matter. Choices about technology and tools for infrastructure management, as well as development environments and tools, also play an important role. Below is a highlight of some of the technologies I’ve used in the past few months that I believe I will be using on a regular basis over the next few years for a range of products and solutions. </p> <h2 id="ansible">Ansible</h2> <p><a href="" title="Ansible">Ansible</a> is a powerful software orchestration and management platform. Ansible can be used to issue commands on machines (or <em>hosts</em>) over SSH. The hosts can be local or remote, physical or virtual machines, and can be accessed via username and password, or more commonly using a <a href="" title="Public key infrastructure">PKI</a>. Commands are issued via <em>tasks</em>, and can be for the purpose of <em>configuring</em> hosts for specific purposes, or for directly running services or applications. Ansible provides a large library of <a href="" title="Ansible Modules">modules</a> for performing common tasks on hosts. The true power of Ansible comes from needing to perform repeatable tasks on a large set of hosts. </p> <p>There are a relatively few number of core concepts that need to be understood in order to be proficient using Ansible.<sup id="fnref:ansible-concepts"><a href="#fn:ansible-concepts" class="footnote">1</a></sup> Once this is accomplished, a great deal of power is suddenly at your fingertips. </p> <p>I use Ansible to setup all new clouds VMs that I create with a consistent configuration. Using the powerful Ansible <em>roles</em> concept, I can easily configure machines for particular purposes, for example, <a href="" title="MySQL">MySQL</a> servers, <a href="" title="Redis">Redis</a> instances, <a href="" title="Apache Tomcat">Tomcat</a> containers, and <em>lots</em> of other things. I’ve used Ansible for complex deployments of service-oriented applications for production. I’ve come to rely on Ansible on a daily basis, and it’s actually fun to use. </p> <p>One of the simplest task is to issue single “ad-hoc” commands against a host or a set of hosts. This can be extremely useful, and often powerful. I often use it to restart a service (Tomcat) or check some state (running processes). Just to show an example, this command stops all Tomcat servers in the specified inventory of hosts: </p> <pre><code>$ ansible -i dev tomcat-servers -s -m "service" -a "name=tomcat7 state=stopped" </code></pre> <p>I once had a rogue Java process running on one of 20+ VMs that I had launched. At least, that’s what I suspected. That’s a lot of VMs to log into to investigate. So, instead I simply issued the following Ansible command: </p> <pre><code>$ ansible -i dev all -m "shell" -a "ps aux | grep java" </code></pre> <p>The output quickly told me where the process was running so I could stop it. </p> <p>A <a href="" title="Ansible Best Practices">best practice</a> suggestion is to define your entire infrastructure in a master <code>site.yml</code> Ansible <em>playbook</em>. This is not only useful for deploying everything in a new environment, but this general-purpose playbook can also be used to perform very specific actions using the powerful concepts of <em>tags</em> and <em>host limits</em>.<sup id="fnref:ansible-master-playbook"><a href="#fn:ansible-master-playbook" class="footnote">2</a></sup> A command like this could configure and deploy all Tomcat applications: </p> <pre><code>$ ansible-playbook -i prod site.yml --limit tomcat-servers </code></pre> <p>Combining host limits with tags can narrow down the tasks executed even further, for example to only re-deploy the Tomcat applications (without necessarily configuring everything). That could look something like this: </p> <pre><code>$ ansible-playbook -i prod site.yml --limit tomcat-servers -t "tomcat-war" </code></pre> <p>These simple examples show how you can organize your Ansible playbooks to be able to configure an entirely new environment, and deploy your entire infrastructure, but also perform <em>cross-cuts</em> to execute very specific tasks, against a limited set of hosts. </p> <p>Software will always need to be deployed on some hosts that need to be configured, and other infrastructure (for example, databases) needs to be put in place. Ansible is a great and elegant tool for all these tasks, so I expect to be using Ansible for a long time.<sup id="fnref:ansible-alternatives"><a href="#fn:ansible-alternatives" class="footnote">3</a></sup> </p> <h2 id="docker">Docker</h2> <p><a href="" title="Docker">Docker</a> is a virtualization project and set of tools that leverage capabilities of the Linux kernel to run processes in independent containers. Containers can roughly be seen as a lightweight virtual machines, providing process isolation (from the host machine) and resource controls (memory, CPU, I/O, network). </p> <p>Process isolation and sandboxing is certainly a core feature of containers, but from a practical day-to-day perspective, far more important (to me) are two tangible benefits of containerizing your applications:</p> <ol> <li>Dependency management </li> <li>Application interface </li> </ol> <p>By <em>dependency management</em> I mean in the same was you would use <a href="" title="Virtual Environments">Python virtual environments</a> to manage and separate dependencies for different applications. Often a developer will have all application dependencies correctly configured in their environment, but will forget to document or update the setup scripts to install the dependencies when installed in a fresh environment. This has the obvious downsides of breaking deployments and delaying integration. If applications are shared, tested and delivered as containers, these issues can often be avoided. </p> <p>Providing a consistent <em>application interface</em> to start and stop applications can greatly simplify deployments, and application lifecycle mangement (e.g. restarts). I’ve been responsible for complex deployments of service-oriented applications where different components (services) are written in completely different languages and frameworks. As such, each component has a different process for managing its lifecycle: <a href="" title="Grails">Grails</a> applications running on Tomcat, Python web services using WSGI, Java command-line interfaces, Python scripts, shell scripts etc. No single application interface is particularly difficult by itself, but taken all together it gets somewhat unwieldy. How do you start and stop 8 diffrent Java CLI programs running on different VMs? How do you shutdown only a subset of the processes on a subset of VMs? How do you easily restart two Python web services and a Java process that are coordinating their activities? </p> <p>With containers and Docker, the launching of an application is described in the relevant <a href="" title="Dockerfile reference">Dockerfile</a>. Once the Docker images are created (and published on <a href="" title="Docker Hub">Docker Hub</a>), the lifecycle management of the applications is identical regardless of the underlying application or service implementation details (including choice of programming languages or frameworks). </p> <p>Now everything is a container that can be started, restarted and stopped. The simplicity of that common application interface is a huge benefit. And how do you manage the containeres across your deployment infrastructure? With <a href="#ansible">Ansible</a> of course. </p> <h2 id="redis">Redis</h2> <p><a href="" title="Redis">Redis</a> is a multi-purpose in-memory <em>data structure store</em>. Data can be persisted to disk and Redis can be used as a database, but I’ve never used it like that. I’ve used it as a key-value cache, and as a message broker. Redis is great as a simple cache, but one of the benefits of Redis is that it is general-purpose, allowing the same technology to be used in different scenarios. If Redis can solve multiple needs for a project, then I can reduce the number of technologies that I incorporate into the overall software stack. </p> <p>As I mentioned, I’ve used Redis as part of a message system, more specifically for task processing using queues. A queue is simply an ordered list with first-in-first-out (FIFO) characteristics. Push tasks into one side of the queue, and take tasks out from the other side. <a href="" title="JSON">JSON</a> makes it easy to serialize and store complex structured task descriptions in the queue. </p> <p>There are also other uses for Redis that would be interesting to explore in the future, including sorted sets for statistics applications. For example, consider the following commands to track the occurrence of terms (in some data source): </p> <pre><code>; ZINCRBY myzset 1 "my-word" "1"; ZRANGE myzset 0 -1 WITHSCORES 1) "my-word" 2) "1"; ZINCRBY myzset 2 "my-word" "3"; ZRANGE myzset 0 -1 WITHSCORES 1) "my-word" 2) "3" </code></pre> <p>These statistics can be stored ephemerally in Redis, and then queried and later stored in an <a href="" title="Online Analytical Processing">OLAP</a> system. I’m sure there are other interesting situations where Redis would be a good fit. </p> <h2 id="cloud">Cloud</h2> <p>I’ve created VMs for various needs for a while, but this year I really started to make heavy use of cloud infrastructures, mainly centered around <a href="" title="OpenStack">OpenStack</a>. I don’t know the in and outs of OpenStack, but I’ve become familiar with the <a href="" title="OpenStack Horizon">OpenStack Horizon Dashboard interface</a>, which is serviceable. I’m a <em>user</em> of cloud infrastructure, but it’s still useful to know some of the basics of how OpenStack works, and in particular how to use the compute capabilities provided by its <a href="" title="OpenStack Nova">Nova</a> component. For instance, there is a <a href="" title="OpenStack Command-Line Interface Reference">command-line interface to Nova</a> for managing compute instances (VMs) that I’ve found useful when setting up infrastructures. </p> <p>As a cloud <em>user</em>, the important part is how to manage and organize a large set of resources, and for that <a href="#ansible">Ansible</a> has been the key for me. Next I want to explore the specifics of <a href="" title="Amazon EC2">Amazon’s EC2</a>, but again, the details are not so important because I will be using Ansible there too. </p> <h2 id="python">Python</h2> <p>I’ve played with <a href="" title="Python">Python</a> for many years, and I really like it. To me, it’s an elegant language and it’s easy to get things done using it. There is no shortage of 3rd party modules (libraries) and frameworks, and documentation is usually pretty good. The sooner you move to Python 3 the better probably (because <a href="" title="PEP 0404 -- Python 2.8 Un-release Schedule">there will be no Python 2.8</a>). </p> <p>Of course make sure you use <a href="" title="Virtual Environments">virtual environments</a>, to keep your dependencies organized. In fact, that’s how I install the <a href="#ansible">Ansible CLI</a>, and the <a href="#cloud">OpenStack CLI</a> set of tools like the Nova client. </p> <p>What I’ve used Python for a lot lately is writing simple system management and monitoring scripts. For example to ensure that all deployed web services are running and responding to simple HTTP requests. Another example is to ensure that task queues in <a href="#redis">Redis</a> are not growing faster than they can be processed for an extended period of time. Yet another example is a script to organize component-specific documentation files before building a system-wide documentation website. Python makes it easy to write well organized and maintainable code.<sup id="fnref:python-programmer"><a href="#fn:python-programmer" class="footnote">4</a></sup> Furthermore, the script dependencies are documented in <a href="" title="Requirements files">requirements files</a> so that they can easily be installed in a new virtual environment, or in a <a href="#docker">Docker image</a>, which is of course how the Python scripts are deployed. </p> <hr /> <p>This brings things around: <em>I’ve really enjoyed writing <a href="#docker">containerized</a> Python 3 scripts for monitoring processing queue stored in <a href="#redis">Redis</a> (and other tasks) that have been deployed on an <a href="#cloud">OpenStack cloud</a> using <a href="#ansible">Ansible</a></em>. </p> <h2 id="extra">(Extra)</h2> <p><a href="" title="Git">Git</a> is of course great for source code management and versioning. I wanted to mention <a href="" title="Jekyll">Jekyll</a> again and what a great choice it is for information-centric websites. Since writing <a href="">Deploying Jekyll with Git</a>, I’ve used Jekyll for organizing the documentation for a large-scale application, which has worked out really well. Webpages in Jekyll are written in <a href="" title="Markdown">Markdown</a>, which is how I also write this blog. Writing in Markdown keeps your content in a simple non-proprietary format, while making it easy to publish to the Web. <a href="" title="OS X">Mac OS X</a> is an excellent platform for developers. It’s a modern operating system with great application support, while also providing access to all the greatest Linux-based tools from the command-line, often installable via <a href="" title="Homebrew">Homebrew</a>. I write a lot of code in <a href="" title="TextMate 2">TextMate 2</a>, use <a href="" title="LaunchBar 6">LaunchBar 6</a> for clipboard management, and <a href="" title="TextExpander">TextExpander 5</a> for improved typing efficiency. </p> <div class="footnotes"> <ol> <li id="fn:ansible-concepts"> <p>Some of these tasks include <a href="" title="Ansible Inventory">inventories and hosts</a>, <a href="" title="Ansible Playbooks">playbooks and tasks</a>, <a href="" title="Ansible Variables">variables</a>, <a href="" title="Ansible Roles">roles</a> and <a href="" title="Ansible Modules">modules</a>. <a href="#fnref:ansible-concepts" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:ansible-master-playbook"> <p>Of course the <code>site.yml</code> playbook is only a set of includes of more specific playbooks, which can also be run separately. <a href="#fnref:ansible-master-playbook" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:ansible-alternatives"> <p>There are of course other orchestration platforms besides Ansible, but I have not found them as approchable or elegant. <a href="#fnref:ansible-alternatives" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:python-programmer"> <p>Although of course it requires some diligence on the part of the programmer. <a href="#fnref:python-programmer" class="reversefootnote">&#8617;</a></p> </li> </ol> </div> Mon, 28 Dec 2015 08:00:00 +0000 post Push notifications for developers <p>As a software developer or engineer, there can often be times when you want to be notified that some event took place on the systems you are working with. For example, you want to start a long running database import process and want to be told when it has been completed, or you want to know when an important service is no longer responding to requests on your production system. </p> <p>Sending such notifications can of course be done via email, but it’s somewhat clunky and error prone and typically requires some setup and configuration. I’ve been using a more modern solution that I’ve been very happy with so far: push notifications sent directly to my phone. </p> <p>I’ve been using <a href="" title="Pushover">Pushover</a> and it has worked great. </p> <p>Pushover has a native app for iOS (and Android), so all notifications end up on my phone. Pushover provides an API that can be called using standard HTTP requests to actually send the notifications. So, as long as you have <code>curl</code> (or something similar) on the system where notifications should be sent from, there is no other configuration needed. </p> <p>You do need to create a Pushover account and generate a token to pass to the API. Once that is done, you can send a notification like so: </p> <pre><code>$ curl -s --form-string "token=[your token]" --form-string "user=[your user id]" --form-string "message=Message" </code></pre> <p>Pushed notifications end up on your phone (or other device), for example, like this: </p> <table> <tbody> <tr> <td> <img src="/static/images/posts/push-notifications-for-developers/iphone-lock-screen.jpg" title="Pushover notification on iPhone lock screen" /> </td> <td> <img src="/static/images/posts/push-notifications-for-developers/iphone-pushover.jpg" title="Pushover application on iPhone" /> </td> </tr> </tbody> </table> <p>You can also set different importance levels to the notifications, if that becomes necessary. I’m using this both for routine development tasks and for monitoring production systems. </p> <p>Entering the <code>curl</code> command and parameters is a little bit tedious, especially if you just want to send a one-off notification. I’m using <a href="" title="TextExpander">TextExpander</a> on my Mac, so I’ve mapped “;push” to generate the <code>curl</code> command for me. You might want to consider something similar. </p> <p>This means I can, for example, type: </p> <pre><code>$ mysql database &lt; data.sql; ;push </code></pre> <p>and have that automatically expanded to: </p> <pre><code>$ mysql database &lt; data.sql; curl -s --form-string "token=[my token]" --form-string "user=[my user id]" --form-string "message=Message" </code></pre> <p>with my Pushover token and user ID already included. I can then just change the notification message, hit enter, and wait for the notification to come in as I do other work. </p> Sat, 24 Oct 2015 08:55:00 +0000 post → Adapting to new technologies <p>Jeff Sussna writes in <a href="" title="Crossing the Ocean With Containers">“Crossing the Ocean With Containers”</a>: </p> <blockquote> <p>The constraint that gates a company’s ability to absorb change often isn’t a new technology itself, but rather the ability to hire people who can comprehend the implications of that technology, and who can operate it based on that understanding.</p> </blockquote> <p>The computer and technology industry has always moved quickly, and practitioners have to keep up to stay relevant in a shifting field. For software developers and engineers, there is always the latest trends in data storage and management, frameworks, languages, testing processes, build systems, source code management, application deployment methods, software orchestration, and so on. </p> <p>This multitude of options and shifting technological landscape poses certain challenges to individual developers in choosing the best tools and processes for their work. This is important to help solve immediate problems and challenges, but also to further their general expertise, career wise, and to satisfy curiosities. </p> <p>But as Jeff Sussna clarifies, it poses an even larger challenge for companies that need to attract the right people that take advantage of the right tools and understand the technological choices that are being made, for the long term. </p> <p><a href="" title="Permanent link to Adapting to new technologies">∞ Permalink</a></p> Wed, 21 Oct 2015 16:20:00 +0000 link Deploying Jekyll with Git <p>When developing software, and especially Web applications, I’ve found that thinking about the best deployment method early on is usually a good thing. Solving the deployment situation answers the question: <em>What’s the best way to make the latest version of my software available to users?</em> </p> <p>The answer very much depends on the software being developed, and the tools used to do so. I’ve recently played with a tool called Jekyll, so I’ll talk about that. </p> <h2 id="jekyll">Jekyll</h2> <p><a href="" title="Jekyll">Jekyll</a> is a simple but powerful tool for creating Web sites. It’s a little different from Web application frameworks like <a href="" title="Django">Django</a>, or other similar frameworks that are available.<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup> A traditional Web application is typically backed by a database and supports dynamic content generation. When a request comes in from a Web browser the application first needs to determine what the request is for. Then it calls the code in the application that is responsible for handling the request. The application might fetch some data from a database, and then use a template engine to iterate over the data to generate the final page content. Finally, the result is returned and sent over the wire to the requesting browser. The backend server is not just returning static files, but runs application code that handles requests and returns results based on the request and the current state of the database. </p> <p>Jekyll works very much the same way, but with the difference that all the resources available to be requested are pre-generated. For this reason, Jekyll and similar frameworks are often referred to as <em>static Web site generators</em>. Keep in mind that Jekyll is very different from hand-coding your own HTML pages. Jekyll provides an advanced template system, enables data-driven page generation, has built-in support for <a href="" title="Markdown">Markdown</a> and <a href="" title="Sass">Sass</a>, and can incrementally re-compile pages when their content changes. </p> <p>The whole point of a static Web site generator is that it pre-generates the content to be served, i.e. the Web pages. This greatly simplifies content deployment. The entire Web site can be deployed as simple resources on a Web server (e.g. <a href="" title="nginx">nginx</a>), just like images or other static content. There is no need to run a separate application on the backend server to interpret requests and determine the exact content to return. </p> <h2 id="what-does-static-mean">What does static mean?</h2> <p>The word “static” sounds limiting. Really, the word “static” should only be used to describe how the resulting Web site is deployed. This does not mean that you cannot create data-driven Web sites, or that you cannot provide dynamic experiences to users. There just isn’t any dynamism on the backend, <em>per se</em>. </p> <p>Much of the dynamism on the Web today comes from the client-side, taking place directly in the browser using JavaScript and CSS. Client-side frameworks like <a href="" title="AngularJS">AngularJS</a> and <a href="" title="Ember">Ember</a> help develop rich user experiences, and other JavaScript frameworks make it easy to integrate dynamic content like maps (e.g. <a href="" title="Leaflet">Leaflet</a>). </p> <p>Jekyll also supports <a href="" title="Data files">data files</a> that can be parsed and used during content generation. These data files can be created by hand, by tools, or generated from databases. But keep in mind that when the data changes, the Web pages that serve that data need to be re-generated. </p> <p>Finally, using Jekyll to generate the content of a Web site or application does not preclude using a separate companion backend application to handle data storage and management, e.g. via an HTTP-based API. I have not tried it myself, but this setup could provide for a nice separation between Web page content and backend data services. </p> <p>To create and build your first Jekyll Web site is easy:<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup> </p> <pre><code>&gt; jekyll new &gt; cd &gt; jekyll build </code></pre> <p>The generated site will end up in a directory called <code>_site</code>. </p> <h2 id="deploying-with-git">Deploying with Git</h2> <p>Jekyll is a very interesting and powerful framework, but I really wanted to discuss deployment scenarios using <a href="" title="Git">Git</a>. The basic idea we want to achieve is the following: </p> <ol> <li>Keep all content version controlled using Git</li> <li>Keep a remote Git repository on the deployment server</li> <li>To re-deploy updated content, simply push changes to the remote Git repository </li> </ol> <p>Git has two kinds of repositories: “bare” and “non-bare”. The main difference is that bare repositories do not contain a working directory. That is, you cannot directly see the files that are part of that repository.<sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup> Another minor difference is the convention that the name of the root directory of bare repositories ends with <code>.git</code>. It is not recommended that you push changes from a non-bare repository to another non-bare repository, because it already contains a working directory and it can cause issues unless you know what you are doing.</p> <p>We want to have:</p> <ul> <li>A bare repository on the deployment server</li> <li>A non-bare clone of the repository on our local machine where we do work </li> </ul> <p>We login to our remote server to setup our bare repository: </p> <pre><code>remote&gt; mkdir -p ~/git/ remote&gt; cd ~/git/ remote&gt; git init --bare --shared . </code></pre> <p>Notice the <code>.git</code> ending on the directory name. Then on our local machine: </p> <pre><code>local&gt; mkdir -p ~/git/ local&gt; jekyll new ~/git/ local&gt; cd ~/git/ local&gt; git init . local&gt; git add . local&gt; git commit -m "first release" local&gt; git remote add origin local&gt; git push --set-upstream origin master </code></pre> <p>For the non-initiated Git user, this might seem like a lot of strange commands. But we did the following: We created an empty bare Git repository on the remote machine to hold the content of our site. We then created the initial content on our local machine using Jekyll (it creates sample pages that we can modify later). We made the Jekyll site a non-bare Git repository, added all the files in the current directory to be committed, and then committed them using a simple commit message: “first release”. Next we told Git about our remote Git repository, found at <code>~/git/</code> on the machine <code></code> that we can access using the user <code>user</code>. In the last step we pushed our first commit to the remote server (<code>--set-upstream</code> tells Git to remember the link between our local repository and the remote one). </p> <h3 id="automatic-deployment-with-git-hooks">Automatic deployment with Git hooks</h3> <p>Almost done. The last step is to configure the remote repository to re-build the entire Jekyll site when new content is pushed into that repository. This is done using <a href="" title="Git Hooks">Git hooks</a>. Hooks are just scripts that are run at certain times in the Git workflow process. We will use the <code>post-receive</code> hook that runs after Git has done all its work. For bare Git repositories, hooks are stored in a folder called <code>hooks</code> in the root repository directory. Make sure that there is a file called <code>post-receive</code> with the following contents and that it is executable. </p> <pre><code>remote&gt; cat ~/git/ #!/bin/sh GIT_REPO=$HOME/git/ TMP_GIT_CLONE=$HOME/tmp/ PUBLIC_WWW=$HOME/webapps/lovelysite git clone $GIT_REPO $TMP_GIT_CLONE jekyll build -s $TMP_GIT_CLONE -d $PUBLIC_WWW rm -Rf $TMP_GIT_CLONE exit remote&gt; chmod 755 ~/git/ </code></pre> <p>The assumption here is that <code>$HOME/webapps/lovelysite</code> is the location where your Web provider tells you to put the contents of your site.</p> <p>Now, when we push new content, the <code>post-receive</code> script is run. The script creates a temporary clone of the repository with a working directory, and then uses that to build the site. Once the site is built and deployed, the temporary clone is removed. </p> <p>With this setup, your deployment strategy becomes: </p> <pre><code>local&gt; git add [files to update] local&gt; git commit -m "added new content" local&gt; git push </code></pre> <p>It doesn’t get much more simple than that. Revision control and deployment strategy closely integrated.<sup id="fnref:4"><a href="#fn:4" class="footnote">4</a></sup> </p> <div class="footnotes"> <ol> <li id="fn:1"> <p>There are <a href="" title="Comparison of web application frameworks">many Web application frameworks available</a>, in a range of different languages and with different architectural styles and patterns. <a href="#fnref:1" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:2"> <p>Jekyll first has to be installed and the <code>jekyll</code> binary has to be on your <code>$PATH</code>. See <a href="" title="Jekyll">Jekyll’s Web site</a> for instructions. <a href="#fnref:2" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:3"> <p>There is a relationship between bare and non-bare repositories. Each non-bare repository has a <code>.git</code> folder that contains an object database with all the revisions, along with other metadata and scripts. Next to the <code>.git</code> folder are all the files in the working directory. A bare repository on the other hand does not have a working directory, instead all the files in the <code>.git</code> directory in a non-bare repository are in the root repository directory. <a href="#fnref:3" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:4"> <p>In an ideal world, you also want to have a non-deployment remote repository that you can push to without causing a re-deployment. You can for example host this on <a href="" title="GitHub">GitHub</a> or in your <a href="" title="Dropbox">Dropbox</a> folder. Make this remote repository your default remote repository, and name your remote deployment repository <code>deploy</code>. Now your deployment strategy becomes: <code>git push deploy master</code>. <a href="#fnref:4" class="reversefootnote">&#8617;</a></p> </li> </ol> </div> Sun, 25 Jan 2015 11:52:00 +0000 post → Data is a public good <p>Kathryn Sullivan, the <a href="" title="NOAA">US National Oceanic and Atmospheric Administration (NOAA)</a> administrator:</p> <blockquote> <p>The big data revolution could lead to currently unimagined uses for the data we receive from satellites. Entrepreneurs could come up with new applications and ideas for mashing up data. But the data itself should, I believe, be regarded as a public good. How to guarantee this, in a world where public budgets are squeezed and space exploration is becoming increasingly affordable for private players, is a question that deserves serious thought and active engagement.</p> </blockquote> <p>I certainly agree that data is a public good, and unrestricted access to data is important to fuel new discoveries and put it to new valuable uses. </p> <p>Sullivan brings up a good question about the balance between government and private-funded satellite programs (or access to space in general). The private sector is certainly contributing to exploring space and the earth, which is good for the short term, but will it also serve us in the long term? </p> <p><a href="" title="SpaceX">SpaceX</a> is resupplying the International Space Station (ISS) and <a href="" title="Planet Labs">Planet Labs</a> has <a href="" title="Planet Labs Gets $95 Million More to Photograph the Whole World Every Day">raised well over $100 million</a> in order to take high-resolution (5 meters or less) images of the entire globe. </p> <p><a href="" title="Permanent link to Data is a public good">∞ Permalink</a></p> Thu, 22 Jan 2015 21:20:00 +0000 link Increased productivity with shell scripting <p>In the past 20 years or so I’ve used and played with different computers and operating systems, but for the last 10 years I’ve almost exclusively used a Mac, both at home, and at work when I can get away with it (and I have).<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup> </p> <p>The Mac is a great productivity tool for my work-related tasks, which includes software development, application deployment, database management and those sort of things. One big reason the Mac is so productive to me is that the core of the Mac operating system, OS X, is POSIX-compliant and provides a <a href="" title="Command-line Interface">command-line interface</a>, or CLI for short. </p> <p>The CLI is powered by a <a href="" title="Unix shell">Unix shell</a> that interprets the commands you enter and executes them. There are many different shell interpreters but the default one on OS X is <a href="" title="BASH">Bash</a>. </p> <p>Bash provides a lot of powerful features, but I’ve found that learning just a few simple commands and tricks can go a long way to becoming more productive, especially in automating repetitive tasks. <!--more--></p> <h2 id="the-basics">The Basics</h2> <p>First of all, there are a lot of basic shell commands that are needed to do almost anything useful. Examples include: <code>ls</code>, <code>cp</code>, <code>mv</code>, <code>more</code>, <code>grep</code>, <code>tail</code>, <code>cat</code>, <code>echo</code>, <code>wc</code>, <code>sleep</code>.<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup> I will not describe these commands here but it’s easy to find examples of how they work on the web, or you can read the documentation using the <code>man</code> command, e.g.:</p> <pre><code>&gt; man mv </code></pre> <p>Another useful functionality provided in the shell is “pipes”, which is a method to direct the output of one command to become the input of another command. This makes it possible to compose commands into more complex structures. A simple example: </p> <pre><code>&gt; cat file.txt | grep "world" | wc -l </code></pre> <p>The command above outputs the content of the file <code>file.txt</code> to <code>grep</code>, which matches lines containing the word “world”. These matching lines are then provided as input to <code>wc</code> (short for “word count”), which counts the lines (the <code>-l</code> flag tells <code>wc</code> to only count lines). So, the above command answers the question: How many lines in <code>file.txt</code> contain the word “world”? </p> <h2 id="repetition">Repetition</h2> <p>It is often useful to execute the same command multiple times for different values. This facilitates automation and batch jobs. The simplest example is the for-loop, common from most programming languages. The command takes a variable name, and a list of entries to iterate over. The simplest example would be: </p> <pre><code>&gt; for i in 1 2 3; do echo $i; done 1 2 3 </code></pre> <p>The variable name given is <code>i</code> and it can be referenced using <code>$i</code>. The list consists of the three numbers 1, 2, and 3. For each item in the list, we execute the command <code>echo $i</code>. As expected, the result is that the shell prints the numbers 1, 2, and 3. </p> <p>We can do something slightly more useful: </p> <pre><code>&gt; for i in `ls *`; do echo $i; done </code></pre> <p>The above command is a complicated way of listing all files in the current directory. The result from the command <code>ls *</code> makes up the list we are looping over (notice the back ticks). Then each entry, i.e. each filename, is echoed back in the shell. </p> <p>There is also a <code>while</code> command that loops until a condition is no longer true. An example would look like this: </p> <pre><code>&gt; i=1; while [ $i -lt 4 ]; do echo $i; i=$[$i+1]; done 1 2 3 </code></pre> <p>The loop condition is true while the variable <code>i</code> is less than 4. The variable <code>i</code> is initially set to 1 and is incremented each time the loop body is executed. The semi-colon separates two commands, executing the second after the first. </p> <h2 id="processing-data">Processing Data</h2> <p>There is another set of commands that are useful for processing data during batch operations. We start with <code>cut</code>. The <code>cut</code> command allows extracting specific values from a (structured) line of text. Here is an example: </p> <pre><code>&gt; echo "Hello terrible world" | cut -d " " -f 1,3 Hello world </code></pre> <p>We specify that the delimiter is a space, and that we want the first and third fields (using the <code>-f</code> flag). <code>cut</code> can also find specific byte or character ranges. </p> <p>Another useful command is <code>read</code> which can assign variable names to fields in a line of text. </p> <pre><code>&gt; read f1 f2 f3 &lt;&lt;&lt; "Hello terrible world"; echo $f1 $f3 Hello world </code></pre> <p>The <code>&lt;&lt;&lt;</code> operator indicates a <a href="" title="Here String">“here string”</a>, which I will not go into here, but essentially it provides a way to specify input to the <code>read</code> command <em>inline</em>, directly in your script, without necessarily reading the contents from a file. </p> <p>Together, <code>cut</code> and <code>read</code> allow us to filter data sets, and extract data into variables that we can reference in our scripts. Here is a contrived example: </p> <pre><code>&gt; read f1 f2 &lt;&lt;&lt; `echo "Hello terrible world" | cut -d " " -f 1,3`; echo $f1 $f2 Hello world </code></pre> <h2 id="example-images">Example: Images</h2> <p>Let’s put our knowledge to practice. <a href="" title="ImageMagick">ImageMagick</a> is a set of command-line tools for processing images. ImageMagick provides a tool called <code>identify</code> that shows details about a particular image. </p> <pre><code>&gt; identify image1.png image1.png PNG 240x135 240x135+0+0 8-bit sRGB 15.3KB 0.000u 0:00.009 </code></pre> <p>We see the image name itself, the image type, the image dimensions and some other metadata, each piece of information separated by a space (good to know, right?). If we are only interested in the image name and the dimension we can do this: </p> <pre><code>&gt; identify image1.png | cut -d " " -f 1,3 image1.png 240x135 </code></pre> <p>Need to move things around? </p> <pre><code>&gt; read image type dim &lt;&lt;&lt; `identify image1.png | cut -d " " -f 1,2,3`; echo $type $image $dim PNG image1.png 240x135 </code></pre> <p>If you have a directory full of <a href="" title="Portable Network Graphics">PNG</a> images, you can do this: </p> <pre><code>&gt; for i in `ls *.png`; do identify $i | cut -d " " -f 1,3; done image1.png 240x135 image2.png 240x135 </code></pre> <p>You can perhaps imagine how easy it is to batch-convert a folder full of images using the <code>convert</code> tool, also part of ImageMagick. </p> <p>Bonus example where we use <code>printf</code> to re-format the output: </p> <pre><code>&gt; for i in `ls *.png`; do read file size &lt;&lt;&lt; `identify $i | cut -d " " -f 1,3`; printf "%-20s %s\n" "$file" "$size"; done image1.png 240x135 image2.png 240x135 </code></pre> <h2 id="example-database">Example: Database</h2> <p>Here is another example that I’ve come across if my work where scripting is very useful. Let’s assume we have a MySQL table with summary reports for a set of data records. Each report, among other things, has a unique ID (column <code>id</code>) and a count (column <code>count</code>) for the number of records in that report. </p> <p>If you execute a MySQL <code>SELECT</code> query from the MySQL prompt, you get a table-like structure: </p> <pre><code>mysql&gt; SELECT id,count FROM `report`; +-----+--------+ | id | count | +-----+--------+ | 1 | 100000 | | 2 | 10000 | | 3 | 1000 | +-----+--------+ </code></pre> <p>However, if you execute the same query from the command-line and re-direct the result to a file, you get a space-separated format, which is easier to handle given our newly acquired skills of filtering and processing such data. </p> <pre><code>&gt; mysql -e "SELECT id,count FROM \`report\`" &gt; data.txt &gt; cat data.txt id count 1 100000 2 10000 3 1000 </code></pre> <p>Let’s further assume we have a Web service that calculates some statistics for each report, given the report’s ID, and we want to process a large amount of reports. The first thing to do is to remove the first line in the file, which contains the column headers. For sake of simplicity, this can be done manually using a text editor. </p> <p>Now we can read each line of the file using a somewhat familiar command:<sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup> </p> <pre><code>&gt; while read i; do echo $i; done &lt; data.txt 1 100000 2 10000 3 1000 </code></pre> <p>The last part of the command, <code>&lt; data.txt</code>, sends the contents of the file as input to <code>read</code>, causing each line in turn to be read and associated with the variable <code>i</code>. </p> <p>Now we can break up the data for each line:</p> <pre><code>&gt; while read i; do read id count &lt;&lt;&lt; $i; echo "updating: $id; count: $count"; done &lt; data.txt updating: 1; count: 100000 updating: 2; count: 10000 updating: 3; count: 1000 </code></pre> <p>Now, of course, instead of simply echoing the report ID we can make a request to the Web service using <code>curl</code>, something like this: </p> <pre><code>&gt; curl -X POST -d "{\"id\": 1}" http://localhost/api/update/ --header "Content-Type:application/json" </code></pre> <p>Better yet, put this command in a script: </p> <pre><code>&gt; cat #!/bin/sh curl -X POST -d "{\"id\": $2}" http://$1/api/update/ --header "Content-Type:application/json" &gt; chmod 755 </code></pre> <p>This script can be called like this: </p> <pre><code>&gt; ./ localhost 1 { status: OK } </code></pre> <p>The assumption is that this call to the Web service running at <a href="http://localhost/api/update/">http://localhost/api/update/</a> will update the statistics for the report referenced by the ID (here: 1). The “{ status: OK }” output on the console is the result from the Web service call, here showing that the request was successful. </p> <p>Putting all this together we can now do the following: </p> <pre><code>&gt; while read i; do read id count &lt;&lt;&lt; $i; echo "updating: $id; count: $count"; ./ localhost $id; done &lt; data.txt updating: 1; count: 100000 { status: OK } updating: 2; count: 10000 { status: OK } updating: 3; count: 1000 { status: OK } </code></pre> <h2 id="conclusions">Conclusions</h2> <p>The result of all this is that we have automated the extraction of information from a MySQL database and performed data processing using a Web service with a few simple shell commands. </p> <p>There is so much more that can be done, but this has hopefully given an indication of the power of knowing some simple commands, and really how easy it is to be more productive using the shell and the command line. </p> <div class="footnotes"> <ol> <li id="fn:1"> <p>Most recently I’ve used the 13” MacBook Pro, which is a great choice for a lot of situations. <a href="#fnref:1" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:2"> <p>It is also very useful to know a text editor that can be run in the shell, rather than needing a windowing server to display the interface. This is in particular useful in cloud infrastructures and in any kind of remote login scenario where this might be the only option. My current recommendation is <a href="" title="Vim">Vim</a>. <a href="#fnref:2" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:3"> <p>For this to work properly the <a href="" title="Internal field separator">“IFS”</a> variable value needs to be inspected, or best, cleared, which can be done like so: <code>unset IFS</code>. <a href="#fnref:3" class="reversefootnote">&#8617;</a></p> </li> </ol> </div> Sun, 18 Jan 2015 16:15:00 +0000 post