Hugo<![CDATA[Stéfan's blog]]> 2020-05-21T10:57:56-07:00 <![CDATA[Voice capture org-mode tasks on Android]]> 2020-04-28T00:00:00+00:00 0001-01-01T00:00:00+00:00 Where I show how to hook up org-mode and Google Assistant

I often want to capture tasks on the go—in a hurry. When there’s no time to fire up organice or Orgzly, being able to transcribe tasks comes in really handy.

In this post, I show how, on Android phones, you can hook up Google’s Assistant with org-mode, so that you can speak notes and have them appear as TODO items in a buffer.

Set up Google Assistant

First, we need to teach Google Assistant a new keyword, and tell it to store transcribed notes in an accessible location. We do this via the free If This Then That service. Add the “Log notes in a Google Drive spreadsheet” applet, and configure it as follows:

  • What do you want to say? Add a task to $
  • What’s another way to say it? (optional) new task $
  • And another way? (optional) task $
  • Drive folder path (optional) Google Assistant

This would allow you to say task <description> and have Google Assistant log that to a spreadsheet in the Google Assistant folder of your drive.

Save the applet and try it out: launch Google Assistant and say “task test out capture system”. Then, locate and open the new spreadsheet in your Google drive. The URL should be of the form:

Note down that long string after /d/—this is your spreadsheet ID.

Set up org-mode conversion

Go to Tools -> Script Editor, and include the script provided at

You have to customize two variables: the spreadsheet ID, and a random “token” (a password to make it harder for other to abuse the service).

Now, publish the script to the web: Publish -> Deploy as web app.... Set Who has access to the app to Anyone, even anonymous and note down the published URL.

Use it!

I have the following script that downloads TODOs and append them to an org-file:



curl -s -S -L -d "$TOKEN" "$ASSISTANT_TO_ORG_URL?clear=1" >> $ORG_INBOX

I then have the following in my daily org checklist:

[[shell:~/scripts/][fetch tasks]] : [[file:~/org/][tasks]]

The first link launches the script that fetches the latest tasks, and the second opens the tasks file.


Having a quick, hands-free way to capture tasks has been tremendously helpful to me. I hope you find it useful too!

<![CDATA[Scientific Python at Microscopy & MicroAnalysis 2019]]> 2019-08-06T00:00:00+00:00 0001-01-01T00:00:00+00:00 Where we demonstrate a new way of doing interactive image analysis.

Today, I presented a talk titled “Scientific Python: A Mature Computational Ecosystem for Microscopy” [PDF] at the Microscopy and MicroAnalysis conference in Portland.

A few members of the audience familiar with scientific Python told me they had learned something, so I’ll highlight the few topics that I think may have qualified.

SciPy 1.0 paper

The first official release of SciPy was in 2001, and a mere 16 years later we reached 1.0. This says a lot about the developer community, and how careful they are to label their own work as “mature”! To celebrate this project milestone, we published a preprint on arXiv that outlines the project history and its current status. It mentions, among other achievements, that SciPy was instrumental in the first gravitational wave detection, as well as the recent imaging of the black hole in Messier 87.

NumPy __array_function__ protocol

The 1.17 release of NumPy (2019-07-26) has support for a new array function protocol, that allows external libraries to pass their array-like objects through NumPy without them being horribly mangled. E.g., you may call NumPy’s sum on a CuPy array: the computation will happen on the GPU, and the resulting array will still be a CuPy array.

Here is an example:

In [24]: import cupy as cp

In [25]: x = cp.random.random([10, 10])

In [26]: y = x.sum(axis=0)

In [27]: type(y), y.shape
Out[27]: (cupy.core.core.ndarray, (10,))

In [28]: import numpy as np

In [29]: z = np.sum(x, axis=0)

In [30]: type(z), z.shape
Out[30]: (cupy.core.core.ndarray, (10,))

Note how the result is the same, whether you use CuPy or NumPy’s sum.

Whereas NumPy used to be the reference implementation for array computation in Python, it is fast evolving into a standard API, implemented by multiple libraries.

PyTorch and TensorFlow easily consume Python images

Images in scientific Python (scikit-image, opencv, etc.) are represented as NumPy arrays. It is trivial to pass these arrays into deep learning libraries such as TensorFlow:

from tensorflow.keras.applications.inception_v3 import (
    InceptionV3, preprocess_input, decode_predictions
from skimage import transform

net = InceptionV3()

def inception_predict(image):
    # Rescale image to 299x299, as required by InceptionV3
    image_prep = transform.resize(image, (299, 299, 3), mode='reflect')
    # Scale image values to [-1, 1], as required by InceptionV3
    image_prep = (img_as_float(image_prep) - 0.5) * 2
    predictions = decode_predictions(
        net.predict(image_prep[None, ...])
    plt.imshow(image, cmap='gray')
    for pred in predictions[0]:
        (n, klass, prob) = pred
        print(f'{klass:>15} ({prob:.3f})')

Chelsea the Cat

For example, when running inception_predict on, I get:

Egyptian_cat (0.904)
       tabby (0.054)
   tiger_cat (0.035)
        lynx (0.000)
 plastic_bag (0.000)

Looks about right!


Philipp Hanslovsky, at SciPy2019, demonstrated his Python ↔ Java bridge called imglyb. In contrast to many previous efforts, this library allows you to share memory between Python and Java, avoiding costly (and, potentially fatal, dependent on memory constraints) reallocations. E.g., he showed how to manipulate volumes of data (3-D arrays) in Python, and to then view those using ImageJ’s impressive BigDataViewer, which can rapidly slice through the volume at an arbitrary plane.

Lazy viewing of data using dask

This is a trick I borrowed from Matt Rocklin’s blog post.

When you have a number of large images that, together, form a stack (3-D volume), it may not be possible to load the entire stack into memory. Instead, you can use dask to lazily access parts of the volume on an as-needed basis.

This is achieved in four steps:

  1. Convert into a delayed function, i.e. instead of returning the image itself it returns a dask Delayed object (similar to a Future or a Promise), that can fetch the image when needed.

  2. Use this function to load all images. The operation is instantaneous, returning a list of Delayed objects.

  3. Convert each Delayed object to a dask Array.

  4. Stack all of these dask Arrays to form the volume.

Note that each one of these steps should execute almost instantaneously; no images files are accessed on disk: that only happens once we start operating on the dask Array volume.

Here is the code:

from glob import glob

from dask import delayed
import dask.array as da

from skimage import io

# Read one image to get dimensions
image = io.imread('samples/Test_TIRR_0_1p5_B0p2_01000.tiff')

# Turn imread into a delayed function, so that it does not immediately
# load an image file from disk
imread = delayed(io.imread, pure=True)

# Create a list of all our samples; since a delayed version of `imread`
# is used, no work is done immediately
samples = [imread(f) for f in sorted(glob('samples/*.tiff'))]

# Convert each "delayed" object in the list above into a dask array
sample_arrays = [da.from_delayed(sample, shape=image.shape, dtype=np.uint8) for sample in samples]

# Stack all these arrays into a volume
vol = da.stack(sample_arrays)

I have 101 slices of 2048x2048 each, so the resulting dask Array volume (at this stage fully virtual, without any data inside) is:

We can do numerous operations on this array, such as summing it with vol.sum(axis=0), although this still yields an uncomputed dask Array. To get actual values, we need to call:



To visualize a volume like the one above, I could have sliced into it and displayed the result using matplotlib. However, I used this opportunity to play around with a brand new open source image viewer called Napari.

Napari allows you to visualize layers interactively, similarly to GIMP or Photoshop. In Napari’s case, these layers can be images, labels, points, and a few others.

While this isn’t explicitly documented (Napari is still in alpha!), I had some insider knowledge (👋 J!) that Napari supports both dask and Zarr arrays. So, we can pass in our volume from the example above as follows:

import napari

with napari.gui_qt():
    viewer = napari.view(vol, clim_range=(0, 255))

(Instead of the context manager, you may also use %gui = qt in Jupyter or IPython.)

I also happened to have ground truth labels available, so I loaded those up the same way I did the volume, and added it to the visualization:

viewer.add_labels(labels, name='Labels')

If you’d like to play with Napari yourself, I have a 3D cell segmentation example available online.


Toward the conclusion of my talk, I emphasized the role of community in building healthy scientific software ecosystems. In the end, it is all about people. I briefly highlight two community groups:

  • PanGeo, whom I think sets a great example of how to organize field-specific interest around existing open source tools, and building scalable online analysis platforms without reinventing the wheel.

  • OME, the Open Microscopy Environment, who is leading the charge on open data exchange formats for microscopy. Interestingly, it looks like Zarr—the chunked, compressed array container—may well be part of the next open standard they recommend.


Thank you to the organizers of M&M 2019 for inviting me to speak; I very much enjoyed our session, and look forward to working with this community on making scientific Python an even better platform for mirocroscopy analysis!

<![CDATA[Designing for Intelligent Augmentation]]> 2019-07-05T00:00:00+00:00 0001-01-01T00:00:00+00:00 How can we better design systems that combine computational inference and human intelligence?

The first issue of the Harvard Data Science Review was published last month. I enjoyed Michael Jordan’s opinion piece, in which he considers progress made (and, yes, hype!) in so-called artificial intelligence. Particularly, I was excited about the phrase “Intelligent Augmentation”—the pursuit of systems that enhances our ability to reason about a subject, by providing additional analysis, data, and emphasis. Jordan writes:

IA will also remain quite essential, because for the foreseeable future, computers will not be able to match humans in their ability to reason abstractly about real-world situations. We will need well-thought-out interactions of humans and computers to solve our most pressing problems. And we will want computers to trigger new levels of human creativity, not replace human creativity (whatever that might mean).

A week ago, I attended a family event where I got pulled into a surprisingly animated argument around Google Maps: is it accurate, is it helpful, and what are we losing by relying on it. The argument took all the predictable turns (yes, kids nowadays cannot use maps any more, and if Google decided to summon us all to the Mountainview HQ as an April Fool’s joke, the results would be comically sad). But an interesting outcome was the question: “How can we better design systems to help humans make good decisions?”

Lemmings walking off a cliff, which they do—but to migrate, not commit suicide

E.g., Google Maps tells you where to drive—they maybe even give you one or two route options. But the overarching goal is to get you to your destination in the shortest amount of time. What if you felt like taking a scenic drive, or wanted to explore a bit? In that case, a map that showed a compass and traffic for all nearby roads would be much more helpful. How many times do we drive past a national monument such as Bodie, or a street festival on the next block over without realizing it? Maps could certainly alert us to these.

At the Berkeley Institute for Data Science, I build a lot of open source research software. I’ve learned that systems that work with humans are often both simpler to develop and ultimately more effective than fully automated systems. When we wrote Inselect with the Natural History Museum, it would have been very hard to do a 100% accurate segmentation of insect speciments (especially since many of the photos this would be applied to contained insects unseen during training). But if you can provide reasonable accuracy, humans can easily adjust for minor discrepancies and still save a lot of time.

With this blog post, I encourage software designers to:

Think about how to best empower your users, rather than to prescribe their behavior implicitly through design decisions. Be mindful that users may have experience to contribute and a desire to execute their own plan; augment their ability to do so effectively.

Circling back to Jordan’s article, I encourage you to read the various commentaries (for now, the easiest way to find them is to scroll down to the article on the journal front-page). I enjoyed, e.g., David Donoho’s, where he discusses the requirements for “true intelligence” in AI, although I think he may have misinterpreted Jordan’s intent with the term “augmented intelligence”.

I’ll end with a quote from Greg Cane’s commentary:

For the humanist, intelligence augmentation must now and forever be our goal. Machine learning only matters insofar as it makes us fundamentally more intelligent and deepens our understanding.

<![CDATA[Linking to emails in org-mode (using neomutt)]]> 2018-10-31T00:00:00+00:00 0001-01-01T00:00:00+00:00 Where we store links to emails in org-mode, and open them using neomutt.

Update 2018-11-2: Change the URL scheme to message://. See “Other Systems” below.

org-mode is, to me, is one of the most valuable parts of the emacs ecosystem. I use it to take notes, plan projects, manage tasks, and write & publish documents.

Nowadays, a lot of work arrives via email, and so it is helpful to be able to refer to messages directly from my notes or lists of tasks.

The simplest option might be to store URLs pointing to an online inbox such as Fastmail or GMail, but I wanted a solution that was both future proof (i.e., what if I moved my emails to a different provider?) and worked with my terminal-based mail client of choice, neomutt.

I started with a solution provided by Stefano Zacchiroli, and simplified it for my specific use-case.


The solution has two parts: sending email links from neomutt to Emacs, and later opening those links from Emacs by invoking neomutt. The first achieved via org-protocol, the latter via launching neomutt and then simulating keypresses.

When launching neomutt, we have to tell it in which directory the message lives. We therefore use notmuch to find the message file first, based on its Message-ID. maildir-utils would be another way of doing so. Please note that you have to have notmuch or maildir-utils set up already for this scheme to work.

I initially avoided the org-protocol package, because installation looked complicated. That, it turns out, is only the case if you care about web browser integration, which we don’t.

Neomutt configuration

First, we have a Python script that can parse an e-mail and share the Message-ID and Subject with emacs. I call it, and make it executable using chmod +x

#!/usr/bin/env python3

import sys
import email
import subprocess
import urllib.parse

# Parse the email from standard input
message_bytes =
message = email.message_from_bytes(message_bytes)

# Grab the relevant message headers
message_id = urllib.parse.quote(message['message-id'][1:-1])
subject = message['subject']

# Ask emacsclient to save a link to the message

We then configure neomutt (typically in ~/.muttrc) to call the script with a shortcut. I chose Esc-L (the same as Alt-L).

macro index,pager \el "|~/scripts/\n"

Emacs configuration

Using org-protocol, we instruct emacsclient to intercept URLs with the org-protocol:// scheme, as used by our script. We also tell org-mode how to handle special URLs of the form message:// Neomutt needs to know which Maildir folder to open, so we ask notmuch to tell us where the message is located.

In my ~/.emacs file I have:

; Make sure org-protocol is loaded
; Now, org-protocol:// schemas are intercepted.
(require org-protocol)

; Call this function, which spawns neomutt, whenever org-mode
; tries to open a link of the form message://
(defun stefanv/mutt-open-message (message-id)
  "In neomutt, open the email with the the given Message-ID"
      ((message-id (replace-regexp-in-string "^/*" "" message-id))
                   (format "notmuch search --output=files id:%s" message-id))))
       (mail-dir (replace-regexp-in-string "/\\(cur\\|new\\|tmp\\)/$" ""
                                           (file-name-directory mail-file)))
       (process-id (concat "neomutt-" message-id))
       (message-id-escaped (regexp-quote message-id))
        (format "l~i %s\n\n" (shell-quote-argument message-id-escaped)))
       (mutt-command (list "neomutt" "-R" "-f" mail-dir
                           "-e" (format "push '%s'" mutt-keystrokes))))

    (message "Launching neomutt for message %s" message-id)
    (call-process "setsid" nil nil
                   "-f" "gnome-terminal" "--window" "--"

; Whenever org-mode sees a link starting with `message://`, it
; calls our `mutt-open-message` function
(org-add-link-type "message" 'stefanv/mutt-open-message)

There are a few caveats: if you use maildir-utils, the search command is mu find -f l i:%s instead of notmuch; and if you are not on Linux, then setsid (which we use to launch a detached background process) is not going to work, and you will want to use a different terminal emulator.

Other Systems

Charl Botha mentioned in the comments that, on MacOS, org-mac-link lets you grab hyperlinks from a wide variety of apps. Email messages, specifically, are stored as message://message-id URLs, which MacOS knows how to open. This post has been updated to use the same link schema.


That’s it! I’ve added the code to Please file issues and PRs there, or tell me about your use cases in the comments below.

<![CDATA[Search headers in org-mode]]> 2016-12-15T00:00:00+00:00 0001-01-01T00:00:00+00:00 Where we show a quick way of navigating org-mode files.

In org-mode, I often have the need to jump to a top-level heading matching some word.

Since an org-mode buffer can be searched just like any other, I can simply invoke forward search with C-s, but this will match all occurrences of the text, instead of limiting the search to headings only.

This makes it hard to search for a phrase like “Travel”, for which I have a top-level heading, but also often occurs elsewhere in my notes.

I have a solution of the following form:

  1. Launch a regular expression search
  2. Pre-fill the text input with ^* so that only headings are matched.

First, define a custom search function. It puts the keys ^* in the “unread command events” list (i.e, a list of events waiting to be seen by emacs), and then launches interactive forward regular expression search.

(defun stefan/isearch-heading ()
  (setq unread-command-events (listify-key-sequence "^* "))
  (isearch-mode t t nil t))

Next, we add a keybinding for org-mode:

(defun org-mode-keys ()
  (local-set-key (kbd "C-c g") 'stefan/isearch-heading)
(add-hook 'org-mode-hook 'org-mode-keys)

And that’s it! Pressing C-c g (for “go”) in org-mode will now present you with a search prompt. Typing a heading name will take you there directly, at which point you can choose to expand it with the TAB key.

<![CDATA[WebSockets in Python (and some Redux)]]> 2016-10-20T00:00:00+00:00 0001-01-01T00:00:00+00:00 Where we make pushing messages from a python backend to the browser easy.

As part of our work on Cesium and its web frontend, we’ve developed an easy mechanism for Python web developers to push messages from their Python backends to the browser.

There are plenty of potential use cases, but consider, e.g., that you want to verify a credit card number submitted by your user. Traditionally, you’d submit the number, and then poll the backend repeatedly from the browser. Not very elegant :/

But with a WebSocket connection, you submit the credit card number and then forget about it. The server will let you know when it’s done by pushing a message to the frontend.

Not only does this solve the annoying polling problem, but it opens up the door to an entirely new universe of tools, such as Dan Abramov’s fantastic Redux. Many of these Javascript libraries rely on the server being able to notify the frontend when it needs to update itself.

Let’s talk a bit about Redux. The principles behind it are simple and elegant:

  1. The entire state of your app is centrally stored (in the equivalent of a Python dictionary);
  2. the state is immutable, and
  3. can only be updated through a central function call.

That centralization in turn enables other features such as logging, hot reloading, time travel, etc.

One of the great joys of Redux lies in moving away from the traditional Model-View-Controller pattern. With MVC, you are never quite sure how changes propagate through the system. With Redux, it is highly predictable. Say your app has a toggle button, and associated state {toggle: true}. An action (e.g. “the red button was clicked”) is submitted to the central dispatcher which then calculates the new state of the app:

new_state = reduce(current_state, action)

The implementation could look something like this:

function reduce(current_state, action) {
  switch (action.type) {
    case 'toggle_button':
      return {toggle: !current_state.toggle}
      return current_state;

The toggle button monitors the app state, and when state['toggle'] is updated, re-renders itself.

By vastly simplifying flow of information, by disentangling mutation and asynchronicity, and by getting rid of JQuery & hidden state stored somewhere in the bowels of the DOM, Redux has, for me, returned the joy of web development.

But, I’m getting distracted. WebSockets—in Python!

Pushing messages from your Python web server to the user’s browser can now be as simple as this:

from Flow import flow

self.flow.push('', 'message to the user',
               {'data': 'to ship along'})

Please take a look at the more detailed technical description (with code!) on the Cesium blog.

<![CDATA[Compile TensorFlow from source with gcc6.1]]> 2016-09-14T00:00:00+00:00 0001-01-01T00:00:00+00:00 Where we build TensorFlow from scratch.

Due to two bugs in gcc 6.1 affecting the re2 library (one of which has been fixed in 6.2)), TensorFlow cannot be compiled from source one some systems (including my Debian Testing install).

To work around the issue, modify tensorflow/workspace.bzl and change the re2 description to:

  name = "com_googlesource_code_re2",
  remote = "",
  commit = "86503cb89d82b723ae0bce35e1e09524910cd319",

The re2 library is now downloaded from my fork, which applies a one line patch.

Compile the TensorFlow Python package as usual with:

bazel build -c opt //tensorflow/tools/pip_package:build_pip_package

After installing the pip wheel using

pip install /tmp/tensorflow_pkg/*.whl

you should have a working installation. If importing fails with

ImportError: cannot import name 'pywrap_tensorflow'

switch out of the TensorFlow source directory and try again.

<![CDATA[Amazon owns your ebooks]]> 2016-08-21T00:00:00+00:00 0001-01-01T00:00:00+00:00 Where we realize Amazon owns your books.

Did you know that Amazon does not sell ebooks? But, wait, you say, I have bought ebooks from them myself! Sadly, that is not the case. I will summarize part of the Kindle Store Terms of Use quoted at the bottom of this post:

  1. You do not own any Kindle book that you have “bought”, even if it said “buy with one click” in the store (the book still belongs to Amazon, you’re just leasing it).
  2. You may only read Kindle content through sofware approved by Amazon.
  3. You may not copy ebooks “bought” from Amazon to other devices (such as a Kobo or Nook reader). In fact, you may not engage in any activity that bypasses their Digital Rights Management (or DRM).

I bought rent more than 120 Kindle books, and to my shame only realized the invasive nature of these terms now.

Since your agreement with Amazon is simply a license to view material, they have the ability to withdraw that right whenever they choose. Which they have done, in an ironic twist, with George Orwell’s 1984.

As you may imagine, this whole situation riles me up quite badly, and spoils many a dinner-time conversation (sorry, friends). I am particularly irate because I just bought myself a beautiful Kobo Glo HD, in a slow but measured move to lessen my dependence on Amazon. Now, lo and behold, I cannot access any of the 120 books which, if I gave my money to other ebook stores, I would have owned!

So, caveat emptor, dear reader. Do not support Amazon’s great Kindle Swindle.

If you buy books from Kobo, you will be able to export your books to your Kindle devices, albeit encumbered by Digital Rights Management (DRM).

The deal with Digital Rights Management (DRM)

While this post is about Amazon Kindle, it is worth mentioning Digital Rights Management (DRM). DRM is a mechanism that distributers use to lock their books so that you may not easily copy it illegally. Of course, and very predictably, the DRM is not difficult to circumvent, and those who want to steal electronic books do so with impunity. Still, it is probably illegal to remove DRM in the United States, and if you’re a law abiding citizen, you cannot convert a protected ebook to another format such as PDF, even if a PDF would be more convenient for you.

Some distributors are worse than others, though, and unsurprisingly Amazon is one of the worst. Not only do they lock their books (making it difficult to copy to other devices), but they use a proprietary format (.mobi) that only works on their readers. If you want to copy their books to another device, you have to first remove the DRM which, as their terms state, is not allowed. You might start to wonder whether such business practices aren’t anti-competitive and perhaps even illegal.

There is a perfectly usable format for ebooks already in existence (.epub), that is also supported by many different models of readers. So, while locking books with some kind of DRM is far from ideal, it is still preferable to using a proprietary format to boot.

For those buying from the Kobo store, some of the books on their shelves are available DRM-free, and is indicated as follows:

Kobo DRM information

Defective by Design maintains a list of DRM-free publishers and stores.

Please support the Electronic Frontier Foundation, who is currently challenging DMCA Section 1201 (which makes it illegal to circumvent DRM). There is no good reason why you should not be allowed to read material you bought on any device of your choosing. For more on the DRM, refer to author Cory Doctorow’s articles in the Guardian.

Below follows an excerpt, mentioned above, from the Amazon Kindle Store’s Terms of Use.

1. Kindle Content

Use of Kindle Content. Upon your download of Kindle Content and payment of any applicable fees (including applicable taxes), the Content Provider grants you a non-exclusive right to view, use, and display such Kindle Content an unlimited number of times, solely through a Reading Application or as otherwise permitted as part of the Service, solely on the number of Supported Devices specified in the Kindle Store, and solely for your personal, non-commercial use. Kindle Content is licensed, not sold, to you by the Content Provider. The Content Provider may include additional terms for use within its Kindle Content. Those terms will also apply, but this Agreement will govern in the event of a conflict. Some Kindle Content, such as interactive or highly formatted content, may not be available to you on all Reading Applications.

Limitations. Unless specifically indicated otherwise, you may not sell, rent, lease, distribute, broadcast, sublicense, or otherwise assign any rights to the Kindle Content or any portion of it to any third party, and you may not remove or modify any proprietary notices or labels on the Kindle Content. In addition, you may not attempt to bypass, modify, defeat, or otherwise circumvent any digital rights management system or other content protection or features used as part of the Service.

<![CDATA[Python & Matplotlib on OSX]]> 2016-02-26T00:00:00+00:00 0001-01-01T00:00:00+00:00 Where we discuss two ways of installing Matplotlib on MacOS.

One day, we will hopefully have a grand unified build and package management system for Python where everything is free & open and Just Works (TM). Until then, you have two options:

brew Python + pip

  1. brew install python3
  2. pyvenv -v ~/envs/py3
  3. source ~/envs/py3/bin/activate
  4. pip install matplotlib


  • Pip is the standard Python package management tool, and uses the official Python Package Index (PyPi) repository.
  • Wheels on PyPi are built by authors themselves from open recipes.
  • Binary wheels may not be available for all packages.
  • Pip is not the best of package management tools.


  1. Download and install miniconda
  2. conda create -n py3 python=3.5 matplotlib
  3. source activate py3


  • Conda is a great package management tool.
  • Conda environments are well tested; almost everything works out of the box. This includes fast linear algebra using MKL.
  • Some of the conda build recipes are closed and binary wheels may not be available for all packages1.
  • Conda and pip do not always play well together.
  • Conda packages are supported almost exclusively by a single company.

  1. Some members of the community maintain their own channels, but there are still some issues to be aware of when mixing those channels and the official ones. ↩︎

<![CDATA[VPN-only access to the internet]]> 2015-01-24T00:00:00+00:00 0001-01-01T00:00:00+00:00 Where we set up a firewall that allows traffic to leave only via the VPN.

A few days ago, I noticed that my outbound email wasn’t being delivered—it seems as if my ISP blocks access to some outbound ports. An easy workaround is to route traffic through the work VPN, but knowing myself I’d forget to switch it on, leaving my outbound mail stranded.

I needed a way of blocking any access to the internet, unless it was leaving through the VPN (since I was sure I’d notice that pretty quickly).

I’d have preferred to use firewalld, which is neatly integrated into Ubuntu, but as of 01/24/2015 it doesn’t allow filtering outbound traffic1. What follows, then, is a simple approach implementing the following rule:

Block wifi traffic unless it goes to either the local network or the VPN.

Create a script (I called it ~/scripts/fw-up) that sets up the firewall:


# Clear any existing rules
iptables -F

# Allow outbound DNS
iptables -A OUTPUT -p udp --dport 53 -j ACCEPT
iptables -A INPUT -p udp --sport 53 -j ACCEPT

# Allow TCP access to the work VPN
# Replace X.Y below with your VPN address range
iptables -A OUTPUT -p tcp -d X.Y.0.0/16 -o wlan1 -j ACCEPT

# Allow any traffic destined for the vpn to go out
iptables -A OUTPUT -o vpn0 -j ACCEPT

# Allow local traffic
iptables -A OUTPUT -p tcp -o wlan1 -d -j ACCEPT

# Drop everything else on the wifi
iptables -A OUTPUT -p tcp -o wlan1 -j DROP

Make sure the script is set to executable (chmod +x fw-up).

Add a symlink to if-up.d to ensure that the firewall gets built whenever the network is reconfigured:

sudo ln -s ~/scripts/fw-up /etc/network/if-up.d/iptables

Now, whenever you connect to a wifi hotspot, internet traffic will be blocked until you fire up your VPN. If, on occasion, you need to work without the VPN, simply raze the firewall:

sudo iptables -F

Update: Typically, you wouldn’t want the firewall to go up when connecting to the wireless at work, so I added the following conditional in the fw-up script:

iptables -F

if [[ `iwgetid -r` != 'WorkSSID' ]]; then
   # Firewall rules go here

  1. The idea would be to set up a zone, say vpn_only, and to assign your home wifi SSID to it. A direct rule that blocks any non-VPN traffic can then be added to the new zone (this should be possible soon, and is on the firewalld TODO list). ↩︎

<![CDATA[Visualizing and pruning the docker image tree]]> 2015-01-18T00:00:00+00:00 0001-01-01T00:00:00+00:00 Where we prune unused Docker images, and save a lot of disk space.

Visualizing the Docker image dependency tree is a useful way of checking image sizes and inter-image dependencies. Since images occupy a lot of disk space, we may also want to prune unused images.

Visualizing image dependencies

In version 1.2.0 of Docker, the image dependency tree is available via the docker images --tree command:

$ docker images --tree
Warning: '--tree' is deprecated, it will be removed soon. See usage.
├─511136ea3c5a Virtual Size: 0 B
│ ├─5bc37dc2dfba Virtual Size: 192.5 MB
│ │ └─61cb619d86bc Virtual Size: 192.7 MB
│ │   └─3f45ca85fedc Virtual Size: 192.7 MB
│ │     └─78e82ee876a2 Virtual Size: 192.7 MB
│ │       └─dc07507cef42 Virtual Size: 192.7 MB
│ │         └─86ce37374f40 Virtual Size: 192.7 MB
│ │           └─d76983dc2ebd Virtual Size: 213.3 MB
│ │             └─04a01662a6a8 Virtual Size: 214.5 MB
│ │               └─7769c00dfefe Virtual Size: 525.9 MB
│ │                 └─6ac8d6e449b1 Virtual Size: 525.9 MB
│ │                   └─e3a84ca24205 Virtual Size: 525.9 MB
│ │                     └─26f10d07659d Virtual Size: 525.9 MB
│ ├─e12c576ad8a1 Virtual Size: 198.9 MB
│ │ └─102eb2a101b8 Virtual Size: 199.1 MB
│ │   └─530dbbae98a0 Virtual Size: 199.1 MB
│ │     └─37dde56c3a42 Virtual Size: 199.1 MB
│ │       └─8f118367086c Virtual Size: 228.5 MB
│ │         └─277eb4304907 Virtual Size: 228.5 MB Tags: ubuntu:utopic, ubuntu:14.10

However, the Docker team is trying to streamline its client, and has scheduled this feature for deprecation. How, then, do we replicate its behavior?

Enter DockerViz. Grab a binary from and place it somewhere on your path.

Querying the Docker image status

The Docker server can be queried via its public API. It is typically available either on http://localhost:4243 or /var/run/docker.sock.

One of the following two calls should therefore extract the image status:

curl -s http://localhost:4243/images/json?all=1
echo -e "GET /images/json?all=1 HTTP/1.0\r\n" | nc -U /var/run/docker.sock

On my machine, the second query returns:

HTTP/1.0 200 OK
Content-Type: application/json
Date: Sun, 18 Jan 2015 17:41:34 GMT


DockViz parses this JSON and outputs a formatted tree:

$ cat ~/scripts/docktree 
echo -e "GET /images/json?all=1 HTTP/1.0\r\n" | nc -U /var/run/docker.sock | tail -n +5 | dockviz images --tree
$ docktree
├─511136ea3c5a Virtual Size: 0.0 B
│ ├─5bc37dc2dfba Virtual Size: 192.5 MB
│ │ └─61cb619d86bc Virtual Size: 192.7 MB
│ │   └─3f45ca85fedc Virtual Size: 192.7 MB
│ │     └─78e82ee876a2 Virtual Size: 192.7 MB
│ │       └─dc07507cef42 Virtual Size: 192.7 MB
│ │         └─86ce37374f40 Virtual Size: 192.7 MB
│ │           └─d76983dc2ebd Virtual Size: 213.3 MB
│ │             └─04a01662a6a8 Virtual Size: 214.5 MB
│ │               └─7769c00dfefe Virtual Size: 525.9 MB
│ │                 └─6ac8d6e449b1 Virtual Size: 525.9 MB
│ │                   └─e3a84ca24205 Virtual Size: 525.9 MB
│ │                     └─26f10d07659d Virtual Size: 525.9 MB
│ ├─e12c576ad8a1 Virtual Size: 198.9 MB
│ │ └─102eb2a101b8 Virtual Size: 199.1 MB
│ │   └─530dbbae98a0 Virtual Size: 199.1 MB
│ │     └─37dde56c3a42 Virtual Size: 199.1 MB
│ │       └─8f118367086c Virtual Size: 228.5 MB
│ │         └─277eb4304907 Virtual Size: 228.5 MB Tags: ubuntu:14.10, ubuntu:utopic

Note that, on my system, the first branch of the tree is dangling, i.e. not associated with a tagged image–I must have removed a tagged image earlier, and these are its remaining dependencies.

Pruning unusued images

Built and downloaded Docker images quickly gobble up a lot of space:

$ sudo du -hcs /var/lib/docker/
10G	/var/lib/docker/
10G	total

The docker images command allows us to list dangling images:

docker images --filter dangling=true --quiet

And we obtain a list of containers (images that were fired up and modified) using:

docker ps -aq

I remove both of these with the following script (WARNING: This will delete ALL containers and any unused, downloaded images, so use with caution!):

CONTAINERS=$(docker ps -aq)
IMAGES=$(docker images --filter dangling=true --quiet)
if [[ $CONTAINERS ]]; then
    docker rm $CONTAINERS
    echo "No containers to remove"

if [[ $IMAGES ]]; then
    docker rmi $IMAGES
    echo "No dangling images to remove"


$ docker-clean
$ sudo du -hcs /var/lib/docker/
6.6G	/var/lib/docker/
6.6G	total
$ docktree
└─511136ea3c5a Virtual Size: 0.0 B
  ├─e12c576ad8a1 Virtual Size: 198.9 MB
  │ └─102eb2a101b8 Virtual Size: 199.1 MB
  │   └─530dbbae98a0 Virtual Size: 199.1 MB
  │     └─37dde56c3a42 Virtual Size: 199.1 MB
  │       └─8f118367086c Virtual Size: 228.5 MB
  ├─d497ad3926c8 Virtual Size: 192.5 MB
  │ └─ccb62158e970 Virtual Size: 192.7 MB
  │   └─e791be0477f2 Virtual Size: 192.7 MB

Note that, now, all branches of the tree are associated with tagged images. If I remove ubuntu:utopic, I can again run the pruning process to get rid of its left-over dependencies.

<![CDATA[SciPy2013 Proceedings]]> 2013-06-27T00:00:00+00:00 0001-01-01T00:00:00+00:00 Where I announce the SciPy 2013 proceedings, and talk about how we did it.

The talks for the 12th (fantastic) Python in Science conference just concluded, and I am happy to announce the conference proceedings.

This may come as a surprize to some, since in the past we have been unable to publish the proceedings in a timely manner. So, what changed?

Review process

For 2013 we followed a very light-weight review process, via comments on GitHub pull-requests. This change has an important consequence: in contrast to the traditional review process, where reviewers critically pull apart papers, the process now changes into a constructive conversation–the reviewer becomes an ally to the author, helping them to get their paper signed off on.

In addition, this is a very familiar process to most members of our community who regularly collaborate to open source projects. Most such projects nowadays follow a similar methodology for discussing and integrating patches.


Since we can’t expect reviewers to check out and build the papers themselves, a paper build bot is provided to generate PDFs from pull-requests, which contain papers in plain-text ReStructuredText format (see the proceedings repository for examples, and all papers starting 2010).

For authors, tools are provided to convert the ReStructuredText papers to PDFs in IEEE Computer Society paper style.

Help us with the final review

We welcome your feedback on the proceedings! If you spot a mistake, please submit a pull request on GitHub.


Finally, a big shout-out to the amazing team of people who organized this year’s conference, and to the wonderfully inclusive and talented Scientific Python community, of which I am proud to be part of.

<![CDATA[Replicable super-resolution paper]]> 2012-10-15T00:00:00+00:00 0001-01-01T00:00:00+00:00 Where I summarize 5 years of my life in 10 pages.

While I loved the topic of my PhD, I had to take a break after staring at that problem for several years. Thereafter, I was side-tracked by several other projects, and never got around to publishing a paper on my dissertation.

Here, then, is a summary of the simple but effective super-resolution algorithm described therein:

I also submitted this work to NIPS: the reviewers liked the paper, but they were not convinced of its novelty. Having spent a lot of time studying the existing literature, all I can say in response is that, while solving the problem as a sparse linear system was well known at the time, phrasing Drizzle as a linear operator and using it for super-resolution was not.

But the proof of the pudding is in the eating! Have a look at the results and published code – you can download it all (including a sample data-set) and play with the different reconstruction parameters. Quite a bit of the code has since graduated into scikit-image.

<![CDATA[scikits-image 0.7]]> 2012-09-30T00:00:00+00:00 0001-01-01T00:00:00+00:00 We’re happy to announce the 7th version of scikits-image!

Scikits-image is an image processing toolbox for SciPy that includes algorithms for segmentation, geometric transformations, color space manipulation, analysis, filtering, morphology, feature detection, and more.

For more information, examples, and documentation, please visit our website.

New Features

It’s been only 3 months since scikits-image 0.6 was released, but in that short time, we’ve managed to add plenty of new features and enhancements, including

  • Geometric image transforms
  • 3 new image segmentation routines (Felsenzwalb, Quickshift, SLIC)
  • Local binary patterns for texture characterization
  • Morphological reconstruction
  • Polygon approximation
  • CIE Lab color space conversion
  • Image pyramids
  • Multispectral support in random walker segmentation
  • Slicing, concatenation, and natural sorting of image collections
  • Perimeter and coordinates measurements in regionprops
  • An extensible image viewer based on Qt and Matplotlib, with plugins for edge detection, line-profiling, and viewing image collections

Plus, this release adds a number of bug fixes, new examples, and performance enhancements.

Comparison of pupular segmentation algorithms

Contributors to this release

This release was only possible due to the efforts of many contributors, both new and old.

  • Andreas Mueller
  • Andreas Wuerl
  • Andy Wilson
  • Brian Holt
  • Christoph Gohlke
  • Dharhas Pothina
  • Emmanuelle Gouillart
  • Guillaume Gay
  • Josh Warner
  • James Bergstra
  • Johannes Schönberger
  • Jonathan J. Helmus
  • Juan Nunez-Iglesias
  • Leon Tietz
  • Marianne Corvellec
  • Matt McCormick
  • Neil Yager
  • Nicolas Pinto
  • Nicolas Poilvert
  • Pavel Campr
  • Petter Strandmark
  • Stéfan van der Walt
  • Tim Sheerman-Chase
  • Tomas Kazmar
  • Tony S Yu
  • Wei Li
<![CDATA[Adobe's new free font: Source Code Pro]]> 2012-09-26T00:00:00+00:00 0001-01-01T00:00:00+00:00 Adobe yesterday released its free and open source Type family, Source Code Pro, which includes an eye-pleasing monospace font ideally suited for coding. In the past, and at the recommendation of Fernando Perez, I’ve used the beautiful (but non-free) Consolas by Microsoft; now, which is best?

To install on Linux:

  1. Grab the font.
  2. Copy the files to ~/.fonts.
  3. Run fc-cache -f -v.

The font should now be available for selection in apps such as Firefox, Gnome Terminal, etc. To make it the default font in Emacs::

    (set-default-font "Source Code Pro")

Here’s a comparison of Consolas (left) and Source Code Pro (right):

Comparison: Consolas vs Source Code Pro

Comments also on Google+.

<![CDATA[Emacs package management]]> 2012-09-25T00:00:00+00:00 0001-01-01T00:00:00+00:00 I recently tried to install MuMaMo as one of the dependencies for Takafumi Arakaki’s Emacs-based IPython notebook. The instructions on the MuMaMo webpage were as clear as mud and aimed primarily at Windows users. Enters apt-get for Emacs!

My Emacs setup is shared across multiple machines: a synchronized elisp folder, containing *.el files, along with my .emacs configuration. el-get allows you to share your package installation folder in a similar fashion. Here are some relevant configuration snippets:

; Everything gets installed into ~/elisp, a folder
; I sync across all my machines

(setq el-get-dir "~/elisp/el-get")
(setq el-get-install-dir "~/elisp/el-get/el-get")
(add-to-list 'load-path el-get-install-dir)

; If el-get is missing, install it automatically

(unless (require 'el-get nil t)
   (lambda (s)
     (goto-char (point-max))

; Install these packages, and call the specified configuration snippets
; after each load
(setq el-get-sources

        (:name ethan-wspace
         :after (progn
                  (global-ethan-wspace-mode 1)
                  (set-face-background 'ethan-wspace-face "gray95")))

        (:name column-marker
         :after (add-hook 'font-lock-mode-hook
                          (lambda () (interactive) (column-marker-1 80))))

; Also install these packages, no configuration required
(setq my-packages
       '(el-get maxframe markdown-mode ein python)

; Check packages and install any that are missing
(el-get 'sync my-packages)

There are two ways to specify packages to be installed: either include them in the my-packages list, or add them to el-get-sources, which in addition allows further customization upon successful loading of the package.

What’s in your stack? Here’s my list of Emacs packages:

Org Mode, Ethan's wspace, Tab Bar, Column Marker, Max Frame, EIN, Python,

Do you know of any other useful packages? Let me know!