“One Secret Trick My Kidnappers Don't Want Me to Know”


Because you watched...

Monday, 12 September 2016 (#)

Netflix suggests that slow television naturally leads to Star Trek.

Funny if curated; funnier if actually derived from data.

(Music: R.E.M., “Ignoreland”)

Fractional nines

Monday, 5 September 2016 (#)

Conversationally, you can describe system availability in terms of nines. A two nines (available 99% of the time) system would be built differently from a five nines system (99.999%).

Outside the realm of high availability, what could we call a system available only 80% of the time? It’s less than 90% (“one nine”), so it would strictly be zero nines. To be more specific, we’ll need to introduce fractional nines. Having done that, we could refer to it as 0.7 nines.

Let’s go through the math.

Natural nines

Start with the natural numbers. We have f:ℕ↦ℝ.

NinesAvailability (%)

Simplify by considering availability as a fraction (in [0,1]) rather than a percentage:


Note that this is exactly

availability ( n ) = 1 - 0.1 n

In addition to the natural numbers, this function is defined for all reals. To plot using R:

avail <- function(n) { return(data.frame(n, a = 1 - 0.1 ^ n))}

d_nat <- avail(1:7)
d_real <- avail(seq(0, 7, 0.1))

svg(file = 'output.svg', width = 7, height = 5)

ggplot() +
  geom_line(data = d_real, aes(x = n, y = a), color = 'red') +
  geom_point(data = d_nat, aes(x = n, y = a)) +
  coord_cartesian(ylim = c(0.75, 1), xlim = c(0, 6)) +
  xlab('Nines') + ylab('Availability') +


Nines to availability

Here are some examples not restricted to natural nines.


Since the function is invertible, consider:

nines ( a ) = - log 10 ( 1 - a )

So 80% availability is 0.69897, or approximately 0.7, “nines”. Likely nothing to boast about, but quantification is the first step.

(The 0.09 is included to foil attempts to consider a 9% available system as “one nine”: it’s 0.04 nines. However, extreme chutzpah may be appropriate in situations where a system is available less than ten percent of the time.)

(Music: Alkaline Trio, “We’ve Had Enough”)

Using HTTP caching libraries

Sunday, 6 December 2015 (#)

Efficient use of HTTP used to require a lot of custom code. However, libraries can dramatically reduce the amount of code you’re maintaining, and also allow sharing any improvements.

Efficient use of HTTP/1.1 means caching (RFC 7234), conditional requests (RFC 7232) and compressed Content-Encoding, amongst other things. For an RSS reader, I previously wrote a bulk feed downloader that took advantage of all these things in around 700 lines of perl.

A large part of that logic is included in httplib2, a great Python library that acts as an in-process caching proxy. To allow use of Python’s de facto standard Requests library, I’ve been using cachecontrol: “The httplib2 caching algorithms packaged up for use with requests.”.

Instead of:

    sess = requests.session()
    response = sess.get('http://example.com/')


    sess = CacheControl(requests.session(), cache = FileCache('.web_cache'))

Hey presto; all your HTTP calls backed by a persisted cache that obeys HTTP’s rules for cache freshness. On the Java side, Apache HttpComponents does a pretty good job of the same thing.

What’s the result? Firstly; I’m down to under 400 lines (now, of Python). Secondly, and more importantly, much of my code is now in a common open source library. I can benefit from others’ fixes, and contribute my own as well.

The switch from Perl threads to Python’s concurrent.futures is also welcome.

I’ve lost some functionality from my own code. I no longer have a summary of how much bandwidth was saved due to compression at the end. However, I don’t miss it, and this kind of rewrite is a great chance to throw away behaviour that I’m not actually using.

The moral of the story is: treat self-maintained code as a liability to be reduced where possible. Unless there’s good reason, prefer de facto standard libraries, and architectures that allow small, well-defined libraries to be introduced.

(Music: Motörhead, “R.A.M.O.N.E.S.”)
Joseph Walton <joe@kafsemo.org>