Explainer: what is metadata? Should I worry about mandatory data retention?

What can metadata reveal about us? How easy is it to bypass metadata collection? And shouldn’t we be keeping an eye on terrorist websites?

‘Metadata retention is an absolute farce’. Photograph: Martial Trezzini/AP

With all of the talk about “metadata” data retention schemes surrounding yesterday’s announcement by the Australian Coalition, we need a straightforward and easy way to understand what metadata is. Here’s my attempt:


Data is pretty straightforward, so let’s look at it in the scope of a web request. Say you’re logging into Facebook and you look at a picture that a friend has updated. The content of that picture is the data; the digital contents of the file that mean the file can be displayed on any computer which supports decoding that file is the data itself.


Metadata is the data that is generated when you view data, or when you access data.

When you visited that Facebook page, you probably hit a URL. The metadata created was that you hit that URL, at that time, who queried that URL (meaning your IP address, which your ISP can easily correlate with who you physically are), as well as any other information, such as the size of the request (which could be used to intuit the kind of information transferred).

So with a basic Facebook request, we’ve now got:

  • The time of day the request was made
  • Who made the request (your IP address)
  • How big the request was
  • Any other resources that were related to that request (such as files that tell your browser how to display the page you’re looking at)

This is a basic example of the kinds of metadata that can be generated.

What the collected metadata let us discover

Let’s jump straight to a more ribald example. Let’s say you want to watch some pornography. You browse to your favourite website, spend some time there, and then decide to watch some funny cat videos. With metadata, which provide you URLs, any individual would be able to figure out how long you spent watching XXX-rated content. The metadata also gives whoever is reading the log (be they an ASIO officer, RSPCA officer, someone from the local council or even a hacker) information about your go-to whoa time, and the kind of films you’re into.

Okay, so how can the fact that I look at porn be used to fight terrorism?

First off, it’s a furphy to say that data retention can fight terrorism. Data retention would provide a mechanism for security services or police to access your browsing history and use it to prove a case against you for a particular crime you are accused of. Without data retention there are still, surprisingly, people who are accused of and convicted of crimes and who are dealt with accordingly, by the justice system.

People who have been planning (or attempting to plan) terrorist attacks on Australian soil have been detected and caught through old fashioned policework. This data retention information provides another source of evidence for police to build a tighter case. Data retention in and of itself does nothing to “fight” terrorism.

But wait a minute – if they have all that data, won’t they be able to see what terrorists do and get them before they commit a crime?

That’s the rolled-gold claim of the software vendors and the security hawks who peddle these pieces of software. Through a few complex algorithms and some number crunching, we can detect criminal intent before it actually happens. Technology to correlate between actions and intent has been around for a while, and if you’ve been weirded out when Google ads track you around the internet and show you ads for things you’ve searched for, you’ve seen this technology in action. When you’ve then been doubly confused that Google is showing you an ad for something completely irrelevant to you, you’ve found something even more interesting: a misfiring of the algorithm or an “overfit”.

This is where stuff gets very interesting and also very, very scary. The government (or whatever other contractor manages and performs dredging on this vast dataset) could set certain red flags or websites which when visited would trigger something to happen. That trigger could be to log more information about the request, it could be to look for more requests like it, it could be to add flags to a person’s file.

So what, shouldn’t we be keeping an eye on terrorist websites?

It depends on what you’re trying to do. If you’re trying to catch people who are either researching terrorism or documenting terrorist websites, then sure. If you’re trying to combat terror, this mechanism is only going to work once. Terrorists are engaged in what is known as “asymmetric warfare”, they don’t play by the existing rules which is why it’s particularly difficult to combat them and sniff them out.

The war on terror has also shown they’re incredibly capable of adapting to whatever we throw at them. If data retention was purely about combating terror, then we wouldn’t have had the broader slip by Tony Abbott saying that retained data would also be used for other “law enforcement purposes”. These “law enforcement purposes” are already incredibly broad, and not constrained. As this Telegraph article shows, more than half of UK councils are using extraordinary powers under anti-terror laws to spy on people who … don’t use their bins correctly.

Okay, but that’s an extreme and ridiculous example. With oversight, surely this stuff can be used correctly?

Systems aren’t perfect and people certainly aren’t perfect. As someone who has worked with software systems in secure settings for years, stuff slips through the gaps. But Abbott himself said it would be used for other forms of criminal investigations, so there are going to be multiple points of access (or multiple tiers of access) to the system. With those multiple tiers come multiple points of failure that could be abused, leading to massive privacy breaches.

But what about measures needed to keep our country safe?

As of the 6 of August, 2014, there’s only ever been one fatal attack classified as a “terrorist attack” on Australian soil. That was the Hilton hotel bombings. That attack took place in 1978.

As addressed earlier, data retention will do nothing to keep Australians safe, and in many instances the false positives created by systems scanning for “behaviour patterns” will waste the time and energy of security services, and potentially allow people to slip through the cracks.

There are obviously two broader concerns here. The first is: do we want to live in a society where such mass warrantless surveillance of our citizenry is a mundane fact of everyday life? Do we trust the government (or really, whomever they contract in to handle this) to securely manage our browsing histories and make sure they’re only ever used ethically and when absolutely justified?

We already have a system with stringent checks and balances that works to protect privacy and make sure data is being used legitimately, they’re called warrants – and the police, Asio, Asis and others use them every single day.

Again, data retention is not about fighting terror, data retention is about something much bigger. Data retention is step one towards substantial internet control by the government. With legislation mandating that ISPs put in place infrastructure to snoop on and retain internet traffic, who’s to say agencies won’t soon hand over the data of those engaging in alleged piracy to copyright agencies in order to institute a three strikes system?

The data retention play isn’t about data retention, it’s about the government wresting a greater degree of control over how we use the internet, and putting into place a system of mass surveillance that will almost certainly be misused to the detriment of many Australian citizens.

How easy is it to bypass metadata collection under a data retention regime?

It’ll take you less than five minutes. Go to a website like easyvpn or strongvpn, sign up for a VPN service, and follow their simple instructions to route all of your traffic across an encrypted channel that can’t be snooped on. The metadata the government will see if you put all of your traffic down an encrypted pipe will amount to the size of what you’re transferring (maybe, it depends on what kind of inspection they do) and that you’re connecting to a VPN.

The best thing about those VPN services? They don’t keep logs, so there’s nothing to hand over. This point in and of itself completely explains why data retention is an absolute farce, and is in no way a deterrent to terrorism.

A version of this explainer was first published on Dan Nolan’s Tumblr page