

hide cookie message

What is Big Data? Understanding Big Data, and how it affects us all

We create 2.5 quintillion bytes of data every day

Big Data

We explain the term Big Data: what it is, and why it matters to everyone. (See also: Big data demystified.)

The 1992 film Sneakers features a scene where Sir Ben Kingsley and Robert Redford's characters - both activist hackers whose lives have taken divergent paths - discuss the idea of power.

'It's not about who's got the most bullets,' Kingsley states, 'It's about who controls the information. What we see and hear, how we work, what we think... it's all about the information!'

Over twenty years later those words are increasingly becoming a reality, but the real power isn't merely the acquisition of information, instead it resides in being able to find the patterns hidden in its midst. Trend analysts have of course been doing this very thing for years, but due to  our ever expanding use of the internet for all aspects of modern life there is now so much data being created that it requires a new approach - this is the world of 'Big Data'.

IBM recently stated that people create a staggering 2.5 quintillion bytes of data every day (that's roughly equivalent to over half a billion HD movie downloads). This information is generated from a huge variety of sources including social media posts, digital pictures, videos, retail transactions, and even the GPS tracking functions of mobile phones. In fact IBM go on to say that '90 percent of the data created in the world today has been created in the last two years.' (Also see: Big goals for Big Data.)

Harnessing Big Data


'Big data marks the beginning of a major transformation,' explain authors Kenneth Cukier and Viktor Mayer-Schonberger in their book Big Data. 'Just as the telescope enabled us to comprehend the universe and the microscope allowed us to understand germs, the new techniques for collecting and analyzing huge bodies of data will help us make sense of our world in ways we are just starting to appreciate.'



'We all like to think that we're in control of everything,' said Brantingham, 'but in fact all of our behaviour is very regular, very patterned in ways that is often frightening to us. Offenders are no different, they do exactly the same things over and over and over again, and their criminal offending patterns emerge right out of that regularity of their behaviour.'


The Human Genome project, which set out to decode the genetic building blocks of life, took a team of dedicated scientists a decade of intensive work to complete. The same task now, using the new technologies behind big data, would take as little as a day. Astronomers are able to map the heavens with much higher powered telescopes than ever before and collate the terabytes of data that they produce in a fraction of the time. Particle physicists working on the Large Hadron Collider were able to record the behaviour of subatomic particles travelling at unbelievable speeds and process that data to potentially discover the Higgs Boson (or 'God') particle, which could change the future of science in a profound way. (Read our article: Analytics key to Big Data.)

Big Data: life changing

All of these advances in data collection and manipulation are exciting and possibly life-changing. None more so than the recently announced 'Brain Research through Advancing Innovative Nanotechnologies' programme (or BRAIN for short), whose goal is to map the activity of the human brain. Using state of the art equipment the scientists will be attempting to monitor, record and analyse millions, even billions, of neurons simultaneously. The programme, which was revealed by President Obama in April, hopes to find cures to conditions such as Alzheimer's, strokes, and autism, with additional research into other areas of mental illness.

There are those who argue that the nature of big data means discoveries can be made without the scientists truly understanding the origin or motive of the problem - so they'll know how something happens, but not necessarily why. This argument opens up the philosophical nature of research, but some big data proponents respond in more pragmatic terms.

'As humans we have been conditioned to look for causes,' Kenneth Cukier and Viktor Mayer-Schonberger explain, 'even though searching for causality is often difficult and may lead us down the wrong paths. In a big-data world, by contrast, we won't have to be fixated on causality; instead we can discover patterns and correlations in the data that offer us novel and invaluable insights. The correlations may not tell us precisely why something is happening, but they alert us that it is happening. And in many situations that is good enough.'

Read more: Big Data Protects Intel's Info.

Image courtesy of jurvetson.

IDG UK Sites


IDG UK Sites


IDG UK Sites


IDG UK Sites
