Big data spells death-knell for punditry

Nate Silver's successful US election predictions show the merit of data analysis and why businesses should embrace it
US election
Nate Silver's election predictions have lessons for business, says Dan Vos

The results of the US presidential election are now in. Barack Obama has been re-elected, but the election has also officially ushered in a new era. An era where analysis has displaced opinions; where data scientists have pushed out the so-called experts. These results provide excellent insight that progressive organisation should take note. Data must be the key driving factor in your business.

Prior to the election, there were countless heated debates against data scientists in the political domain. The poster child of the election data scientist is Nate Silver from the New York Times' blogsite Five Thirty Eight. Unfortunately, for the pundits who decried Silver's pragmatic approach, he could not have been much more accurate. His predictions on the US presidential race were accurate. Silver predicted every state correctly; 50 out of 50; 100% right. His successful analysis doesn't stop there, Silver continued to be spot-on in his predictions for nearly every US Senate contest. Silver's success should also have a profound impact on how businesses leverage data analysis.

It provides clear evidence that the business world needs to rely on measured performance and projection over opinion and spur-of-the-moment inspirations. Organisations should reject the so-called expert's opinion, that is based on instinct and outdated metrics, as a viable path to set the direction for decision making. This new era, based on more complex data models, has the potential to permeate into many areas of life.

Of course, this data driven approach is being fuelled by advancements in technology. Our datasets have expanded faster than our ability to count them, processing capability has exploded through the use of the cloud and outlets to display data have multiplied to include tablets, phones, and many more platforms. We are at the prefect intersection of tools and capability to deliver upon this possibility. We have the processing power and capability to deliver on it here and now.

There is another driving force that is just starting to gain momentum in the business space. Organisations are becoming aware that seemingly unpredictable events can be broken down and presented in a meaningful ways to project outcomes and therefore impact decisions. As this concept gains more momentum, businesses of all types will need to find new ways to embrace predictive analytics. Organisations will need the tools and talents to identify patterns of behaviour that, left unnoticed, could prove to be fatal to the health of the company. The stage is set for future success: businesses that know v businesses that guess.

Businesses, like elections, have for too long relied on an expert's voice to guide decisions and predict how the winds would shift in their favour. There was an allowance for expert driven decisions because every business operated this way. That meant every business could tolerate a certain number of poor decisions and continue to survive. But as big data takes hold and decisions are predicted based on information, the number of poor choices a business can make starts to drop. The analysis and subsequent decisions that used to be acceptable is going to lead to increasingly poorer performance relative to their competitors. Businesses that guess cannot survive in this analytical marketplace.

Pundits made a huge mistake in this election. So where did they go wrong? They went after the pollsters, the analysts and the people that lay out quantitative projections based on models, not opinions. However, the pundits can't escape their accusations this time because the models were undeniably correct. This same pattern is going to emerge in the organisation where the established experts will hotly contest new ideas that go against their intuition. Opinions will vary but at the end of the day, there will be a clear winner, the people that understand the data.

We need to expand the world of analysis to drive our decision making by the insights found in statistical models and the predictions of likelihood of success.

Dan Vos is a principal of Palador, a consultancy that focuses on delivering real-time solutions. He can be followed at @PaladorDan.


  • Media Network Activate London

  • Activate London find out more


About us

  • Guardian Professional Networks
