Why did (almost) everyone call the election wrong (again)? - BBC News
Newsnight's editor assesses why so many people underestimated Labour's vote at the general election.
This video can not be played To play this video you need to enable JavaScript in your browser. Why do pollsters - and the media - keep getting elections so wrong? Ian Katz reports

Few species can match the brutality of a teenage child appraising its parent.

I was reminded of this the morning after last month's election as I passed my 18-year-old daughter on the stairs. "I'm never going to believe another word you say about politics," she announced matter-of-factly. "Because you've been wrong about EVERYTHING."

It was hard to argue. The 2015 election, Brexit, Trump, and now Corbyn's sort of moral victory… I'd called them all wrong. I was, as they say in American sport, "Oh for four". The only comfort was: most of the media and political world were, too.

Over the last month, I've been reflecting on why we keep getting surprised, for a Newsnight film. Has the political landscape changed in some profound way we have not yet got our heads around? Or have we simply been through a period of freak political weather?

Not impressed... Ian Katz's daughter is sceptical now of pundits and pollsters

And, more immediately, how did most of the media, the pollsters and even much of the Left underestimate Labour's vote so badly?

In the spirit of group therapy, I thought I'd start with someone who was even wronger than me. Martin Boon has long been one of Britain's most respected pollsters. This time his company, ICM, got it quite spectacularly wrong; their eve-of-election poll gave the Tories a 12-point lead, a full 10 points bigger than the actual result.

I found him in contemplative, even penitent, mood. In 2015, ICM got it wrong by overestimating the Labour vote. This time, they tried to address the problem by making sceptical assumptions about how many younger voters (among other groups) would turn out - and ended up massively underestimating Labour's vote.

"We were bamboozled by the turnout which we predicted wouldn't happen in the way it did," he said. "And I have to hold up my hands and say that…

"The problem for me is that the techniques which didn't work in 2015 did work in 2017, and indeed the techniques which the likes of me applied in 2017 wouldn't have worked retrospectively in 2015."

The result of the election was a shock to many

With a degree of humility not often encountered in either politics or the media, he said pollsters had to think hard about whether "classical orthodox polling techniques" were still worth persevering with.

One source of comfort to pollsters and journalists mulling over why they didn't see last month's result coming is the fact that most politicians didn't either. A source told me the Labour Party's internal predictions, minutes before the exit poll was released, were for a Tory majority of around 60 seats.

Labour MP Jess Phillips said she and other MPs simply weren't hearing anything on the ground to make them doubt the widely shared belief that they were heading for a drubbing.

This video can not be played To play this video you need to enable JavaScript in your browser. Tony Blair: "There's been so many political upsets, it's possible that Jeremy Corbyn could become prime minister"

"What we potentially missed in classic campaigning and classic polling is the people we're not talking to, and still I'm driving round my constituency thinking, 'Did you vote for me? Did you vote for me?' We just weren't talking to the right people."

One man not willing to don sackcloth and ashes just yet is ITV's political editor Robert Peston, who was more upbeat than many in the media about Jeremy Corbyn's prospects: right up to polling day when, he says, he was persuaded by senior politicians on both sides that his instincts were mistaken.

Like many of us, Mr Peston confessed he was still trying to find his bearings in a world where many of the things we thought were true no longer seem to apply. "The old rules have gone and we've got to try and make sense of how politics works. And the truthful answer is we're all feeling our way a bit."

So what about the man who, perhaps more than anyone, can claim to have divined the rules of modern politics? Even Tony Blair, a man not famous for self-doubt, says the events of the last two years have made him rethink some of his assumptions about politics.

Robert Peston: "The old rules have gone"

"For most of my political life I've been saying, 'I think this is the right way to go, and what's more it's the only way to win an election.' I have to qualify that now. I have to say, 'No, I think it's possible you end up with Jeremy Corbyn as prime minister.'

"I personally think it's a surer route to power to fight it from the centre but I'm being open with you in saying that I accept now what if you'd asked me a year ago I'd have said is impossible."

Given that there's a fair chance we'll be grappling with another UK election in months rather than years, how can we do better at reading it than we have done on recent votes?

A good person to ask seemed to be one of the few commentators who called the 2017 election almost exactly right, Rod Liddle of the Spectator and the Sunday Times.

Liddle's prescription: "Get out of town, get out of London. Unless the polls change the way they are being done ignore them. And don't follow the herd."

Most pollsters and pundits underestimated how well Jeremy Corbyn would do in the election

Of course, the BBC and other media organisations did have lots of on the ground reporting from across the country during the election and some of it did suggest that Mr Corbyn was doing better than most pollsters and pundits thought. But there's a tendency to tune out evidence that doesn't fit the prevailing narrative.

One person who never doubted that Mr Corbyn would surprise his detractors is Matt Turner, a (just) 22-year-old who, while not doing his finals last month, was helping to edit Evolve Politics, one of a clutch of pro-Corbyn websites which claimed to have their finger closer to the national pulse than traditional media.

Although there is never a shortage of seers claiming to be wise after any surprise event, Turner has the betting slip to prove it: he put money on a hung parliament at 10-1 back in April.

"Sites like ours had our ear to the ground and we gave a more accurate reflection of what people were actually feeling. People have accused us of living in a bubble when we've accurately predicted the hung parliament. If anything it's now the Westminster media who are living in that bubble."

The one common thread among all those I talked to was an acknowledgement that social media - simultaneously mobilising, and polarising - has clearly changed the way millions of people experience politics. And we haven't yet worked out how to take the pulse of an election played out in 50 million timelines.

Figuring out how to do that may be the most urgent challenge facing all of us whose job it is to read the political runes. For the foreseeable future, though, you'd be best advised to ignore all political predictions. And I, my daughter at least will be pleased to know, won't be making any.

Ian Katz is editor of BBC Newsnight - watch his full report here