Galton’s Crowd forecasts #BTW2017 outcome better than Pollsters

Some time on Saturday September 23 2017 put up an elections prediction market which closed on Sunday September 24 2017 at 14:00 hrs, four hours before the voting stopped. A total of 4,144 people provided their guesses. How did the crowd do? Much better than the pollsters.

Before we go on to the results let me remind you of the Diversity Prediction Theorem,  a very easy to verify fact. If N individuals are asked to guess a true measurement T and s_i: i=1,\ldots,N are their single guesses and c is the mean of their guesses (also called the crowd guess) then the theorem states:

(c-T)^2=\frac{1}{n} \displaystyle{\sum_{i=1}^{N}} (s_i-T)^2 - \frac{1}{n} \displaystyle{\sum_{i=1}^{N}} (s_i-c)^2

which simply reads that the square error of the crowd’s guess equals the mean squared individual error minus the variance of the individual errors. Which means that the crowd does better than any of its members as variance increases.

This works well when the crowd members are uncorrelated and fails miserably when not as manias, information cascades and bubbles show.

On a party by party basis we can thus compute the mean individual squared error vs the crowd squared error and we can do so for the crowd of 4,144 laymen as well as for the crowd of the 13 pollsters whose last polls can be found here together with the guesses of the 4,144 lay people.

In each cell the left hand side number is the mean individual (polling firm or layperson) squared error and and on the right you see the crowd’s (of firms or person) squared error. Clearly with the exception of Die Grünen and Die Linke the crowd guessed much better than the pollsters.

One can legitimately ask whether the pollsters should ask people to predict the elections instead of (or in addition to) asking them to reveal their preference. The former is immune to strategic response whereas the latter is not. Greetings from Galton.

Caveats: The laymen were still guessing on Sunday perhaops giving them an advantage over pollsters. The crowd of the pollsters is 13 strong whereas the laymen are 4,144 also an advantage for the laypeople. On the other hand the pollsters have the entire statistical toolkit at their disposal and a larger sample than each of the laymen.

For the convenience of Stata users here are the datasets:


This entry was posted in Big Data, data, markets, nowcasting, science and tagged , . Bookmark the permalink.