Intelligent Systems Conference 2017

7-8 September 2017 | London, UK

Amplifying Prediction Accuracy using Swarm A.I.

Louis Rosenberg Niccolo Pescetelli
Unanimous A.I. University of Oxford
San Francisco, CA, USA Clarendon, UK
[email protected] [email protected]

Abstract—In the natural world, many species amplify the week period was compared to the predictions made by the
accuracy of their decision-making abilities by working together Sports Analytics Machine (SAM), a super-computer built by
real-time closed-loop systems that converge on optimal solutions the University of Salford to predict English Premier League
in synchrony. Known as Swarm Intelligence (SI), the process has games using rigorous mathematical models [2]. Because SAM
been deeply studied in schools of fish, flocks of birds, and swarms results are published weekly by the BBC to reflect an “expert”
of bees. The present study looks at the ability of human groups to assessment of weekly matches, this allowed for comparison of
make decisions as an Artificial Swarm Intelligence (ASI) by professional level predictions with novice-based human
forming similar real-time closed-loop systems online. More swarms.
specifically, the present study tasked groups of typical sports fans
with predicting English Premier League matches over a period of II. SWARMS AS INTELLIGENT SYSTEMS
five consecutive weeks by working together in real-time as
swarm-based systems. Results showed that individuals, who The decision-making processes in honeybee swarms have
averaged 55% accuracy when predicting games alone, were able been observed to be remarkably similar to the decision-making
to amplify their accuracy to 72% when predicting together as processes in neurological brains [3], [4]. Both employ large
real-time swarms. This corresponds to 131% amplification in populations of simple excitable units (i.e., bees and neurons)
predictive accuracy across five consecutive weeks (50 games). that work in parallel to integrate noisy evidence, weigh
competing alternatives, and converge on decisions in
Keywords—Swarm Intelligence; Artificial Swarm Intelligence; synchrony. In both, outcomes are arrived at through a real-time
collective intelligence; human swarming; artificial intelligence competition among sub-populations of excitable units. When
one sub-population exceeds a threshold level of support, the
I. INTRODUCTION corresponding alternative is chosen. In honeybees, this enables
Artificial Swarm Intelligence (ASI) strives to amplify the optimal decisions over 80% of the time [5]-[7]. It is this
collective wisdom of human groups by connecting participants amplification of intelligence that Artificial Swarm Intelligence
online into real-time closed-loop systems that are modeled aims to enable among distributed networked humans.
after biological swarms. Prior studies have shown that such
“human swarms” can produce significantly more accurate
predictions than traditional methods for tapping the collective
intelligence of groups, such as votes, polls, surveys, and
markets. For example, one recent study tested the ability of
human swarms to forecast the outcome of College Bowl
football games (in the US) against the Las Vegas spread. A
swarm was comprised of 75 amateur football fans was tasked
with predicting each of 10 college bowl games. As individuals,
the participants averaged 50% correct (i.e., coin flip accuracy).
But, when working together as a real-time swarm, those same
participants achieved 70% accuracy against the spread. Not
only is this a significant accuracy increase, it also enabled the
75 amateur football fans to out-predict the football experts at
ESPN [1].
While prior studies have documented the ability of Fig. 1. Usher-McClelland model of neurological decision-making.
Artificial Swarm Intelligence to amplify the predictive ability
of online groups in singular events, no long-term study has The similarity between neurological intelligence and swarm
been performed to assess consistency of swarm-based intelligence becomes even more apparent when comparing
predictions over time. To address this, the present study tasked decision-making models that represent each. For example, the
human swarms with predicting all of the scheduled English decision-making process in primate brains is often modeled as
Premier League (EPL) matches over a period of five weeks in mutually inhibitory leaky integrators that aggregate incoming
2016. The objective was to assess whether or not a statistically evidence from competing neural populations. A common
significant amplification of human intelligence could be framework is the Usher-McClelland model [8] represented in
measured when comparing individual prediction accuracy to Fig. 1 above. This can be directly compared to swarm-based
swarm accuracy. In addition, swarm performance over the five decision models, like the honey-bee model in Fig. 2 below. As

61 | P a g e
978-1-5090-6435-9/17/$31.00 ©2017 IEEE
 Intelligent Systems Conference 2017
7-8 September 2017 | London, UK
shown, these swarm-based decisions follow a very similar
process, aggregating incoming evidence from sub-populations
of swarm members through mutual excitation and inhibition.

Fig. 2. Mutually inhibitory decision-making model in bee swarms.

Fig. 3. A human swarm answering a question in real-time.

III. ENABLING “HUMAN SWARMS” It is important to note that participants do not simply vary
Unlike many other social species, humans have not evolved the direction of their input, but also modulate the magnitude of
the natural ability to form a closed-loop Swarm Intelligence. their input by adjusting the distance between the magnet and
That’s because we lack the subtle connections that other the puck. Because the puck is in continuous motion across the
organisms use to establish tight-knit feedback-loops among decision-space, in order to apply force users need to
members. Schooling fish detect vibrations in the water around continually move their magnet so that it stays close to the
them. Flocking birds detect motions propagating through the puck’s rim. This is significant, for it requires participants to be
group. Swarming bees use complex body vibrations called a engaged continuously during the decision process, evaluating
“Waggle Dance”. Thus to enable a real-time Artificial Swarm and re-evaluating their contribution. If they stop adjusting their
Intelligence among groups of networked humans, specialized magnet to the changing position of puck, the distance grows
technology is required to close the loop among members. and their applied force wanes. Thus, like bees vibrating their
bodies to express sentiment in a biological swarm or neurons
To address this need, an online platform called UNU was
firing activation signals to express sentiment in a neural-
developed in 2015 to allow distributed groups of users to login
network, the participants in an artificial swarm must
from anywhere around the world and participate in a closed
continuously express their changing preferences during the
loop swarming process [9]. Modeled after the decision-making
decision process or lose their influence over the collective
of honeybee swarms, the UNU platform allows groups of
outcome.
independent actors to work in parallel to 1) integrate noisy
evidence; 2) weigh competing alternatives; and 3) converge on IV. PREDICTION STUDY
final decisions in synchrony, while also allowing all
participants to perceive and react to the changing system in To assess the predictive ability of human swarms over an
real-time, thereby closing a feedback loop around the full extended period, a formal study was conducted over a five
population of participants. week period using groups of randomly selected human subjects
from a pool of individuals who self-reported being enthusiasts
As shown in Fig. 3, participants in the UNU platform of EPL football. Each weekly group consisted of 25 to 31
answer questions by collectively moving a graphical puck to participants who engaged the experiment via online access to
select among a set of alternatives. Each participant provides the UNU swarming platform. Each subject was paid $2.50 for
input by manipulating a graphical magnet with a mouse or their participation in each weekly session, which required them
touchscreen. By positioning their magnet, users impart their to make predictions for the outcome of all 10 English Premier
personal intent on the puck. The input from each user is not a League matches being played that week, first as individuals on
discrete vote, but a stream of vectors that varies freely over a standard online survey, and then as part of a real-time
time. Because the full population of users can adjust their Artificial Swarm Intelligence comprised of the full weekly
intent at every time-step (200 ms), the puck moves, not based group. In addition, the researchers compared results to the
on the input of any individual, but based on the dynamics of predictions made by SAM, a sports super-computer at the
the full system. This enables real-time physical negotiation University of Salford which uses ten years of data and
among all members, empowering the group to collectively sophisticated algorithms to predict EPL games.
explore the decision-space and converge on the most agreeable
solution in synchrony.

62 | P a g e
978-1-5090-6435-9/17/$31.00 ©2017 IEEE
 Intelligent Systems Conference 2017
7-8 September 2017 | London, UK
Across the full five week period, predictions were made for
a total of 50 games wherein the participants were required to
forecast one of three outcomes for each game: 1) Team A wins
the match; 2) Team B wins the match; or 3) the match ends in a
tie. It is worth noting that tie games occur at a rate of
approximately 25% in EPL matches, making it a significant
outcome possibility. It is also worth noting that 94% of the
swarm participants were American citizens for whom EPL is a
foreign sport covered mostly by foreign media. This context is
relevant when comparing performance of the human swarm to
the performance of the SAM super-computer, which is a UK-
based analytical system, designed specifically to predict EPL
outcomes. In other words, it allows us to test if groups of
American fans, working together as artificial swarms, can
produce comparable results to a rigorous computational model
that is used by the BBC to forecast the UK’s national sport.
In Fig. 2 below, a snapshot of a human swarm comprised of
31 participants is shown in the process of predicting a match
between Arsenal and Watford. As shown in the figure, the
swarm is given five options to choose among, enabling the
swarm to identify which of the two teams will win and whether
the winning team will prevail by a single goal (“by 1”), by 2 or
more goals (“by 2+”), or if the swarm believes the match is too
close to call. In the example shown below, a large majority of
participants have already shifted their pull towards Arsenal,
and so the puck is currently heading in that direction.

Fig. 4. A human swarm predicting an EPL match in real-time.

If the swarm converges on an answer that indicates one of Fig. 5. A time-series of swarm converging on a final prediction.
the two teams will win, that is selected as the predicted
outcome for the given match. If, on the other hand, the swarm V. RESULTS
converges on “too close to call”, the swarm is given a second For each of the five weeks of the testing period, predictions
question asking if the predicted outcome is most likely a tie. In were made for the full slate of 10 matches that were played by
the example shown in Fig. 4 above, the artificial swarm English Premier League teams. For each set of 10 matches, a
demonstrated strong conviction that Arsenal would beat group of participants provided their individual predictions via a
Watford by a wide margin. In Fig. 5, a series of snapshots private online survey. The group also logged into the UNU
demonstrate how the swarm converged upon this final answer platform for real-time swarming and made predictions by
over time. It should be noted that all predictions made by the working together as a unified swarm. In addition, data was
swarm were converged upon in under 60 seconds. collected from the BBC indicating the predictions made by the

63 | P a g e
978-1-5090-6435-9/17/$31.00 ©2017 IEEE
 Intelligent Systems Conference 2017
7-8 September 2017 | London, UK
SAM super-computer for the same games. After the games a normal distribution with that mean and standard deviation,
were played, the results were scored by computing the number statistical significance is revealed. The value indicates the
of correct predictions and the percentage of correct predictions probability of obtaining the score of the swarm by chance.
for each test case. For individuals, the average values were
computed across the 25 to 31 participants in each group. These
results are shown in Table 1 below:

TABLE I. SUMMARY OF PREDICTION RESULTS OVER 5 WEEKS

INDIVIDUALS SWARM SAM
# Group %Correct %Correct %Correct
Games Size
Week 1 10 28 42% 60% 40%
Week 2 10 31 60% 80% 70%
Week 3 10 31 58% 80% 90%
Week 4 10 25 59% 60% 50%
Week 5 10 31 55% 80% 70%
MEAN 29 55% 72% 64%
StDev 2.7 7% 11% 19%

Assessing the raw results, it’s clear that the swarm had the Fig. 6. Individual vs Swarm predictions, assessed weekly.
best performance of the three experimental cases tested,
achieving 72% accuracy when predicting English Premier To aggregate the results from the five weeks into one, the
League games. This was significantly more accurate than the overall number of hits made by the swarm and the number of
same individuals, when predicting independently, as they hits made by the average individual (rounded to the closest
averaged only 55% accuracy across each group. And finally, integer) are compared using a two-proportion z-test, with the
the analytical super-computer, SAM, achieved a result in the null hypothesis that the two hit rates are the same. A z-statistic
middle of these two cases, generating 64% accuracy. was obtained using the following formula:
Thus, at a first level of analysis it’s clear that by working z = (pIND – pSWARM) / sqrt(p*(1-p)*(2/50))
together as a swarm, individuals who averaged 55% accuracy
Where, pIND is the hit rate of the average individual,
when working alone were able to amplify their accuracy to
pSWARM is the hit rate of the swarm and p is the total sum of
72% by forming real-time swarms and making the predictions
hits made by both the average individual and the swarm and
together. This corresponds to 131% amplification in predictive
divided by the total number of predictions (i.e., 100). The result
accuracy across five consecutive weeks (50 games). This also
show that the average individual was significantly worse than
corresponds to a performance level that not only matched, but
the unified swarm intelligence (z = -1.78, p = .03). The
slightly exceeded, an “expert source” of game predictions, the
aggregated results can be shown in a single profile, as depicted
SAM super-computer used by the BBC to publish expert picks.
in Fig. 7 below. The red line indicates the superior
Thus, by forming artificial swarms of approximately 30
performance of the human swarm as compared to the
individuals, groups of EPL fans (mostly American) were able
individual forecasters.
to make game predictions at an expert level.
To assess statistical significance, the swarm’s performance
was compared to the performance that would be expected by
chance from a matching population using a bootstrap approach.
This was performed as: For each week, a random sample of 10
individuals who participated in that week’s trial was generated
including the first individual’s prediction for the first match,
the second individual’s prediction for the second match and so
on until the sample included 10 predictions from the 10
randomly selected individuals. An average accuracy was then
computed for these predictions. This procedure was repeated
(i.e., random selection of 10 individuals and response
assignment) 10000 times, computing the average distribution
of correct answers for that week. Fig. 7. Individual vs Swarm predictions aggregated for all five weeks.

Distributions are shown in Fig. 6. The mean of the VI. CONCLUSIONS

distribution represents the average number of correct
predictions that should be expected by chance, by a matching Can swarms of novice participants such as casual sports
forecaster’s population. It can be seen that swarms are well fans rival the predictive abilities of a respected expert source?
above the mean as compared to individual predictions. By The results presented herein suggest this may be the case. As
computing the distance of the swarm performance for each demonstrated across five consecutive weeks of EPL match
week from that week's mean in the form of a z-score distance predictions, swarms of approximately 30 average sports fans
and computing the value of the cumulative density function of were able to achieve competitive results to the SAM super-

64 | P a g e
978-1-5090-6435-9/17/$31.00 ©2017 IEEE
 Intelligent Systems Conference 2017
7-8 September 2017 | London, UK
computer that the BBC employs for providing expert level REFERENCES
predictions to the public. In fact, 30 average sports fans, when [1] Rosenberg, Louis. “Artificial Swarm Intelligence vs Human Experts,”
working together as an Artificial Swarm Intelligence, out- Neural Networks (IJCNN), 2016 International Joint Conference on.
IEEE. J. Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd
predicted the SAM super-computer in four of the five weeks. ed., vol. 2. Oxford: Clarendon, 1892, pp.68–73.
Even more significant, by thinking together as a unified swarm [2] McHale, Ian. Boshnakov, Georgi, and Kharrati, Tarak. “A Bivariate
intelligence, the groups of approximately 30 casual sports fans Weibull Count Model for Forecasting,” Association Football Scores.
were able to significantly amplify their collective performance Working Paper 2016.
across all five weeks of EPL match predictions, boosting their [3] Seeley T.D, Buhrman S.C 2001 “Nest-site selection in honey bees: how
overall prediction accuracy by 131% as compared to the well do swarms implement the ‘best-of-N’ decision rule?” Behav. Ecol.
average individual participant. Sociobiol. 49, 416–427
[4] Marshall, James. Bogacz, Rafal. Dornhaus, Anna. Planqué, Robert.
These promising results suggest that continued research Kovacs, Tim. Franks, Nigel. “On optimal decision-making in brains and
and development is warranted in the area of Artificial Swarm social insect colonies.” Soc. Interface 2009.
Intelligence, as it is very likely that improved methods could [5] Seeley, Thomas D., et al. "Stop signals provide cross inhibition in
result in even higher levels of intelligence amplification. collective decision-making by honeybee swarms." Science 335.6064
Current research efforts include exploring human swarms of (2012): 108-111.
larger sizes, comparing human participants of varying expertise [6] Seeley, Thomas D. Honeybee Democracy. Princeton Univ. Press, 2010.
and demographics, developing improved swarming algorithms [7] Seeley, Thomas D., Visscher, P. Kirk. “Choosing a home: How the
scouts in a honey bee swarm perceive the completion of their group
for combining large numbers of individual sentiments in real- decision making.” Behavioural Ecology and Sociobiology 54 (5) 511-
time, and developing optimized methods for aggregating 520.
results across multiple parallel swarms. [8] Usher, M. McClelland J.L 2001 “The time course of perceptual choice:
the leaky, competing accumulator model.” Psychol. Rev. 108, 550–592
ACKNOWLEDGMENT [9] Rosenberg, L.B., “Human Swarms, a real-time method for collective
Thanks to David Baltaxe and Chris Hornbostel of intelligence.” Proceedings of the European Conference on Artificial Life
Unanimous A.I. for their significant efforts in supporting the 2015, pp. 658-659
data collection for this study. Thanks also to Unanimous A.I. [10] Rosenberg, Louis. Baltaxe, David and Pescetelli, Nicollo. "Crowds vs
Swarms, a Comparison of Intelligence," IEEE 2016 Swarm/Human
for the use of the unu.ai platform for this experiment. Blended Intelligence (SHBI), Cleveland, OH, 2016, pp. 1-4.

65 | P a g e
978-1-5090-6435-9/17/$31.00 ©2017 IEEE