Market Simulations

A New Approach to Marketing Strategy

Simulate Your Market

Copyright © 2016 Concentric Inc.

THE CHANGING SITUATION

With the proliferation of new media types, the explosion of available data, the rising influence of social
networks, a general desire to reduce advertising spend, and the increasing number of new product
failures, marketers are finding it increasingly difficult to pick strategies that produce desired outcomes.
The reaction to this challenge has been a massive investment in data and insights, which has also failed to
provide a solution to the problem. A recent survey from the CMO council identified the following:

 92% of CIOs are under pressure to integrate various data sets

 71% of CMOs do not use the data they have purchased
 61% of CEOs struggle to evaluate the impact of marketing initiatives on business goals
 66% of CFOs are unable to validate ROI of marketing investments

In this turbulent and fragmented environment, simulation has emerged as a new technology that solves
the challenges of the C-suite by delivering the following benefits:

This white paper defines the simulation concept as it applies to marketing, explains how our product
works, and identifies applications to a wide variety of business questions.
THE SIMULATION CONCEPT

When pilots practice flying new airplanes, they use flight simulators. When doctors train, they use models
of human bodies. When engineers design computer chips, they test their ideas in silico before sending
them to the factory. When marketers test ideas, strategies, and tactics about consumer markets, they use
Concentric’s simulation application, Concentric Market™.

Concentric Market™ simulates consumer markets with accuracy that matches or exceeds the highest
statistical standards of any existing marketing analytic tool, while providing insights that go further. Our
users diagnose the performance of offline, online, and social marketing vehicles. They explore strategies
to develop and launch new products and evaluate the risk of new competitive entrants. They forecast
consideration, perceptions, word-of-mouth, sales, and return on investment for each of the brands
competing in the market and for each of their target consumer segments.

Ultimately, Concentric simulations allow users to build better marketing strategies faster because they
recreate market dynamics, validate to multiple metrics, and learn with artificial intelligence. The following
sections go into more detail on each of these three capabilities.

Recreates market dynamics

The two most widely used marketing analytics approaches are brand tracking and marketing mix
modeling. The former measures the consumer preferences in the market, but has no predictive power to
forecast what would happen if consumers or the marketing change. The latter attributes ROI per media
per brand, but does not take into account consumer preference changes.

Simulation offers the same capability of brand tracking reports and marketing mix modeling and
combines them to allow analysis of what-if scenarios about consumers and marketing strategies. In this
way, simulation models the entirety of the market, rather than approaching the consumer and the
marketing strategy as two separate systems to be analyzed.

Validates to multiple metrics

Since simulation has a broader ambition than other marketing analytic tools, i.e., to replicate the entirety
of the market, it requires a more stringent validation to ensure its accuracy. Concentric Market™ uses the
market data in two ways to validate the model: for calibration and for a hold-out test. Data from the
calibration period is used to fit the simulation, whereas data from the holdout period is only used after
running the simulation to evaluate how well it forecasts. Users commonly forecast from one quarter to
two years, but some users have forecasted up to 10 years in the future.

While a holdout comparison is a good basis for judging a forecasting model, a single comparison on sales
for a particular brand is too small a set of validating measures for the scope of a simulation. Concentric
Market™ users validate to multiple brands, and to outcomes beyond sales such as perceptions, word-of-
mouth, and awareness. Additionally, users also evaluate all metrics by segment, a statistical benefit that
ensures appropriate model specification.
Learns with Artificial Intelligence

As part of Concentric’s conceptualization of a simulation, the application includes various forms of

machine learning. We define machine learning as a type of artificial intelligence that provides computers
with the ability to learn without being explicitly programmed. Machine learning focuses on the
development of algorithms that can teach themselves to grow and change when exposed to new data.
This capability is embedded into a pre-defined process for human and machine interaction:

The system enables the user to repeat this process within minutes, creating a cycle of learning and
exploration that could not be achieved through an individual or even a team. This leads to broader
exploration of alternatives and learning through feedback, a process that in turn identifies risks and
rewards.
HOW IT WORKS

Machine learning on its own does not deliver the benefits of simulation. In order to create a working
model of a market, Concentric combines additional methodologies and scientific concepts. Included in
Concentric Markettm are principles from marketing analytics, behavioral economics, network science, and
agent-based modeling. Agent-based modeling is the core engine in Concentric Markettm and
understanding its components reveals how simulation works.

The components of our Agent-Based Model are:

1) Variables and Rules

2) Initial Conditions
3) Calibration Algorithms
4) What-if Scenario Analysis

VARIABLES AND RULES

Our algorithms start by describing the interactions of the key components (or agents) of the system with
a few simple individual-level rules that lead to aggregate behavior, matching what we observe in reality.

To begin with, the model accounts for the current marketing strategy of the brand and its competitors.
All activities of the brand communicate to the consumer and the system allows for all marketing
investments to be analyzed simultaneously.

Consumers have properties, including consideration and perceptions of different brands. These
properties are the drivers that govern what is important to each consumer when making decisions.
Consumers are also connected in a social network – the fabric of a consumer market. Finally, consumers
engage in behaviors related to products: they shop, choose among alternatives, experience the products,
and share their experiences with their social network.

A few rules define how consumer properties drive the behaviors. For example, the more consumers
consider a product, the more likely they are to make a purchase.

We use variables to change the impact of properties on behaviors. For example, the greater the social
density, the more likely information will spread about the alternatives in the market.

While behaviors are intentional, consumers are also affected by events, such as overhearing a friend talk
about a new product or reading an online review. These events are likely to change consideration and
perceptions. In addition to people in their social network, consumers are affected by outside factors, such
as marketing, influencers, the category seasonality, or the product itself. These external factors create
additional events that change the properties of consumers, and can be influenced by media investment
and creative quality.

So what we have now described is the entirety of the Concentric Markettm model. The diagram below
shows all the variables, rules, and intervening factors.
The most important point here is that unlike most marketing analytics approaches, which have
dependent variables (like sales numbers) and independent variables (like marketing investment), an
agent-based model is a system dynamics model in which many variables are interdependent. This means
that everything affects everything else. Sales are not a final outcome: the more people buy a product, the
more people experience it, and the more people change or solidify their perceptions and consideration.

INITIAL CONDITIONS

All variables and rules of the above schema are informed through a diverse set of information and data
that forms the initial conditions of the agent-based model:

1) Consumer research like brand trackers, focus groups, or panels, to inform consumer properties
2) Media and sales data, which are usually used in any marketing effectiveness analysis
3) Information about the actual performance of the product, like product specs and industry reports
4) Social tracking information about the volume and content of conversations

These disparate data sources, which were previously analyzed in isolation, are now integrated into initial
conditions for Concentric simulations.

There are two types of initial conditions:

On one hand we have inputs over time, like marketing investment (from a media plan), product changes
which may occur during the year, category seasonality, and some market events. On the other hand, we
have settings that we are only interested in inputting at the first moment of the simulation, because we
want to forecast their evolution over time.

Here is the key to the system dynamics approach: necessary assumptions are limited to the rules of the
market and the initial conditions (which are both verifiable by measuring simulation output). The
forecasts result from the logical evolution of the variables over time based on the interaction of these
conditions and rules.

CALIBRATION ALGORITHMS

Users are interested in four key metrics: consideration, perceptions, sales, and word of mouth. The first
thing to do when running Concentric Markettm is to make sure the simulation forecasts match the real
world measures of these KPI’s.

To do so, Concentric Market™ compares the simulated results with the real data and measures the error
between them. To improve the match between the simulation and the real world, a user identifies a set
of parameters to vary. Perhaps these are chosen because the user did not have the best information or
simply had no information at all.

An automatic calibrator systematically adjusts parameters, runs Monte Carlo simulations, calculates the
error margin between the updated simulation and the real results, and continuously narrows the search
until it finds the variable settings or the rules that best fit the simulation to the observed real world
results.
WHAT-IF SCENARIO ANALYSIS

Once the user is satisfied with the validity of the diagnostics and the forecasting accuracy of the
simulation model, the user creates what-if scenarios.

There are plenty of options to explore in Concentric Market™. Users change the media budget or mix for
their own brand or for a competitors. They introduce new products or competitors into the market or
change features of the product. They prepare for unanticipated PR or viral events. They change the
nature of the consumer segments and anticipate trends before they happen.

All simulations in Concentric Market™ are probabilistic, which means that when a user re-runs the same
scenario twice, they may get different results. As Concentric Market™ runs more and more simulations
using a process called Monte Carlo analysis, it forms a distribution of outcomes. This distribution is easily
represented through a box plot, which shows the minimum and maximum of outcomes, the top and
bottom 25%, the middle 50%, and a median outcome. The box plots help the user understand the
confidence to have in simulation forecasts: the wider the box plot the lower the confidence of the
baseline scenario, which is the scenario used to calibrate to past performance.
 Box-Plot

Box plots of Monte Carlo outcomes also help in comparing scenarios. Adding a new scenario, called
“Messaging Shift,” we see that it performs better than the baseline, as the median outcome is higher. By
showing the two scenarios side by side on four different metrics, the user can identify trade-offs.
Although the messaging shift scenario increased sales, it has a potential downside for awareness and
word of mouth. The most useful part of Concentric Market™ is the comparison of multiple scenarios over
multiple metrics – there is no limit to the number of ideas and strategies that one can test and explore.
SUMMARY

Concentric Markettm provides a system that combines social science and marketing science to solve
specific marketing challenges and enhance executive processes. Simulation reinvigorates marketing
innovation by empowering marketers to evaluate and optimize “outside the box” programs that they may
have been uncomfortable executing in the real world by safely evaluating them in a virtual environment.

What makes Simulation unique for solving challenges is that it can do all of the following simultaneously:

 Business Strategy Comparison

 Campaign/Channel Attribution
 Media Optimization
 Creative Content Optimization
 CRM Program Optimization
 Corporate Social Responsibility
 In-Store Strategy
 Pricing Analytics
 Product Development
 Product Launch Support
 Trade-dollar Assessments
 Targeted Media Buying

Additional benefits of a simulation approach stem from the implementation of a unifying process to:

 Improve resource allocation

 Make real-time/near-time decisions
 Identify new markets
 Communicate insights
 Facilitate change
 Encourage collaboration
 Enhance transparency
 Accelerate the development of new products/services

With a more complete answer, at lower costs, and at faster speeds, simulations are getting broad
adoption and are set to transform marketing strategy.

We hope you join Concentric on our quest to better marketing strategy.

Thoughts on Simulation
A gathering of our technical blogs

Simulate Your Market

Copyright © 2016 Concentric Inc.

WHAT DOES A SIMULATION LOOK LIKE?

While simulation is a commonly used tool in areas like social science, operations, physics, and engineering, it is not
well established in the marketing community. Often times, we are asked what is going on under the hood. For
those looking to see the “nuts and bolts” of a marketing simulation, we have created a learning map pictured
above. Below we offer some key definitions of the elements of our simulation system.

Perceptions

At the core of an agent-based model is the recreation of consumer behaviors. How they gather information,
influence other, and use the product all interact to change their perceptions. Changing perceptions combined with
Behavioral Economics is the core forecast engine of the system. Consumer-centric forecasting is a central
difference between traditional media-centric models.

Drivers

With consumers at the center of the model, simulation accounts for diverse consumer preferences by detailing the
relative importance of attributes by segment. Brands that are perceived to best meet drivers win more market
share than other brands.
Connectivity

In addition to including consumers, simulation accounts for the effect social networks, online behavior, product
usage and influencers have on changing perceptions. Capturing the impact this earned media is a unique feature of
simulation that allows marketers to consider strategies that go well beyond typical measures of paid media impacts.

Marketing

No model of marketing would be complete without a clear understanding of how media investments stimulate
consumers in the context of their social setting. Simulations allow for the inclusion of marketing mix models or
digital attribution results as an empirical foundation of the system. Additionally, simulations can be used as an
alternative to traditional models of attribution because they account for all touchpoints in the marketing plan
simultaneously.

In summary, the above map of a simulation of a consumer market shows how simulation fuses social and marketing
science to account for the entire competitive environment and to keep consumer behavior at the center of analysis.
FRAMING A MARKET – BRINGING STRUCTURE TO A SIMULATION

Simulation is flexible and somewhat unconstrained by nature. This open-endedness makes simulation useful for
understanding real-world markets in a broad variety of contexts. But given this open-endedness, it becomes
important to remain focused on a specific objective and to bring some structure to the simulation.

Accordingly, the first step in building any Concentric Market simulation is to construct a simulation framework. This
framework should be built based on the consumer decision we are seeking to better understand, in line with some
business objective. The framework consists of the following elements:

 Alternatives: What set of brands or options are in the consumers’ choice set?
 Attributes: What factors do consumers consider when selecting among alternatives?
 Touchpoints: What marketing activities or events influence consumers’ decision or behavior?
 Segments: What different types of consumers exist in the market?

Most importantly, these elements should be selected according to the specific consumer decision you wish to
simulate. Here are some additional guidelines when building a simulation framework, along with the typical
number of framework elements in simulations:

Alternatives (3-10):

 Consider the market share of the competitors. It’s usually appropriate to include the dominant competitors in
the market and to exclude brands with very small share.
 Factor in how directly the alternative competes with your brand. It’s a good practice to consider not only the
size of the competitor, but also whether or not they present a threat or an opportunity for your brand’s market
share. You may choose to include a brand with low market share if they are an emerging threat to your brand.
 Think about if there are any substitutes in adjacent categories that would be helpful to consider.
Attributes (2-6):

 Consider the factors that matter most to a consumer when making a decision. Typically this list can be
summarized into a few key drivers of choice.
 Observe what consumers are talking about and what messages marketers are broadcasting. This can provide
insight into what factors matter the most.
 Think about both tangible attributes (physically experienced by the consumer) and intangibles (more ethereal in
nature, i.e. “cool”, brand reputation).

Touchpoints (5-150):

 Consider anything that can change consumer perceptions or awareness or stimulate conversation in the
marketplace. Touchpoints may include traditional media channels, and also other types of interactions with
the brand (for example, in-store touchpoints, events, PR, salespeople, online blogs).
 Remember to consider not only your brand’s marketing activities, but also the activities of the competitors.
 Think about the level of detail at which marketing strategy decisions will be made when determining how
granular to make the list of touchpoints. For example, if the simulation will only be used to determine broadly
the mix of television and online advertising, then it would probably not be helpful to expand the touchpoint list
to include splits for different TV programs, 15 vs. 30 second ads, etc.

Segments (1-6)

 Consider how the population can be divided into groups based on differences in behavior, media habits, drivers
of choice, and perceptions/awareness about the alternatives.
 Select a set of segments that are mutually exclusive and encompass the relevant population. Segments may be
demographic, attitudinal, psychographic, or based on other factors so long as that holds.
 Think about if there are any key non-purchasing influencer groups (i.e. doctors/pharmacists, children) that
should be included in the simulation. Even if an influencer group does not purchase any of the alternatives,
they may still be active in the social network and have an impact on outcomes.
HOW DATA FITS INTO CONCENTRIC SIMULATION

Building a simulation does not require any data. However, validating a simulation does.

Building a simulation only requires that we make assumptions. We assume initial conditions, constraints, and
behaviors within the system we seek to understand. Qualitative or quantitative data may be used to inform some
of these assumptions, but often times we just apply insights or judgment.

Once we have made assumptions and built a simulation, we run the simulation and see if it accurately reflects real
world dynamics. This validation is what requires data.

Calibrating a valid simulation is an iterative process. The test-and-learn nature of the process is what frees us from
being bound by data when initially building the simulation. However, the validation points by which we judge the
accuracy of our simulation are based on data.
BUILDING CONFIDENCE IN ATTRIBUTION
FROM A SIMULATION APPROACH

Simulation, like any other modeling approach, is an imperfect representation of the real world. A question that
commonly comes up when performing analysis and forecasting is “how can we be confident that what this model
tells us is right?” Of course, all modeling results have some uncertainty, but attribution is a particularly difficult
finding to validate.

In a nutshell, attribution is the decomposition of a variable into the elements driving changes in its
value. Attribution results in a measurement of the impact of each element. For example, marketing mix analysis
often decomposes sales into base (an unexplained contribution) and individual marketing channel contributions.
The difficulty in building high confidence about attribution is that the numbers reported are virtually impossible to
measure in the real world. If we say that TV contributes 30% to the increase in sales, we are making an accounting
statement. There is no way to go in the field and truly isolate that 30% figure. Getting to that 30% requires
assumptions about how we measure effects and how we quantify contribution.

Yet, while attribution is in this sense more art than science, there are ways to increase our confidence in the
attribution results that come from a simulation. More often than not, increasing the confidence requires investing
more time into validation of the model against metrics that are actually observable and measurable in the real
world. Only then, by proxy, is our confidence boosted that our latent results are as valid as the results we can
compare with measured observations.

Below we outline 4 routes that can be taken to increase the confidence in the simulation attribution:

1. Calibrate the simulation to multiple metrics at an aggregate level.

2. Perform a holdout forecast.
3. Compare individual consumer journeys to simulated agent journeys.
4. Conduct in-market tests to assess impacts.

CALIBRATING THE SIMULATION TO MULTIPLE OBJECTIVES

Simulations are often built with a relatively large number of parameters and initial conditions. This often raises a
question of how one can be confident that they have not over-fit the simulation. One way to avoid over-fitting risk
and build confidence is to calibrate to more data points.

Typically, regression approaches fit to one time series, for example, sales over time for one brand. With a market
simulation, one may calibrate to sales for the primary brand of interest as well as sales for its competitors. The
number of data points that build confidence in the fit is then quickly scaled up by the number of competing brands.
In addition to calibrating to sales across brands, one may also calibrate to other metrics beyond sales. Simulations
can be calibrated to brand metrics such as awareness, perceptions, and word-of-mouth volume. The number of
calibration points can be scaled even further by the number of KPIs that are fit.

Fitting a simulation to data from multiple brands across KPIs builds confidence that it is not over-specified and that it
recreates dynamics of the real-world. This topic is explored further in the next blog post.

PERFORMING A HOLDOUT FORECAST

Often stakeholders are interested in the predictive power of a model or simulation. In this case, the evaluation of
the approach goes beyond analytics to explain the past and into how well it projects future outcomes. A holdout
forecast is one approach for assessing predictive capabilities.

When conducting a holdout, the historical data is split between a calibration time frame and a holdout period. The
model or simulation parameters are tuned to the calibration time frame only. This calibrated model is then used to
make a forecast during the holdout period.
Using a holdout, we see how effectively the simulation was able to forecast past outcomes and this provides a
better sense of what sort of predictive accuracy may be anticipated in forecasting future outcomes.

ANALYZING CONSUMER JOURNEYS

One of the advantages of simulation is that it recreates the dynamics that take place at an individual level in the real
world. Aggregate sales outcomes emerge from many individual consumer decisions. One may be interested in
seeing not only how well macro outcomes are fit by a simulation or model, but also how well the approach recreates
individual-level data. These journeys can give us a better sense of how all the marketing activities and earned
touchpoints interact to lead to sales.

If data on individual consumer journeys is available, this can serve as another way to assess the validity of a
simulation. Simulated agent journeys can be recorded and compared to the data on individual consumer
journeys. If the simulated journeys align with the actual consumer journeys, then the simulation is tuned to the
micro-level dynamics of the simulation.
CONDUCTING IN-MARKET TESTS

Ultimately, an audience may choose that the best way to assess the impact of a marketing strategy is to actually put
it into practice in the real-world. An in-market test allows for data to be gathered that can help measure
impacts. Although running a test in the actual market allows one to see how an outcome pans out in the complex
real-world, it is difficult to run a controlled experiment when the many variable factors of the real world come into
play. Even a real-world experiment has its limitations in terms of building confidence in a particular finding.
HOW TO COUNTER THE RISK OF OVER-SPECIFICATION WHEN CALIBRATING AN AGENT-
BASED MODEL

Practiced statisticians are very familiar with the problem of over-fitting when building traditional econometric
models. As the number of parameters approaches the number of data points fitted two things will happen:

1. The model will fit the history progressively more precisely.

2. The model will become progressively more over-specified, eventually to the point of losing all statistical and
predictive value.

This is why experienced model-builders place such high value on parsimony and generally avoid complicating a
regression model with extra variables unless they contribute significant explanatory power. The classical “degrees of
freedom” problem can eliminate a model’s credibility.

To summarize:

As the number of regression parameters increases with a number of data points, over-fitting risk also increases

Similar concerns may arise when building a simulation. Sure, having the ability to adjust a variety of initial conditions
and settings in an agent-based model may provide a lot of flexibility, but may we encounter the problem of over-
specification when calibrating an agent-based model?

The answer is yes, absolutely. With all the various model settings in Concentric, one could simulate an annual sales
figure that matches with historical values with an array of input assumptions. Many of these assumption sets would
not align with reality and result in simulations that are misleading and lacking in insight.

There are two ways to avoid the problem of over-specification in agent-based modeling.

1. Reduce input uncertainty: Decrease the number of freely floating inputs when calibrating your model by
building confidence in your input settings via research, data analysis, expertise, or norms. This will cut down
the model calibration search space to a more manageable size and reduce your risk of over-fitting.
2. Increase the number of validation points: Calibrate to more metrics and trends based on what you know to be
true. Go beyond sales to assess awareness, word-of-mouth volume, and brand equity outputs. Analyze the
sales and brand metrics over time to see if the trends are sensible. Look at brand equity attribution to see if
the relative impacts of each touchpoint are realistic. Carry out sensitivity analyses to see if the simulation
mimics marketing responses you would expect.
By implementing these two approaches, you are attacking the problem of over-fitting risk from both sides: You are
reducing the number of parameters in your search space and increasing the number of data points in your
validation.

Your efforts should always be geared towards the business question you are ultimately trying to answer. This should
help guide where to focus your energy in reducing input uncertainty and selecting your validation points.

Concentric is meant to be used as a “thinking system.” The more thought that goes into the system inputs and
validation efforts, the more confidence you will have in the insights the system generates. So keep this simple rule in
mind:

As Thinking ↑ , Over-fitting risk ↓

Always.
HOW DO MARKETING MIX MODELING AND SIMULATION ACCOUNT FOR UNCERTAINTY

“I can calculate the motion of heavenly bodies but not the madness of people” said Isaac Newton and almost four
centuries later, the sentiment is still strong among practitioners and consumers of predictive analytics that forecast
human behavior.

It is true – we lack theories and empirical evidence to come close to predicting human behavior with accuracy found
in the physical sciences, and making choices based on models of human behavior is risky. But we have identified
ways of quantifying and analyzing that risk.

One of my projects back when I worked in actuarial science was to quantify a range around the expected paid losses
(i.e., the portion of losses that the insurer actually pays). I set to work analyzing the historical data and began
attempting to quantify the range.

Process Risk
After fitting a model to the data, my first approach was to estimate how much variability seemed to be inherent in
the process. I quantified the range by estimating the error term of the model – i.e., the volatility observed in the
actual data around the model fit – and assuming a distribution for the error term. This type of variability is often
referred to by actuaries as “Process Risk”.

Parameter Risk
I proudly shared my results with the group, but then another issue was raised. In order to compute an error term, I
had to assume a model with specific, fixed parameters. What if those parameters, which had been calibrated to a
sample of data, were not accurate? On top of the “Process Risk”, there is additional risk due to uncertainty
surrounding the model parameters – “Parameter Risk”. Adding a bit more statistical rigor to my analysis, I came up
with an estimate of this term.

Model Risk
With both process and parameter risk taken into consideration, I was confident in the range around the expected
loss estimate. Then I was asked a final question. What about “Model Risk”? What do we do if the assumed
functional form or structure of the model does not reflect the reality of the underlying process? This one had me
stumped. After months of analysis and research, I could not solve this puzzle. Actually, it turns out the solution to
the “Model Risk” predicament had alluded the entire group.
In statistical models, the different types of risk can be quantified in a formulaic manner. In simulations – different
types of risk can be assessed in an empirical manner – based on more iterative approaches.

So how does simulation deal with the various types of risk?

Monte Carlo Analysis Assesses Process Risk

This involves running multiple iterations of a simulation with the same initial conditions. The simulation can evolve
differently each time due to the probabilistic and path-dependent nature of the approach. We can then observe
the inherent risk in a given strategy.

Sensitivity Analysis Assesses Parameter Risk

This entails building a series of simulations with inputs that are tweaked in each run. Variation in simulated
outcomes may be observed as the inputs are adjusted. We can then assess how sensitive outcomes are to shifts in
initial conditions or assumptions for which we are uncertain.

Testing Different Rules of the Simulation Assesses Model Risk

Assessing model risk requires critical thinking about the fundamental assumptions made in the simulation
framework. Have we considered all the relevant competitors or drivers of choice? Do we understand the
consumer segments and all the touchpoints that affect them? We should also consider the rules of the
simulation. Are the underlying algorithms relevant in our real world market? Are our assumptions about how
consumer decision making, paid media effects, and the social network accurate? These questions raise challenges
that cannot be addressed overnight. “Model Risk” is best mitigated through collaborative, creative, and strategic
thinking.

So Newton is still right: We cannot account for all types of risk and uncertainty. This is not just because of the
limitations of our analytical methods, but due to the shear unpredictability of a complex, ever-changing world. The
best we can do is to decipher patterns and build insights through data analysis and statistics, research and
observation of the world around us, and our intuition. Simulation can help in our efforts to better understand the
uncertainties and risks involved in markets and business strategies. And these efforts will be most successful with a
team that is open, adaptive, knowledgeable, persistent, and creative.
OVER-SPECIFICATION: COMPARING AGENT-BASED AND MARKETING MIX MODELS

A common objection that we hear during discussions about agent-based modeling is that agent-based models are
over-specified and therefore their findings are invalid. Although it is possible with ABM — as with any other
modeling technique — to encounter the issue of over-fitting, building the simulation diligently will minimize the risk
of over-specification (more details on how to minimize the risk of over-specifying are given in the preceding blog
post). In fact, agent-based models often have a lower risk of over-specification than marketing mix models. In this
post we will explore an illustrative example.
A metric that is commonly used to assess whether or not a model is over-specified is the ratio of calibration data
points to model parameters. If there are too few data points per model parameter, the model is over-specified. In
an extreme case where there is one parameter for every calibration data point, the model can achieve a perfect
fit. The parameters, in a sense, would just be a transformed restatement of what happened historically. This
model would not be very useful for making any forward-looking projections or drawing any conclusions about what
happened in the past.

Let’s compute the ratio of data points per model parameter for an example agent-based model and marketing mix
model below:

MARKETING MIX MODEL

Calibration Data Points:

Marketing mix models are often fit to a time series of a brand’s sales. In this example, let’s assume the model is fit
to two years of historical weekly sales data. That gives us:

2 years x 52 weeks = 104 calibration data points

Calibrating Model Parameters:
Marketing mix models are typically fit by linking changes in sales to different marketing activities. Let’s assume
there are 8 touchpoints that are being analyzed. There will be a coefficient that defines the impact of each
touchpoint on sales. Additionally, each touchpoint‘s marketing activities can also be modulated by parameters that
define the point of saturation and decay rate of the marketing impact. Taken together, there are 3 parameters
defining each of the 8 touchpoint effects plus a y-intercept – so a total of:

8 touchpoints x 3 parameters per touchpoint + 1 = 25 calibrating model parameters

Ratio of Data Points to Model Parameters:

104/25 = 4.2 data points per model parameter
AGENT-BASED MODEL

Calibration Data Points:

Concentric simulations are often fit to multiple time series. These time series correspond to KPIs across competing
brands. Let’s assume the simulation is fit to one year of historical weekly sales data, but not just for one, but for six
competing brands. Let’s also assume that the simulation is fit to tracking data on awareness and perceptions for
each of the six brands. However, in this case, let’s say the tracking study was only run quarterly. The total number
of calibrating data points is then:

6 brands x 52 weeks of sales + 6 brands x 4 quarters of awareness + 6 brands x 4 quarters of perception =

360 calibration data points
Calibrating Model Parameters:
Concentric simulations are built by integrating data from various sources and calibrating a set of parameters to
available calibration data. Let’s assume that the settings defining consumer behaviors and initial preferences and
awareness levels are provided from a consumer survey or tracking study and are therefore constrained. The
parameters that are used to fit the simulation are then based on three types of settings:

1. Touchpoint impacts on awareness and perceptions: 4 per touchpoint

2. Consumer awareness and perception decays: 2 in total
3. Decision-making parameters: 2 in total
8 touchpoints x 4 parameters per touchpoint + 2 + 2 = 36 calibrating model parameters

Ratio of Data Points to Model Parameters:

360/36 = 10 data points per model parameter

In the example above, the risk of over-specifying the ABM is lower than the risk of over-specifying the marketing mix
model. Despite the fact that the ABM is fitting a time period that is half the length, the number of calibrating data
points per parameter is over twice as high as the marketing mix model. This is because an ABM fits an entire
market as opposed to one particular brand in isolation.

Ultimately, there are a number of approaches that may be taken to build confidence in the validity and usefulness of
a given model. These approaches include alternative specification formulas like a log function, calibration against
multiple time series, holdout forecasts, and in-market tests. By diligently taking such approaches, our users
consistently build valid and useful market simulations with agent-based modeling.
ATTRIBUTION TO WHAT?
UNDERSTANDING CROSS-CHANNEL INTERACTIONS

Cross-channel effects occur in situations where the sum of the parts is not equal to the whole. These effects can be
positive (synergistic) or negative (inefficient).

For instance, imagine that you have two media channels at your disposal: TV and Facebook. Should you run them
separately or together? The answer to that question depends on the nature of cross-channel effects between these
two channels: whether there is a positive or negative interactive effect from running these two channels together. If
the sales lift you get from running these two channels together is different from what you would get if you ran each
channel separately and added the sales lift you got from each, then there is a cross-channel effect. The types of
cross-channel effects that can occur are defined as follows:

Synergistic Cross-Channel Effect

A case when the cross-channel effect results in a synergy

In this case, the sum of the parts is less than the two considered as a whole. There is some extra lift in sales that
results from a synergistic interplay between both channels.

Inefficient Cross-Channel Effect

A case when the cross-channel effect results in an inefficiency

In this case, the sum of the parts is greater than the two channels working together, which has created some
inefficiencies.

The cross-channel effect is therefore defined as the difference between the individual lifts and the total lift:

[Lift in Sales from TV] + [Lift in Sales from Facebook] + [Cross-Channel Effect] = [Lift in Sales from TV & Facebook].

The effect of marketing activity on sales is often analyzed using regression techniques using sales as a dependent
variable. Cross-channel interactions may be quantified with regression by including an interaction term in the model
specification. This analytical technique is useful for measuring the effects of media, i.e., allowing for sales attribution
and cross-channel interactions to be computed. But the task of understanding what mechanism actually causes the
cross-channel effects remains a mystery.

To fully understand the mechanism that drives cross-channel interactions – we need to recognize that the linkage
between marketing activity and sales lies in the mind of a consumer. The consumer’s decision-making behavior is
what ultimately drives sales. So how do consumers choose?

Consumer Decision-Making is the Key to Cross-Channel Effects

The growth and acceptance of behavioral economics as a science behavior has been instrumental in allowing us to
take some simple rules to create simulated models of consumer decision-making that take into account heuristics
and rationality. This knowledge has allowed us to build models to measure consumer behavior that go beyond
notions of perfect equilibrium and neo-classical economics that were never originally designed for this purpose.

In simple terms, consumers make decisions based on options they would consider, and select the option they
perceive most highly on attributes that are most important to them (based on the behavioral principle of utility
maximization). This view assumes a path in the consumer-decision making process as follows:

The components of the consumer decision-making process lead to the different cross-channel effects
This approach means that both brand awareness and brand perceptions are important in driving sales. Some
touchpoints may be more effective at building awareness and others at building perceptions. It won’t do the
marketer much good to focus on building brand awareness if the product is perceived very poorly – a perception or
satisfaction bottleneck that may not be solved by throwing more media at the problem. Likewise, it might be
unhelpful to attempt to build deeper engagement for a brand no one recognizes – an awareness bottleneck.

Synergies and inefficiencies may result from the awareness stages that exist in the complex non-linear path to
consumer decisions. Inefficiency may occur when activities focused on awareness-building or perception-building
continue when brand awareness or brand perceptions are already maximized. We may saturate awareness and
observe that any additional marketing yields no incremental benefit. A synergy may occur when touchpoints that
build awareness and others that build perceptions are used in conjunction. We will get more out of our perception-
building media campaign, for instance, if more of the consumers are made aware of the brand.

In short, inefficiencies may occur when we concentrate too much on one step in the consumer-decision making
process, and synergies may occur when we balance our efforts between different steps. See below for a mapping of
TV and Facebook from our example.

A mapping of two channels on two criteria: how well they affect awareness and perceptions

Building understanding of what drives synergies and inefficiencies within a marketing plan can bring valuable
additional insight to strategic decision-making. Thinking about the path in the non-linear consumer-decision making
process – and the various bottlenecks and saturation points therein – may allow a team to consider new creative
strategies that balance awareness building and perception building activities. Simulations may then serve as an aid
to testing these hypothetical strategies prior to implementation to include interim metrics such as brand awareness,
brand perceptions, and WOM activity.

So as you think about the attribution of all of your brand activities, ask yourself: attribution to what?
FROM OPTIMIZATION TO STRATEGY:
DEEPER INTO THE ATTRIBUTION STACK

Sales attribution is a common diagnostic used in evaluating marketing effectiveness. This involves decomposing a
brand’s sales into “Base” and “Media” components. Media is often further divided to attribute sales by distinct
touchpoints. The result can be visualized as a stacked attribution chart as follows:

Stacked Attribution Chart

This chart provides an indication of how much touchpoints are lifting sales and therefore can be used to assess the
Return on Investment (ROI). A planner may use these outcomes to change the mix, putting more weight into the
touchpoint with greater ROI.

Simulation makes it possible to dive deeper into attribution analysis to provide more insights and further guidance
on marketing strategy. Here are four more detailed views of the stack that simulation provides:

1. Understanding how creative and messaging impact the effectiveness of each touchpoint
2. Identifying the synergies and inefficiencies that exist between channels
3. Decomposing the base to understand the role of earned media and brand history
4. Placing the attribution into a consumer-focused competitive context

CREATIVE EXECUTION AND MESSAGING EMPHASIS

The concepts of creative execution and messaging emphasis allow us to go beyond analysis of levels of investment
and reach to analyze the effectiveness of media. Creative execution defines how well the ad is created — the
production value and how effectively it communicates the message. Messaging emphasis defines what attributes
of the product are the focus of the ad. For example, is the ad part of a promotional campaign or a branding
campaign? Unlike most marketing mix modeling, simulation allows us to test the effects of different creative
executions and messaging emphasis and decompose each individual touchpoint contribution further.
 Creative Execution and Messaging Emphasis

CROSS-CHANNEL EFFECTS

Touchpoint impacts may interact with one another through duplication or social network effects. Inefficiencies
occur when reaching the same audience via multiple touchpoints leads to no gain. Synergies occur when messaging
on two touchpoints yields better results than using each touchpoint on its own. Simulation can serve as a guide to
better understanding and replicating these interactions in the consumer’s mind and provide a platform to test
different approaches to fully leverage synergies and avoid inefficiencies.

Cross-Channel Effects

DECOMPOSING THE BASE

The “Base” is often the forgotten piece of the attribution stack – an unexplained variance that does not yield any
actionable insights. Running a simulation that incorporates the effects of earned media – social networks and
product experiences – allows for a better understanding of what drives base sales. Often times, word-of-mouth
(new restaurant opens up) and the actual experience of the product (the joy of driving your automobile) play a
major role in driving a brand’s performance. Sometimes the base is simply a matter of consumer inertia (think
about insurance), seasonality (canned soup), or a brand’s history (Coca-cola vs. Pepsi). Better understanding the
base may provide strategic ideas on how to avoid erosion of the base or how to keep it growing strong.

Decomposing the Base

CONSIDERING THE COMPETITION

Sales outcomes are the aggregate result of many individual consumer decisions. Consumers make choices among a
set of alternatives – and so in many cases a brand’s performance is just as much dependent on what its competitors
do as what the brand does itself. Building a simulation of an entire market where simulated consumers choose
among competitive alternatives allows for an assessment of how a brand will fare in different competitive
environments. If a brand positions itself appropriately, it may rapidly grow by cannibalizing competitive share. If
not positioned appropriately, the entire attribution stack may sink into oblivion (think about Blockbuster Video with
the emergence of Netflix and Redbox). Considering competition is key to evaluating strategies to boost and
maintain market share over the long run.

Considering the Competition

CONCLUSION – MOVING BEYOND OPTIMIZATION TO STRATEGY

The initial view of the attribution stack allows for an assessment of ROI that can guide decisions around how much
to invest in touchpoints. However, assessing the stack from the lens of creative messaging, cross-channel synergies
and inefficiencies, earned media and brand history, and the competitive environment allows for a richer
understanding of how a brand is likely to perform in its market. Simulation provides flexibility in aperture to plan
and react strategically in a variety of ways.
MARKETING EFFECTIVENESS MEASURES: KNOW YOUR OPTIONS

There are a variety of approaches used to quantify the impact of marketing activities and attribute sales to
touchpoint activities. Such methods include:

 Marketing Mix Modeling

 Test and Control Group Experiments
 Attribution Modeling
 Agent-based modeling

In marketing mix models, sales or some other KPI of interest are regressed against marketing activities to
understand how changes in the KPI are related to changes in marketing activities. The resulting coefficients define
the sensitivity of the KPI to touchpoint activities. From this model specification, the quantity of sales to attribute
to a given marketing activity may be derived.

In test and control experiments, one group (the test group) is exposed to a marketing treatment while another
group (the control group) is left unexposed. Ideally, these two groups are otherwise completely the same. The
difference in outcomes (sales, response rate, or other KPIs) between the two groups is then attributed to the
marketing treatment.

In attribution modeling, individual level data is used to assess what individual touchpoint activities (commonly
digital ads) led to a sale or some other action. These models typically link a KPI (such as a sale or website visit) to
one or some combination of activity that preceded it. These accounting approaches include last touch, first touch, or
some other weighting across touchpoint activities.

In Concentric agent-based models, the dynamics of a marketplace are recreated by simulated consumers who
decide between alternatives, interact in a social network, and experience marketing. In this way, the impacts of
marketing activities are considered in the context of all interactions a consumer may have with brands in a market –
including not just paid marketing for a single brand, but competitive actions and earned media. From these
simulations, an attribution to sales and brand metrics may be derived by comparing differences in simulated
outcomes when touchpoint activities are added and removed. The decomposition of sales and other metrics goes
beyond paid media to include earned media and competitive impacts.
MARKETING MIX MODELING AND AGENT-BASED SIMULATION – WHAT IS THE QUESTION?

Often we are asked about how agent-based simulation compares to marketing mix modeling. How do the
methodologies compare? Although parallels between the two exist in the context of planning paid investments in
marketing, the question is flawed because the two methodologies serve very different purposes.

MARKETING MIX AND SIMULATION FOR PLANNING INVESTMENTS IN MARKETING

Marketing Mix Modeling is a statistical approach that explains variation around some baseline value in a dependent
variable (often times sales) due to marketing activities. A regression is constructed between time series of sales
data and marketing to attempt to correlate changes in one to another. The model’s coefficients are calibrated to
minimize the error between actual and modeled sales. The model is then applied to answer questions around
marketing investment level and mix.

Agent-based modeling is an approach based on simulation that recreates a market and its sales from the bottom-
up. Interdependent variables within the simulation recreate the dynamics of the market. Each sale in the
simulation results from the action of an individual simulated agent. The initial conditions and parameters define
how consumers behave, influence one another, make decisions, and interact with media. The parameters of the
simulation are adjusted to minimize error between actual and simulated sales. The calibration processes confirms
that the simulation is an accurate representation of the world. The simulation is then applied to answer a broad set
of questions about marketing and product development resource allocation.

A number of parallels between the two methodologies often create confusion about their similarity. Each has
coefficients or parameters that are adjusted to minimize error in an attempt to better reflect reality. Each
approach may be used to guide decisions on paid marketing investments. The key distinction is that marketing mix
analyzes a single brand’s activity in aggregate, while agent-based simulation recreates a brand’s marketplace based
on individual consumer actions. The differences are summarized further below:

Difference in Methodology Marketing Mix Modeling Agent-Based Modeling

Approach Top-down regression Bottom-up simulation

Scope One brand Multiple brands, multiple segments

Outputs One metrics Multiple metrics

Forecasts Deterministic Probabilistic

Inputs Quantitative time series Quantitative or qualitative info

Marketing mix modeling is well-suited to answer questions related to the optimization of paid marketing
investments. Agent-based simulation provides the capability to answer a broader set of strategic questions about
resource allocation. That brings us to the second section.

MARKETING MIX AND SIMULATION WORK WELL TOGETHER

We have seen a number of cases where a decision maker has both a marketing mix model and an agent-based
simulation in their toolkit for making decisions. In this context, simulation is used to extend marketing mix to
answer more business questions. A typical process is described below.

The marketing mix model is calibrated to specify the impact of the different marketing channels a brand uses. The
coefficients of the marketing mix model are then used to guide the selection of parameters for the agent-based
simulation. The simulation is tuned to replicate a marketing mix model’s response curves and marketing attribution
results, while also calibrating to sales and other KPIs. At this point the simulation is calibrated to multiple objectives
and also incorporates insights on consumers from research or other data.

The mix model or simulation may now be used to answer questions around marketing investment and mix. The
simulation may be applied to address a range of questions. Because a simulation incorporates multiple KPIs,
segments, attributes, competitors, and probabilistic outcomes it may extend mix to do the following:

 Interdependent KPIs: Consider trade-offs between sales, perceptions, awareness, and word-of-mouth
 Consumer segments: Develop a segment targeting strategy
 Attributes driving choice: Test creative messaging
 Competing alternatives: Respond to competitive actions and/or optimize a portfolio
 Probabilistic outcomes: Assess risk based on a distribution of likely outcomes

So in summary, marketing mix modeling is an approach for performing optimization of paid marketing investments
for one brand in a stable environment. Simulation explores a broader range of strategic options that consider the
competitive market and consumer behavior. So the question on how marketing mix compares to simulation always
comes back to what questions you need to answer as a decision-maker.
ADVANTAGES AND DISADVANTAGES OF SIMULATION

Simulation is an approach that is used most commonly in two situations.

The first situation is when uncertainty is high due to sparse data. One such example is a simulation of an ancient
Native American tribe, the Anasazi, a culture that lived between the 9 th and 14th centuries. It is hard to run typical
analytics on the limited available data, so researchers use simulation to understand what happened to the tribe.
A second common use of simulation is for experimentation in a low-cost, low-risk environment. Researchers at CERN
simulate particles colliding in the Large Hadron Collider, before they validate their forecasts in the expensive real-
world collider in Switzerland. More common examples include airline pilots practicing on simulated flights and
doctors learning on test patients.

Both of these applications of simulation are helpful to scientists and researchers, but they come with a set of
advantages and disadvantages. We have grouped these advantages and disadvantages into three broad areas
related to technology, process, and socialization. The following table gives a summary of the advantages and
disadvantages, which we will elaborate on in a three-part series of blog posts.

Advantage Disadvantage

Technology Forecasting under uncertainty Good theories needed

Able to answer many questions No standardized approach

Process Low data requirements to model Challenging to validate

Easy what-if scenario analysis Potential scope creep in projects

Socialization Low cost High skepticism

Innovative approach Political implications

GREAT FORECASTING POWER, BUT A GOOD THEORY IS NEEDED

Data analysis methods such as regression are limited to forecasting effects of events that are similar to what has
already happened in the past. For example, if a brand has been investing in TV ads within a range of $50M to $100M
in the past few years, a marketing mix model is excellent at forecasting what would happen if spend is within those
bounds. However, the model is likely to produce nonsensical results once it extrapolates to forecast what would
happen if TV spend is doubled or if a new marketing channel is deployed.

Simulation has an advantage over these methods in that it allows us to forecast things that have never happened
before and to run scenarios outside of historical bounds. The caveat is that we need a good theory and causal
hypotheses about how the system we are interested in analyzing works. Theories that have high predictive power,
at least in social science, are hard to come by and may take years to develop.

FLEXIBLE, BUT NOT STANDARDIZED

Simulations, and agent-based modeling in particular, provide highly flexible techniques for answering a wide range
of research questions. These questions include what happened in the first moments of the Universe, how wind
turbulence around aircraft works, how the World Wide Web evolves, or how to better design hospitals. Although
simulation can be applied in a variety of contexts, a formalized set of rules and best practices is not always readily
available. For this reason, simulation modeling (especially in social science) is incredibly creative, but may be
daunting for new researchers who have no single reference to consult when starting out.

BUILDING A SIMULATION DOES NOT REQUIRE DATA, BUT VALIDATION DOES

Simulation is an excellent approach to analyze problems when the available data is limited, since no data is
necessary to construct a simulation. Validating a simulation, however, often requires multiple data sources to
achieve a great degree of confidence in its representation of real-world dynamics. The process of validation is a
disadvantage for simulation when comparing to data analytics approaches, since validating simulations is often
more difficult.

For example, if we wanted to simulate traffic on a road, we would not need any data to start. We could construct a
simulation that incorporates modeled cars, driver behaviors, and road conditions and voila: we have a traffic
simulator.

Analysis of this traffic simulation could provide surprising insights – such as the pattern of traffic jams migrating in
the opposite direction that automobiles are traveling. But to test whether such insights are valid, we would need to
use various data. We would seek information about road conditions in a range of contexts – in cities, on highways, in
the U.S. and other countries, and under different weather conditions. We could then recreate all of those scenarios
within the simulation and see how well they match what actually happened in the real world. If the simulation data
and the actual “validation” data match, then we have confidence that our simulated model of traffic is useful and
valid.

To get the simulations to match real-world outcomes, we need to change the theoretical rules guiding the
simulation or test different assumptions until they do. Simulations have the benefit of forecasting multiple metrics
simultaneously, but this can make it challenging to get all of the assumptions synchronized. One change may
improve the forecast for one metric, but degrade the fit for another. Fortunately, expanding computing power and
improving algorithms continue to reduce the time and effort to overcome the process barrier of calibrating and
validating simulations.
RUN ANY WHAT-IF SCENARIO, BUT STAY WITHIN BUDGET

We build simulations with the goal of having a “Petrie dish” in which to experiment in a controlled and low-risk
environment, before taking action in the real world. At the outset of a project, a team can often list off a broad
range of hypotheses to test within the simulator. Once a simulation is built and what-if scenarios can be run, the
desire to keep testing more and more scenarios often grows.

Going back to the traffic simulation example, the initial goal of the simulation might be to determine whether to
replace 4-way-stops with roundabout intersections in a particular section of town. The questions may compound
from there: What will the impact be of traffic lights in other parts of town? How should the signage be placed? How
will this impact traffic during the farmer’s market? It’s easy to see how the number of questions and scenarios can
multiply very quickly.

Enabling a team to test and answer more questions is a great value-add that simulation provides. Projects may start
by focusing on a single research question, but often grow to incorporate more complex ideas. This dynamic and
creative process can build consensus by bringing more stakeholders to the table and ultimately lead to better
decisions.

However, it is important when time and resources are limited to ensure that the scope of a project does not expand
beyond the available budget. Reaching agreement with the working team on the appropriate balance between focus
on a specific deliverable and open-ended exploration is a good step to take early in the planning process.

LOW COST, BUT HIGH SKEPTICISM

Compared to the cost of experimenting in the real world, the use of simulation requires very little time and
resources. Think about marketing: if we were to run various experiments in which we varied the amount we invest
in different channels, we would have to go through dozens of budgets over as many years to gather enough data to
answer a question with certainty. In the meantime, our brand and business may have gone in an undesirable
direction. The alternative to real world experimentation is to run simulations to test different marketing plans.
Within minutes we can test many ideas before acting on a plan and making decisions in the real world.

The disadvantage of this approach is that some audiences today are skeptical of simulation. Most of today’s analysis,
especially in marketing, is based on reporting and building deterministic statistical models to describe what has
happened in the past. Researchers often prefer these descriptive approaches to methods that test theories about
the future. We believe that this skepticism is a result of the relative novelty of simulation in marketing analytics, and
that with more success stories and validated forecasts, this skepticism will subside.

INNOVATIVE, BUT POLITICAL

Simulation may be one of the most innovative approaches researchers engage in today. We have seen people
advance in their careers for their intra-preneurial spirit in introducing simulation within their organization. Long-
standing and thorny problems get tackled every day with social simulation, but the socialization of simulation results
often presents organizational challenges. Because it identifies trade-offs between a range of metrics, simulated
insights may bring conflicting interests to the forefront in organizations where stakeholder incentives are not
aligned.

Building consensus around a simulation is done best when the process is organized from the start. All stakeholders
should agree on the simulation’s framework, assumptions, and questions to be answered at the outset. In our
experience, this is the best way to ensure that the simulation findings are impartial.