Model Risk

CHAPTER 11
Robert Shiller
[Coursera Course ‘Financial Markets’]
I was on stage with the chief economist of, Freddie Mac here at Yale.
It was around 2005 and he was boasting about their stress tests.
So I asked him what if there's a real estate crisis and home prices fall a lot.
They're company that guarantees mortgages on homes, and so he said, "Well,
we have figured out what would happen to our portfolio even under extreme
stress situations."
I said, "What's the biggest price decrease you ever considered for your stress
test?"
They said, "Oh, we considered a 13% drop in home prices."
Robert Shiller
[Coursera Course ‘Financial Markets’]
And then I said to him, "Well, what if it's bigger than that?"
And then he looked chagrined then he said, "We've never seen home price
drops, not since the Great Depression. You're not talking about another
depression, are you?".
We're still friends. I still meet him on vacation but the problem is that home
prices fell 30% right after that meeting or within a couple of years.
……
……
Chagrined: Feeling or caused to feel ill at ease or self-conscious or ashamed
(WordWeb)
Introduction
The use of models by financial institutions is growing fast.
Models are employed for making credit decisions, liquidity management, the
evaluation of credit exposures, the valuation of derivatives, the calculation of
risk measures such as value at risk (VaR) and expected shortfall (ES), the
management of portfolios of financial assets for clients, the assessment of
capital adequacy, customer relationship management, fraud detection, the
identification of money laundering, and so on.
Complex models, with advanced analytic techniques such as machine learning,
are being used to automate some activities previously carried out by humans.
It is now recognized that large financial institutions need a model risk
management function to ensure that models are appropriate for their intended
purpose and are used in the correct way.
Introduction
Models are approximations to reality.
Models are used in finance both for pricing of instruments and for risk
management
The art of building a model is to capture the key aspects of reality for a
particular application without allowing the model to become so complicated
that it is almost impossible to use.
Nearly always, a model relies on some assumptions about the phenomenon
being modeled.
It is important to understand those assumptions and know when they are no
longer appropriate.
Introduction
It used to be the case that models could be developed by a quant on a
spreadsheet and used with minimal oversight.
This is no longer the case.
Models must now be well documented and reviewed periodically to determine
whether they are still appropriate and are working as intended.
When changes are made to models, approval is usually necessary.
Model risk is a component of operational risk.
Introduction
According to the Federal Reserve Board, the term model refers to a:
Quantitative method, system, or approach
that applies
statistical, economic, financial, or mathematical theories, techniques and
assumptions
to process input data into quantitative estimates…
Models in Physics and Finance
Some models in finance are structurally similar to those in physics.
◦ For example, the differential equation that leads to the famous Black–Scholes–
Merton option pricing model.
However, there is an important difference between the models of physics and
the models of finance:
◦ The models of physics describe physical processes and are highly accurate.
◦ By contrast, the models of finance describe the behavior of market variables,
customers, borrowers, and so on.
◦ The phenomena being modeled are a result of the actions of human beings.
◦ As a result, the models are at best approximate descriptions of the phenomena being
modeled.
Models in Physics and Finance
One important difference between the models of physics and the models of
finance concerns model parameters:
◦ The parameters of models in physics generally do not change. For example, the
gravitational pull on the surface of the earth.
◦ By contrast, parameters in finance models change daily. The volatility used to price
an option might be 20% one day, 22% the next day, and 19% the following day.
Introduction
Model risk is the risk of potential loss when models are used for pricing and
hedging
The potential losses may arise out of wrong trading and risk management
decisions due to flaws in the model
Model risk can result from either
Fundamental errors in the model, or
Inappropriate use of a model, particularly outside the environment for which it
was designed
Sources of Model Risk
Incorrect model specification:
◦ Distribution of the underlying asset
◦ Omitting an important risk factor
◦ Wrong assumptions (say, independent vs correlated, market conditions, etc.)
◦ Overfitting
◦ Over-parameterization
Simple models have been found to be more successful than complex models
Complexity should be introduced only when necessary
Sources of Model Risk
Incorrect model application:
◦ Model is applied in a wrong manner (example: bond pricing model applied for CDO)
◦ Outdated model (market conditions might have changed)
Implementation risk:
◦ Different users feed different inputs (same variable can be calculated in a variety of
ways based on different parameters)
◦ Proxy factors when underlying factor is unavailable
Calibration error:
◦ Calibrated using wrong inputs, outdated inputs, or inputs from a wrong sample-
period
Sources of Model Risk
Programming error:
◦ Presence of bugs
◦ Use of incorrect random numbers
◦ Complex code
Data problem:
◦ Data obtained from a third party may be biased
◦ Data not collected properly
◦ Unclear definition
◦ Missing data points
◦ Outliers
Managing model risk
Model risk cannot be avoided completely
It can be mitigated by actions of risk practitioners, senior managers and the
institutions
◦ Examine the models for their sensitivity to the underlying assumptions
◦ Apply models first to simple problems for which solutions are known
◦ Keep the models simple
◦ Stress-test and back-test the models
◦ Use simulations and out-of-sample forecasts
◦ Check the model’s predictive ability
◦ Reevaluate and update models regularly
◦ Maintain proper records
Managing model risk
◦ Senior managers should question the model features, track model performance,
implementation and misuse.
◦ They should be aware of and guard against the tricks played by rogue traders
At the institution level
◦ A proper procedure should be in place for vetting and reviewing the models
◦ Models should be vetted for completeness before their implementation
◦ The institution should also have an Independent Risk Oversight unit (IRO) that is
responsible for overall risk measurement and management
Further Reading
Introduction
Two types of models are used in finance:
Structural models: produce estimates of an output using certain inputs and
assumptions
Example: the Black-Scholes option pricing model
Statistical models: define statistical relation between variables
focus on correlation between variables rather than any causal relationship
Example: models that derive hedge ratios
Model Building Missteps
The art of model building is to capture what is important for valuing and hedging
an instrument without making the model more complex than it needs to be.
A financial institution may find that a more complicated model is an
improvement over a simple model until there is a regime change.
The more complicated model may not then have the flexibility to cope with
changing market conditions.
Simple models are often the most successful.