Lecture 5

Eliciting Expert Judgment

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani

Copyright © 2020-2021, American University of Beirut, All Rights Reserved

Outline
▪ Role of experts in project risk management
▪ Expert opinions from the class
▪ Best practices
▪ Estimating parameters
▪ Assessing subjective probabilities
▪ Heuristics and bias
▪ Obtaining risk and time preferences
▪ Combining opinions from various experts
▪ Assessment protocol

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
2
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Obtaining Risk and Time Preferences

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
3
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Time Value of Money
▪ Most projects take place over a significant time period

▪ The distribution of costs (or revenues) over time may have a

significant impact on the project’s value

▪ Some trade-off decisions may imply moving cash flows in time

▪ Discounting is the correct way to compare amounts of money that

do not occur at the same date
ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
4
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Discounting
▪Discounting allows comparison of non risky amounts received at
different time and of different streams of amounts
▪ Present value for a given rate is the amount which is equivalent to
the stream of cash flows generated by the project
▪ Internal rate of return is a rate such that the NPV of a stream of cash
flows is 0

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
5
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
What is the right discount rate?
▪The time value of money depends on many factors
▪ Impatience
▪ Alternative opportunities
▪ Funding costs
▪ Inflation
▪It depends on the organization and should be set by the
stakeholders of the project
▪It should not include risk considerations
ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
6
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Assessing the Discount Rate
▪ How much additional funds ($𝑀) are the stakeholders of the
project willing to invest now in order to accelerate the project?
▪ Time considerations – by 𝑥 months
▪ Value considerations – which will allow them to reap $𝐵 benefits earlier
▪ Cost considerations – by borrowing (or not investing) money at rate 𝑖

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
7
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Comparing Amounts at Different Dates

0 𝑇−𝑑 𝑇

𝑇−𝑑
𝐵−𝑀 1+𝑖
0 𝑇−𝑑 𝑇

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
8
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Risk Perception / Attitude

10 000 $
0.5

0.5
50 000 $

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
9
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Risk Perception / Attitude

10 000 $
0.5

0.5
-10 000 $

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
10
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Risk Perception / Attitude

10 000 $
0.99

0.01
-10 000 $

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
11
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Risk Perception / Attitude

10 000 Columbian Pesos

0.5

0.5
-10 000 Columbian Pesos

1Columbian Peso = 0.44

Lebanese pound
ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
12
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Risk Perception / Attitude

10 000 $
0.5

For Jeff Bezos

0.5
-10 000 $

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
13
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Risk Attitudes
100
0.5
~ 25
0.5 0
100 Risk averse
0.5
~ 50
0.5 0
Risk neutral 100
0.5
~ 75
0.5 0
Risk seeker
ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
14
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Risk Preferences
▪The « risk premium » is the amount one is willing to pay to
reduce risk
▪It depends on the risk attitude and on the perception of the
risk
▪It is difficult to elicit
▪Examples: insurance, price of reputation, price of flexibility

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
15
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Amount at Risk
500
450
400 risk
350 premium
300
250
200
150
100
50
0
0 200 400 600 800 1000

expected value certainty equivalent

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
Project Risk Management - Eliciting expert judgme
16
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Uncertainty
100
90
80
70
60
50
40
30
20 risk premium
10
0
0 0.2 0.4 0.6 0.8 1

expected value certainty equivalent

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
Project Risk Management - Eliciting expert judgme
17
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Consistent Risk Attitude

▪Finding the certainty equivalent for each uncertain

event is difficult and time consuming

▪Consistency problem

▪Decentralization problem

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
18
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Utility Theory
▪ Utility theory is based on behavioral axioms that are generally
acceptable
▪ Acceptation of these axioms leads to the existence of a utility
function that can be used for any uncertain venture
▪ Utility functions can treat any variables and multiple attributes
▪ Utility is a scale measuring preferences for outcomes.
Conventionally, it takes values from 0 to 1

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
19
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
A concave utility function
1
0.9
0.8
0.7
0.6

utility
0.5
0.4
0.3
0.2
0.1
0
0 2000 4000 6000 8000 10000

amount

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
Project Risk Management - Eliciting expert judgme
20
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Building a Utility Function

-$2,000

0.5

0.5
+$10,000

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
21
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
The Exponential Utility Function
▪ Within a reasonable range of values, many utility curves can be fit by an
exponential function

−𝑥
𝑢 𝑥 =𝑎− 𝑏𝑒 𝑅

▪ R is the risk tolerance (in the units of x)

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
22
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Cognitive Biases
▪ Endowment effect: the value of a good is larger when you own it
than when you don’t

▪ Preference reversals

▪ emotions

▪ Small probabilities (rare events)

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
23
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Combining Opinions from Various
Experts

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
24
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Why use multiple Experts
▪Because assessment is important
▪Because one may need various perspectives
▪Because knowledge is distributed
▪Questions:
▪ How many?
▪ Which ones?
▪ How should you aggregate judgments?

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
25
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Three Approaches

▪Aggregate the judgments (e.g. distributions)

▪Aggregate the experts

▪A combination (e.g. Delphi)

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
26
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Aggregation of Judgments
▪ Easily performed using specialized software (e.g. @RISK)

▪ Weights can be given to experts according to their credibility

▪ For instance, a distribution is produced as a weighted average of

cumulated percentiles.

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
27
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Aggregation of Experts
▪ Facilitates discussion and confrontation of opinions
▪ Workshop configuration, managed by a facilitator
▪ Experts are asked to provide a consensus judgment or a consensus
distribution
▪ May require considerable time, clarification, information sharing
and compromise
▪ May lead to biases in the consensus opinion (bandwagon effect,
halo effect)
ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
28
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Delphi Method
▪ Panel of anonymous experts
▪ Experts answer questionnaires and comment the answers of
others
▪ Opinions can be revised – multiple feedback rounds
▪ A facilitator manages the flow of information
▪ Usually continues until consensus is reached
ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
29
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Expert Risk Assessment Protocol

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
30
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Motivation
▪ Expert risk assessment plays a major role in project risk
management

▪ The process by which expert knowledge is obtained must be

documented and must stand up to professional scrutiny

▪ In general, this means the development of a protocol satisfying a

number of steps and properties.
ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
31
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Steps of an Assessment Protocol
1. Background (evidence dossier)
▪ Identify elements for which expert assessment is needed
▪ Research determining the extent of scientific knowledge and/or
available data
▪ Examine objectives of stakeholders to ensure gathering relevant
information

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
32
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Steps of an Assessment Protocol
2. Identification and recruitment
▪ External or in-house
▪ Relevant expertise (meaning?), diverse perspectives
3. Motivation of experts
▪ Explain the context
▪ Stress the importance of expert knowledge
▪ Explain how information will be used

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
33
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Steps of an Assessment Protocol
4. Elicit experts’ knowledge model
▪ Relationships, chains of reasoning

5. Training
▪ in expressing beliefs and opinions in the form of quantities
(probabilities, values, rates, risk aversion)
▪ in providing honest and un-biased information

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
34
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Steps of an Assessment Protocol
6. Elicitation and verification
▪ Check for consistency
▪ Obtain confirmation
▪ Document information and rationale
7. Aggregation
▪ Encourage discussion and confrontation of opinions
▪ Control interactions to avoid biases
▪ Obtain consensus / weights.
ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
35
Copyright © 2020-2021, American University of Beirut, All Rights Reserved
Readings
▪ Brown, G. E. (2019). Eliciting Expert Opinion in Acquisition Cost and
Schedule Estimating.
▪ Szwed, P. (2016, April). Expert Judgment in Project Management: Narrowing
the Theory-Practice Gap. Project Management Institute.
▪ Hora, S. C. (2007). Eliciting probabilities from experts. Advances in decision
analysis: From foundations to applications, 129.

ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
36
Copyright © 2020-2021, American University of Beirut, All Rights Reserved