Monitoring and Evaluation

For MPH students

1
 Objectives of the session

• At the end of this lecture, students will be able to

 describe the basic concepts in M&E
 discuss importance of monitoring and evaluation
 identify the different types of M&E activities
 Explain the different types of evaluation
 identify the steps in conducting program evaluation
 Explain how M&E works in the Ethiopian Health Sector
2
 Controlling
• Progress of planned
activities must be
controlled

• Controlling takes place

through monitoring and
evaluation

• Monitoring and evaluation

(M&E) provide the critical
link back to planning
3
 Basic concepts in M & E
• Monitoring: is the systematic and continual collection,
analysis, interpretation and use of data on key aspects of
an intervention and/or its expected results to track
progress towards the achievement of planed targets.
It includes group of activities that:
• Follows up priority information about a program and
its expected results.
• Follows up the program’s costs.
• Compares performance with plan
• Provides information that can be used for program
evaluation. 4
 Basic concepts in M & E
• Evaluation is a systematic process of data
collection and analysis about activities and/or
effects of a program, looking to explain and/or judge
the levels of program implementation and/or
achievement of desired effects

5
 Basic concepts in M & E
• Depending on the type of evaluation question, evaluations can
be

– process evaluation : Give explanation for observed level of

compliance/deviation in program implementation

– Outcome/Impact evaluation: Determines the merit or the

value of the program and explains the relationship between
the program and its effects

6
 Basic concepts in M & E
• The M & E process involves
1. Establishment of standards
2. Measurement of performance
3. Comparing actual performance against the
standard
4. Correction of deviations

7
Monitoring and evaluation process

8
 Purpose of M and E
Problem
identification and
cause analysis

Inform Goal and

M and E
stakeholder objective setting

Developing
implementation
plan
9
 Principles of M and E
• Focused and feasible
• Timely
• Useable
• Credible, valid and reliable
• Sensitive
• Ethical

10
 Users of M and E information

To improve
program Managers
implementation

To inform and Donors, Governments and

improve future technocrats
programming

Donors, governments,
Inform
communities and
stakeholders
beneficiaries

11
 Basic concepts in M & E
• Inputs:
– Resources used in the program.
– They include financial, human or material
resources.
– Examples:
• Technical personnel for clinical services
• Anti TB drugs
• Mosquito nets
• Laboratory supplies 12
 Basic concepts in M & E
• Activities:
– Program procedures that are implemented to
obtain desired effects.

– Examples:
• Training of human resources for HIV
counseling and testing
• Providing treatment for malaria patients
• Vaccinating children
• Giving health education 13
 Basic concepts in M & E
• Outputs:
– Are the immediate consequences of the inputs
utilized and program activities conducted.

– Examples:
• Number of malaria patients treated
• Number of clients counseled
• Number of ITNs distributed
• Number of HIV tests carried out 14
 Basic concepts in M & E
• Outcomes:
– Are effects upon the target population that can lead to
the intended ultimate goal of a program.

– The effects include several types and may focus on

awareness, attitudes, behavior, utilization etc.

– Examples:
• Increase of condom use
• Improvement of quality of HIV/AIDS services
• Reduction of risk behaviors 15
 Basic concepts in M & E
• Impacts:
– Are related to long-term accumulative effects of
programs

– They are rarely attributed to a single program or

intervention.

– Examples:
• Reduction of Malaria incidence
• Reduction of HIV/AIDS mortality
• Improvement in quality of life of patients 16
 How do Monitoring and Evaluation relate to each other?

• Purpose
– Both furnish information for decision making

• Complementarity
– Evaluation questions arise from monitoring activities
– Evaluation uses data generated by monitoring
– Evaluations are used to explain observations from routine
monitoring
• Methods
– Both involve systematic collection and analysis of program
related information.
17
 How do monitoring and evaluation differ from each other?

item Monitoring Evaluation

Frequency Periodic Episodic
purpose Descriptive Explanation
Main action Documents progress In-depth analysis of
achievements (How and
why?)

Answer What are we doing? Do the program has any

questions contribution? how? Why?
Undertaken Conducted by program Can be done by internal
by staff or external experts

focus Input, process, output, Effectiveness , relevance

outcome, workplan impact, costeffectivence
18
Uses of Evaluation

Program
Improvement

Generating
knowledge

Accountability

19
 Uses cont…

• Program Improvement
– Evaluations can be conducted to furnish program
implementers with information that can be used to take
improvement actions.
– Such types of evaluations are called Formative
Evaluations.

20
 Uses cont…
• Accountability

– Evaluations are sometimes conducted to provide summary

judgments about a program’s performance.

– And such information from evaluation activities is usually used

by higher level decision makers including donors and political
leaders to decide whether to continue or discontinue program
components.

– Evaluations serving this purpose are called Summative 21

 Uses cont…
• Knowledge generation
– Evaluations sometimes may be designed just to
contribute knowledge.

– Such evaluations usually don’t have specifically

identified users of evaluation findings.

– Like other research activities, findings may be shared

with interested scholars or others through publication or
other channels.

22
 Uses cont….
• What evaluation should not be used for:
– Evaluations should not be initiated to justify
decisions which are already made.

– Evaluations should not be conducted to affect

(positively or negatively) the people involved in the
program.

– Evaluation should not be used as a means to delay

decisions that could happen using already available
credible information. 23
 Different M & E activities
• Input/Output Monitoring:
– Follow up of information about inputs or resources and about outputs resulting
from program activities.
Answers questions such as:
• Which services were provided?
• Which resources were used?
• What is the number of people the program reached?
– Examples:
• Input
– Follow up of total expenditure on health
– Follow up of the number of condoms acquired every quarter by the
program.
• Output
– Performance monitoring - follow up of the number of people who received
24
 Different M & E activities
• Process Evaluation:
– It supplements the monitoring of inputs and outputs with an explanatory
component.
– Answers questions such as:
• Was the intervention implemented according to what was expected
(compliance)?
• What is the implementation degree of the program and what
program/context/users related factors may explain the observed
degree of implementation ?
• Are the planned actions reaching the targeted population?
• Do the users have access to the intervention?
• Are there barriers that make access difficult or unfeasible?
25
 Different M & E activities
• Outcome Monitoring:
– Follow up of information related to the program’s expected outcome.

– No intention to attribute observed changes to a program

– Usually is related to a period of time

– Answers questions such as:

• How does the behavior of target groups look like?

• Was there change in the intended outcome?

– Example – Behavioral surveillance

• How is condom use during casual sex changing over time during the period
of program implementation?
26
 Different M & E activities
• Outcome Evaluation:
– Gives emphasis to causal relations between intervention and effect.

Answers question such as:

• Does the intervention explain the observed effect on the target

population?
• How much of observed changes is due to the intervention

– Example: Was the observed change in condom use due to the intervention? Why?
How? To what extent?

27
 Different M & E activities
• Impact Monitoring:

– Usually, it relates to the follow up of disease trends in the general population.

– Answers questions such as:

• What effects do all interventions have upon HIV and AIDS

prevalence?
• How does all malaria prevention measures influence the incidence
of malaria?

– Example: Biological surveillance for HIV (Systematic follow up of HIV prevalence

28
 Different M & E activities

• Impact Evaluation:

– Analyze the relationships between disease trends, control programs and

other associated factors.

Answers questions such as:

• How much of the change is due to the program?

– Example: Impact evaluation of ITN distribution project

• What was the contribution of ITN distribution in decreasing the incidence of malaria?

29
 Different activities in M & E
Economic Evaluation

-Applied analytic method to identify, measure, value, and

compare the costs and outcomes of alternative interventions.

-It is a comparative analysis of alternative courses of action in

terms of both their costs and consequences. (Drumond, et al.)

-Examples:
- CEA
- CUA
- CBA

30
 Types of Evaluation
• Evaluations could be classified based on
different characteristics including:

– Purpose of evaluation

– Focus of evaluation

– Who conducts the evaluation

31
 Types of Evaluation

• According to their purposes, evaluations could

be classified as

 Formative
 Normative
 Summative

32
 Formative Evaluation:
• Formative evaluations are evaluations conducted with the
primary purpose of furnishing information that will guide
program improvement.

• Is a type of evaluation performed during the entire

planning process and program execution to answer
evaluation questions important to modify an intervention.
33
 Formative Evaluation:
• Such evaluations provide solutions for program improvement
by answering questions such as:

– How can the intervention be modified to achieve its outputs

and outcomes?

– Are there better solutions compared to those proposed by

the program?

– How do the components of this program relate amongst

themselves? 34
 Normative Evaluation
• Usually performed to assess a program’s compliance to
best practices.

– Answers questions such as:

• Is the program following recommended guidelines?
• Does the program follow national guidelines?
• Does the program comply with prescribed norms?

35
 Summative Evaluation
• Summative evaluations are evaluations undertaken to render a
summary judgment on certain critical aspects of a program.

• Usually performed to provide judgment about the worth of a program

or any of its components.

• Answers questions such as:

– Is the program effective?

– Should the program be continued? 36
 Types of evaluation cont
• According to their focus, evaluations could be
classified as:

– Process Evaluation

– Outcome/Impact Evaluation

37
 Process Evaluations
• Also called implementation evaluation
• Focus on the implementation of program activities
answering questions like:
• Was the intervention implemented according to what was
expected (compliance)?
• What is the implementation degree of the program and
what program/context/users related factors may explain
the observed degree of implementation?
• Are the planned actions reaching the targeted population?
• Do the users have access to the intervention?
• What are the barriers that make access difficult or
unfeasible?
38
 Outcome/Impact Evaluation
• Evaluations focusing on the assessment of net effects of
programs on target populations are called outcome/impact
evaluations.
– Outcome – focus on intermediate results
– Impact – focus on long term ‘ultimate’ results

• Involve comparison of:

– Levels of social conditions intended to be addressed by
the program and
– The counterfactual (what would have been expected if
the program was not in place)
39
 Types of evaluations cont…

• Based on people primarily responsible to

accomplish evaluation activities, evaluation
could be classified as:

– Internal Evaluation

– External Evaluation

40
 Internal Evaluation
– Evaluation activities designed and implemented by
program implementers

– Usually serve information for program improvement by

supplementing monitoring activities.

41
 External Evaluation
– Evaluations designed and implemented primarily by
external evaluators

– Used when:
• evaluation expertise beyond the organization’s
capacity is required to answer evaluation
questions

• objectivity is a concern because of issues related

to the purpose of the evaluation

• concerns of multiple stakeholders included in

evaluation questions 42
 Basic steps in conducting evaluations
1. Get the interested parties involved
– People involved or affected by the program and primary users of the
evaluation (stakeholders).
2. Describe the program
– Needs, components, activities, context, resources, logical model
(articulating expected effects and activities)
3. Focus the evaluation design
– Purposes, questions, uses, methods, agreements, users.
4. Gather evidences with credibility
– Collect data from relevant sources.
5. Explain conclusions
– Standards, interpretation, judgment and recommendations.
6. Ensure use and share lessons learnt 43
 Setting indicators for monitoring and evaluation

• Indicators are variables that are measured to tell about the level
of program implementation or achievement of desired effects on
target populations.

• Indicators for program M&E could be related to:

– Availability of resources
– The quality of activities
– Amount of outputs produced
– Achievement of outcomes and impacts on target populations
44
Monitoring and Evaluation in the Ethiopian Health
Sector

• M&E is one of the processes in the Policy,

Planning, Monitoring and Evaluation Core Process
at the FMoH

• Similar structures exist in different RHBs.

• Under PPME are:

– Policy formulation
– Planning
– Monitoring and Evaluation
45
Monitoring and Evaluation in the Ethiopian
Health Sector
• Major responsibilities of M&E structures in the
health sector
– Routine collection and aggregation of data (Health
Management Information System)
– Performance Monitoring and Quality Improvement
– Integrated Supportive Supervision
– Evaluation

46