Scribd
Upload
85 views

C. Lecture 2 - Process Evaluation

1) Process evaluation is important for understanding how a program is implemented and whether it reaches intended participants, which provides context for evaluating the program's impacts. 2) A case study of a housing project in India that built homes for low-income workers but saw many residents move out over time illustrates how process evaluation can identify implementation challenges and help improve programs. 3) Both monitoring during implementation and a later process evaluation of the housing project found issues like homes being demolished and relocated farther from the city, which helped explain why over half of residents eventually moved out.

Uploaded by

DetteDeCastro
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PDF, TXT or read online on Scribd
85 views

C. Lecture 2 - Process Evaluation

1) Process evaluation is important for understanding how a program is implemented and whether it reaches intended participants, which provides context for evaluating the program's impacts. 2) A case study of a housing project in India that built homes for low-income workers but saw many residents move out over time illustrates how process evaluation can identify implementation challenges and help improve programs. 3) Both monitoring during implementation and a later process evaluation of the housing project found issues like homes being demolished and relocated farther from the city, which helped explain why over half of residents eventually moved out.

Uploaded by

DetteDeCastro
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 31

Process Evaluation More than Monitoring

Sharon Barnhardt
Assistant Professor, IFMR Research Director, Centre for Microfinance [email protected]
August 27, 2012

Goals
Appreciate the importance of formalizing the program theory for better programs and better evaluations. Understand why process evaluation is critical for impact evaluation.

Project & Program Lifecycles


Needs Assessment Program Theory Process Evaluation Impact Evaluation Cost-Effectiveness Analysis Collect Baseline Data Design Evaluation Plan Collect Monitoring Data & Assess it Collect Endline Data, Evaluate Impacts Compare outcomes across programs

Project & Program Lifecycles


Needs Assessment Program Theory Process Evaluation Impact Evaluation Cost-Effectiveness Analysis Collect Baseline Data Design Evaluation Plan Collect Monitoring Data & Assess it Collect Endline Data, Evaluate Impacts Compare outcomes across programs Understand Problem

Design Program

Implement Program Use Results to Improve Program


Consider Alternative Programs

Program Evaluation Lifecycle


Needs Assessment
Program Theory Process Evaluation Impact Evaluation Cost-Effectiveness Analysis Collect Baseline Data

Needs Assessment
Issue Why Who Context
Main Players

Understand nature of the problem.


Establish hypotheses for why the problem exists.

Determine specific target population.


Understand related services and their success, unique challenges, etc. Identify key stakeholders.

Needs Assessment
Gather information to clarify the need for a program that addresses a specific issue for a target population

Combine primary and secondary Primary data (household surveys, interviews, focus groups, etc.) Secondary data (official records, test scores, population surveys, etc.)

Housing Project: Gujarat, India


Rate of urbanization in India has been on an unprecedented scale.

Urban poor often live in slums that lack basic public services

Housing Project: Gujarat, India


Issue Why Who Context
Main Players Poor quality of life in urban slums: lowquality, insecure housing Short supply of affordable, small formal units (high transaction costs)

Low-income, self-employed female bidi rollers in Ahmedabad, part of union


Weak transportation and public service infrastructure, limited financing, etc. Union (SEWA), Bidi rollers & families, City government

Program Evaluation Lifecycle


Needs Assessment

Program Theory
Process Evaluation Impact Evaluation Cost-Effectiveness Analysis

Design Program Design Evaluation Plan

Program Theory
Series of hypotheses that identify how change will occur at each step towards obtaining desired results Clearly lays out the sequence of activities & behaviors we expect to see happen in this specific environment If they happen as expected, the program should have an impact Ideas come from many sources (academic literature, successful programs in other places, best practices)

Program Theory - Benefits


1. Illuminates assumptions. 2. Clarifies measures/data needed for evaluation. 3. Allows interaction with other theories to understand what works and what does not.
Theoretical Model Results Chain Logic Model Theory of Change Logic Framework Outcomes Model

Logic Model
Inputs, Activities
Human, financial, community resources Actions that convert inputs to outputs

Output

Outcome
Short-term Objectives: Change in participant behavior, knowledge, skills, etc.

Impact
Long-term Objectives: Change at community, and/or system level

Product of activities: program deliverables

Process Evaluation

Impact Evaluation

Logic Model: Housing example


Inputs, Activities
Land Housing Materials Mortgages Builder Applicants Lottery

Output
Houses built Bidi workers take mortgages Get possession

Outcome

Impact

Families have better housing Secure property rights

Higher earnings Better health Kids flourish

What are the assumptions made at each step?

Detailed Logic Model: Housing


Logic
Inputs (Activities) Get land, materials, financing, builder, and bidi workers Houses built Mortgages given Bidi workers take possession

Assumptions / Threats
Can get inputs Collaboration across players to build houses Bidi workers and their families will leave neighborhoods where their families live in order to get better housing Will stay in the house Wont sell Urban setting means transportation to work possible from anywhere Low mortgage payments free up money for other investments Previous locations density caused health problems

Outputs

Better Housing Outcome (Project goal) Secure housing

Higher earnings Impact Better health (Overall goal) Kids flourish

Detailed Logic Model: Housing


Logic
Get land, materials, Inputs financing, (Activities) builder, and bidi workers Houses built Mortgages Bidi workers take possession

Sources of Indicators Verification

Assumptions / Threats
Can get inputs Collaboration across players to build houses Bidi workers and their families will leave neighborhoods where their families live in order to get better housing Will stay in the house Wont sell Urban setting means transportation to work possible from anywhere Low payments free up money for other investments Previous locations density caused health problems

Outputs

Outcome (Project goal) Impact (Overall goal)

Better Housing Secure housing

Higher earnings Better health Kids flourish

Measurement
Indicators: metrics that quantify and measure specific program effects.
Choosing proper indicators is an important step in determining the overall success of the intervention. Indicators should be SMART Specific Measurable Action-oriented Realistic Timed

Goals of Measurement
Appropriate indicators to accurately capture desired info
Needs Assessment Process Evaluation Impact Evaluation

Identify issue Understand target group Why is there an issue?

Activities completed Who received program? Take-up rate

Did program affect outcomes: participant behavior, health, income, etc.?

Data Collection Methods


Reviews of official records Censuses

Field visits

Participant observations Community interviews Key informant interviews

Focus group interviews

Individual & Household Surveys (single or multiple rounds)

Housing Project

What are the relevant data that should have been collected for the housing project?

Measurement: Housing example


Logic
Land, materials, Inputs financing, builder, (Activities) and bidi workers. Have lottery Houses built Give mortgages Bidi workers take possession Better Housing More housing security

Indicators
Legal rights to land; Lottery dates; list of winners; other public facilities available. Construction matches plans? Down payments made? Who moved reasons why not. Moved in/out/ length of time in house? Repaying mortgage? Neighborhood quality? Job changes? Employed; income? Alive? Suffering health problem? Schooling completion? Education of s/d-in-law?

Assumptions / Threats
Can get inputs Collaboration across players to build houses Bidi workers and their families will leave neighborhoods where their families live in order to get better housing Will stay/ Wont sell Urban setting means transportation to work possible from anywhere Low payments free up money for other investments Previous locations density caused health problems

Outputs

Outcome (Project goal)

Impact (Overall goal)

Higher earnings Better health Kids flourish

Needs Assessment & Program Theory


High quality needs assessment and theory of change help ensure that an intervention is well-designed to achieve the intended impacts

Program Evaluation Lifecycle


Needs Assessment

Program Theory

Process Evaluation
Impact Evaluation Cost-Effectiveness Analysis

Implement program Evaluate Implementation Learn, improve program

Monitoring
Plans & Targets Inputs, Activities Outputs Outcomes

Implementation
Continuous collection of data relevant to implementation Examples: population served, resources used, services provided Aims to inform and improve an interventions design, implementation and management strategies

Process Evaluation
Plans & Targets Inputs, Activities Outputs Outcomes

Implementation
Assess whether program is implemented as designed, and extent that it reaches intended participants Identify problems Improve delivery

Confirms what program is being evaluated, provides data required to investigate variation in results.

Process Evaluation Housing Project


1987: SEWA approached government about project. 1990: 110 houses built & then torn down to make room for longer runway to make airport international. 1993: New houses built but now farther from center of city. About 72 out of 110 winners relocate. Over time, more than half move out.

Monitoring & Process Evaluation


High quality monitoring and process evaluation collects regular, reliable and relevant data to

Identify changes that create a more effective intervention (feedback loop).


Confirm the intervention being evaluated. If the program did not have an impact, suggests why. Help us understand variation in positive impact. High payoff for the effort low cost, indicates whether and when an impact evaluation is necessary

Program Evaluation Lifecycle


Needs Assessment

Program Theory
Process Evaluation

Impact Evaluation
Cost-Effectiveness Analysis

Collect Endline Data, Evaluate Impacts Use results to further improve program

Impact Evaluation
Plans & Targets Inputs, Activities Outputs Outcomes

Impact Did the program achieve its ultimate goals? Is the Program Theory aligned with the results? What are the differences between the logic model and what actually happened? How will these results feedback into future program design?

Housing Projects Impact


Economic & health outcomes unaffected. Children of winners and losers look the same in the long term. Housing is improved. Households faced a tradeoff:
Move to new colony with better housing, but be farther from friends, family, jobs, schools, OR Stay near these people but continue to face insecure housing.

Social interactions changed because of the housing program


More collective action and interaction with new neighbors. Harder to share risks when risks are more correlated

Project & Program Lifecycles


Needs Assessment Program Theory Process Evaluation Impact Evaluation Cost-Effectiveness Analysis Being precise about the assumptions allows us to assess if the program is well thoughtout and likely to have the impact we want to see. Tells us two important things: 1. Was the program implemented as designed? What unexpected things happened? Was there variation? 2. If we conduct an impact evaluation, exactly what program is being evaluated.

You might also like