Logic Models & Measuring

Program Outcomes

Kenosha County Department of Human Services

Planning & Evaluation Unit

Jennifer Madore, Coordinator

Pamela Smith, Associate
Jeff Weiss, Associate

June 25, 2004

Why measure outcomes?
3 To help programs improve services.

3 To see if programs really make a difference in the lives of

people.

3 If you can demonstrate that your efforts are making a

difference for people, you can:
ü Recruit and retain talented staff
ü Enlist and motivate volunteers
ü Engage collaborators
ü Gain support for innovative efforts
ü Win designation as a model program
ü Retain or increase funding
ü Achieve favorable public recognition
 A Logic Model is…
A picture of how a program works

Shows the relationship between what a

program invests (inputs) – what it does
(activities) - and what are the intended
results (outcomes)
The model provides a logical chain of
“If...then…” relationships:
If “x” occurs, then “y” will occur.
 Benefits of Using Logic Models

• Brings detail to broad goals

• Shows the chain of events that link inputs to
outcomes
• Identifies gaps in logic and uncertain
assumptions
• Signals what to evaluate and when
• Summarizes complex programs to
communicate with key stakeholders
What Makes Up a Logic Model?

" Situation
" Assumptions
" INPUTS
" ACTIVITIES
" OUTPUTS
" OUTCOMES
" Environment
 Logic Model “If…then…” Assumptions
ë Certain resources are needed to operate your program
ë If you have access to them, then you can use them to
accomplish your planned activities.
ë If you accomplish your planned activities, then you will
hopefully deliver the amount of service that you intended
(output).
ë If you accomplish your planned activities to the extent
intended, then your participants will benefit in specific way
(initial and intermediate outcomes).
ë If these benefits to participants are achieved, then certain
changes in organizations, communities, or systems might
occur under specified conditions (long-term outcome).
 SITUATION
Problem or need in the
community that you wish to
alleviate (e.g., teen
pregnancy, elder abuse,
juvenile delinquency,
domestic violence, etc.)
 Assumptions

~An assumption is the theory

underlying your program or service.
~Assumptions provides the connective
steps between the situation/problem
and your selected intervention(s).
~Assumptions may be based on
research that supports program
strategies (e.g., best practice).
Resources used to support program activities.

INPUTS
Staff
Money
Supplies
Volunteers
Partners
Equipment
Technology
Facility
What a program does with its inputs; the services it
provides to fulfill mission.

ACTIVITIES
Tutoring
Family Counseling
Client Advocacy
Group Therapy
Educational Workshops
Assessments
Job Training
Products of a program’s activities.

OUTPUTS

# of participants served
# of classes offered
# of training hours provided
# of meals served
# of shelter nights provided

Otherwise known as “Units of Service”

 Benefits for participants during or after their
involvement with a program.

OUTCOMES

40 students participated in weekly tutoring sessions

during the Fall 02/03 semester (output).
So What?
Students academic achievement improved (outcome).
How do you know (i.e., what is your indicator?)
80% (n=32) of students increased their math grade by at
least one letter grade from Spring 02/03 to Fall 03/04.
 TYPES OF OUTCOMES

INITIAL INTERMEDIATE LONG-TERM

Learning Action Conditions
Awareness Behavior Human
Knowledge Practice Economic
Attitudes Decisions Civic
Skills Policies Environment
Opinion Social action
Aspirations
Motivation
Questions that help clarify program
logic and identify outcomes:

¬What do you hope to accomplish by providing

services?

¬What do you want to be able to say about clients

during or after the program?

¬When you provide the services, what is the first

change you expect will happen for participants (i.e.,
initial outcome)?

¬If that change occurs, then what will it lead to (i.e.,

intermediate outcome)? Why do you want that
change to happen (i.e., long-term outcome)?
Logic Model Example - Parent Education Program
(Form 1, RFP Packet)

Activities Outputs Initial Intermediate Long-Term

Inputs
Outcome Outcome Outcome

Average of
Design
10 parents
Parent Parents
attend each
Education increase
curriculum session
Staff knowledge of
child Parents use Reduce
Funds
15 development improved rates of
Facility
(unduplicated) parenting child abuse
Supplies
Provide parents skills and neglect
Research Parents learn new
parent attend
ways to discipline
training that during the
includes 12-week
topics such period
as positive
reinforcement 12 training
and behavior sessions
management are held
once per
week for 12
weeks
Environmental Factors
Resources or barriers which potentially
enable or limit program effectiveness.

Examples of positive influences include

collaborating networks and funding.

Negative factors may include lack of

funding or other resources, political
climate, laws or regulations. Programs
have little control over environmental
factors.
Key Ideas about
Logic Models

There is no one perfect logic model for any

program—what is important is that your
logic model:
--be complete
--be consistent and make sense
--be agreed to by the key stakeholders
 Key Ideas about
Logic Models

The labeling of terms may vary in logic models, for example:

output = process objective

outcome = goal
indicator = objective
long-term goal = impact
outcome target = performance measure

Some models add or subtract some components, but the key

elements of Inputs, Activities, Outputs, and Outcomes should
be included.
Key Ideas about
Logic Models

A logic model is dynamic and should

change with:
[ experience
[ changes in resources/realities
[ changes in visions
[ changes in stakeholders
Outcome Indicators
Outcome indicators are the specific items of information
that track a program's success on outcomes.

Specifying an outcome indicator requires deciding:

1. The specific observable, measurable characteristic or

change that will represent the achievement of the
outcome; and

2. The specific statistic (e.g., number and percent

attaining outcome) the program will calculate to
summarize its level of achievement.
 Outcome Measurement Framework
(Form 2, RFP Packet)

Outcome Indicator(s) Data Source Data Collection

Restate outcomes from What measurable and From what source will How will you collect
logic model worksheet observable evidence you obtain the data the data? Who will be
(FORM 1) in this will you provide to (e.g., surveys, client responsible for
column monitor the progress records, police managing the data and
xxxxxxxxxxxxxxxxxxxx toward the outcome? reports, etc.) reporting?

Parents increase # and percentage of Retrospective Post Administer at final

knowledge of child parents who know phases
Survey session
of child development and
development (initial) developmental milestones

Parents learn # and % of parents that Retrospective Post Administer at final

new ways to identify at least two
Survey session
discipline (initial) appropriate discipline
techniques

Parents use # and % of parents that Telephone survey Administer 3 months

improved parenting report using at least 2
after final session
skills (intermediate) appropriate disciplinary
skills

Reduce rates of child # and % of participants DCFS record Contact DCFS at 3, 6

abuse and neglect with no or fewer report
review and 12 month
(long-term) cases of child abuse or
neglect intervals
 Examples of Outcomes & Indicators
(Anger Management Program)

Outcomes Indicators
Clients increase awareness 90 of 100 (90%) of clients will
of how anger affects their life. identify at least one negative
(Initial) effect of anger in their life.
Clients learn new coping 80 of 100 (80%) of clients will
skills to deal with their anger. identify at least one new
(Initial) coping skill.
Clients use new coping skills. 80 of 100 (80%) of clients will report
(Intermediate) using at least one new coping skill
to deal with their anger.

Clients will have an improved 70 of 100 (70%) of clients will report

quality of life. (Long-Term) a quality of life improvement upon
completion of the program.
 Examples of Outcomes & Indicators
(Injury Prevention Program for Seniors)

Outcomes Indicators

Seniors increase knowledge 95 of 100 (95%) of seniors will

of and identify injury risk identify at least two risk factors
factors within their home. within their home.
(Initial)

Seniors modify home to 80 of 100 (80%) of seniors report

decrease the number of injury that at least two home
risk factors. (Intermediate) modifications have been made
two months after the program.

The rate of household injuries Incidence of household injuries

among seniors decreases. among seniors decreases by 30%
(Long-Term) over a two-year period.
 Data Collection

Data Sources: Data Collection Methods:

* Participants * Survey
* Parents * Interview
* Teachers * Observation
* Referral Source * Pre and post test
* Family Members * Focus Groups
* Employers * Other Agency Records

This is not an exhaustive list.

 Data Reporting Methods
Prepare reports that tell your story. It provides evidence and
documentation of program accomplishments for various
audiences and key stakeholders.

BASIC STEPS
Step 1. Analyze data by key characteristics
age, gender, race, program participation

Step 2.
2 Provide explanation of the findings
what does the data tell us?

Step 3.
3 Present data in clear, understandable form
use short sentences, bullets, bold and
italics to highlight important points
 Sample Data Reporting Methods

Pre/post • Baseline and exit

comparisons

Client Follow-Up • Progress reports 3,6

or 12 months after
program completion

Client Satisfaction • Quality rating scales

Survey 1=Poor; 2=Fair;
3=Good; 4=Excellent
Types of Data Reporting Formats

Tables
Use to show frequencies

Charts
Use to show multiple comparisons

Graphs
Use to provide a visual illustration

Executive Summary
Use to report key findings to stakeholders
 Sample Visual Illustrations
Table, Pie Chart & Graph
Youth Employment Program
June-August 2003
# of Job # of Job
# of Youth Recruited
Referrals Placements
75 63
82
91.4% 84.0%

4th Qtr 1st Qtr

13%
Gender of Program Participants
13% 2nd Qtr
17%
50

40

Number
30

20

10
3rd Qtr
0
57% 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
Percentage of Enrollments by Quarter (Y2003)
Employee Recreation Program Males Females
Additional Resources

http://national.unitedway.org/outcomes/resources/mpo/

http://www.uwex.edu/ces/pdande/evaluation/pdf/LogicETP.pdf

http://www.cleggassociates.com/Resources/LogicModel/index.asp

http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf

http://www.co.kenosha.wi.us/DHS