AEA Annual Meeting

November 2009

Logic Models and Systems Models: Two Sides to the Same Coin?
Presentation Notes
Jan Noga, Pathfinder Evaluation and Consulting
Meg Hargreaves, Mathematica Policy Research

Title Slide: Introduction

• This think tank grew out of conversations we have been having as program co-chairs
of the Systems in Evaluation TIG
• Among evaluators, discussion that has created dichotomy around systems models and
logic models – which is better/more accurate/more informative, etc. – essentially an
either/or debate at times
• We feel this is a false dichotomy
• Are they, instead, complementary approaches? When used together, potential to help
evaluators describe not only context, process, and relationships, but also the
relationships between investments, activities, and results – the program theory
• each contributes value to an evaluation
• systems models and logic models are not the same thing so much as
different perspectives
• provide different types of insight into programs and processes that inform
evaluation design, implementation, analysis, and reporting
• Think tank is a first pass at sharing our thoughts around modeling and its uses in
evaluation

Slide 1: What Do We Mean by System?

• In bullet 1 – the common definition of a system, what you will find from many
sources
• In general, a good definition, although should note that the “complex and
unified whole” is not always obvious to the viewer – sometimes our job to
identify how the whole is unified
• Also, Meg notes that “specific purpose” is not necessarily consistent with
her definition of a system; systems have purpose, but “specific” not
necessarily implied or present
• Four defining characteristics that must be present in a system; otherwise, might
simply be a collection of elements
• Perspectives and relationships are critical defining characteristics
• Elements of a system have relationships to each other – may be functional
or structural

© 2009, Janice Noga and Margaret Hargreaves

 Logic Models and Systems Models 2

Slide 2: Why model?

• A model is simply a visual representation of a system, program, intervention, etc.
• Provides us with a tool to articulate processes, untangle complications, clarify
complexity
• In evaluation, frequently use models to articulate program theory and program logic
(logic modeling)
• Integrating systems models into evaluation allows us to situate a program or
intervention within the greater context as defined by other elements such as larger
organization, legislative/political context, funding, stakeholder relationships, etc.

Slide 3: What do models tell us?

• So, what can we learn from systems and logic models?
• Systems models
• Provide insight into relationships that exist between the different parts of a
systems, between the parts of a system and the system itself, and between
the system and the greater environment/context within which it functions
• Attend to boundaries, relationships, diversity, and patterns of behavior,
including nonlinear feedback and emergent patterns
• Consider the whole, its parts, and its context
• Attempt to identify what is working, what needs to be changed, and what
the challenges are to making such changes
• Should note that systems exist within systems; models can also help us
define nestedness – within a school for example
o School building is system within larger environment defined by
district
o Within building are subsystems defined by grade levels or content
areas
o There are also subsystems defined by classrooms
o Often in educational evaluation, evaluators must look at students
within classrooms, within buildings, within districts, within a state
• Argument against – systems models operate at scale that is much larger
than single program or intervention, best suited to describing program
context, process, and relationships
• Can get complicated if too detailed or trying to define too many things at
once
• Logic models
• Logic models considered more appropriate at specific program or
intervention level
o better suited to describing the sequence of events thought to bring
about benefits or change over time and to portraying chain or
reasoning linking inputs and strategies to outcomes and impact
• Typically used to define a sequence of events and relationships between
the elements of an intervention or program
o In US, frequently see 4-Box Model defining inputs, strategies,
outputs, and outcomes

© 2009, Janice Noga and Margaret Hargreaves

 Logic Models and Systems Models 3

o Model may also include anticipated long-term impacts as a result

of specific intervention/program
• For evaluators, logic models help define the difference between where the
program is now and where it intends to go; guides us in developing a data
collection methodology
• Logic models can also define relationships between stakeholders,
investments, activities, and results – but this is not their focus and they do
so within the tightly prescribed boundaries of the intervention or program
being evaluated/implemented
• Often, taken as having an implied linearity that can shroud process and
flow – program logic may actually be realized in stages or in recursive
manner using straight-line models

Slide 4: Program Theory

• Systematic process for defining what a program must do to achieve desired goals,
anticipated impacts, and the process by which goals and impacts are realized (Chen,
2005)
• Program design and implementation are based on a set of explicit and implicit
assumptions held by stakeholders about what is needed to solve an identified problem
and how the problem is likely to be mitigated by a set of specific actions
• Chen (2005) defines program theory as being simultaneously descriptive and
prescriptive, with a resulting focus on identifying action-oriented, rather than causal,
explanations of program assumptions, processes, and activities.

Slide 5: Modeling: The Big Picture

Action and Change models as exemplified by Chen – way to show how models can complement
each other
• Action model defines elements of program and relationships between them (similar to
system model)
• Change model defines elements of specific intervention being implemented by
program

Slide 6: Two Types of Logic Models

• Logic models based on two elements of program theory
• Theory of action
o very basic definition of key program elements
o a first step in defining the overall program logic
o “Doing X with these participants will cause Y outcomes/impacts”
• Theory of change
o the specific details for each element in the theory of action
o “What inputs/resources will support which strategies with which
participants to produce these specific outcomes/impacts”

© 2009, Janice Noga and Margaret Hargreaves

 Logic Models and Systems Models 4

Slide 7: Change (aka Logic) Models

• Seek to identify descriptive assumptions defining the causal processes underlying a
program’s ability to successfully impact participants
• The model we most commonly see – could be considered the road map for program
design, implementation, and evaluation
• Weaknesses include implications of cause-effect linearity and focus on individuals in
data collection and analysis

Slide 8: Action (aka System Models)

• Program evaluators, on the other hand, are increasingly focusing on the role of action
models in clarifying program theory and evaluating program, rather than individual,
outcomes
• Action models focus on prescriptive assumptions regarding program components and
activities that stakeholders view as essential for program success; seek to answer
questions such as:
o What are the crucial elements of the program?
o What organizational structures and processes are necessary to deliver
services?
o Who is the target audience for these services?
• Prerequisites for change models, establishing the base context within which a change
model can then be implemented
• Think of action model as a basic system model – defines elements, boundaries,
relationships for specific program being evaluated
o One of many systems perspectives that can be used in defining program

Slide 9: Why Do Both?

• Defining and mapping the system takes time and may not be something clients want
to see taking up dollars in a budget
• Still, important to map both the system within which a program operates AND the
logic of the program/intervention you are evaluating
• Why?
• Tempting to jump straight to evaluation of the change model without first taking a
closer look at processes defined within the action model
• As a result, many programs attempt to tie program outcomes to inputs and strategies
by evaluating individual impact without first examining the systemic, policy,
organizational, and implementation processes of the action model that are necessary
for success
• If the elements of the action model are not interacting appropriately, it may be
impossible to effectively implement the transformative processes defined by the
change model
• Thus, it is critical that the prerequisite contextual and organizational elements of the
action model be examined prior to evaluating participant impact

© 2009, Janice Noga and Margaret Hargreaves

 Logic Models and Systems Models 5

Slide 10: Questions to ponder

1. What are strengths and weaknesses of each?
2. How can the two approaches to modeling be integrated to provide a framework for
understanding both the program and the systems in which it functions?
3. How can they be used to inform an evaluation design?
4. How can you use models to capture critical elements of process, context, content, and
program theory in your own work?

Sample Systems Model: Organizational Change

Ziegenfuss’s (2002) adaptation of the Kast and Rosenzweig model is a comprehensible and
helpful model for conceptualizing the organization as an integrated system. The model defines an
organization as an “organized, unitary whole composed of two or more interdependent parts,
components, or subsystems and delineated by identifiable boundaries from its environmental
suprasystem” (Kast and Rosenzweig, 1984, pg. 103).

The model examines five major organizational subsystems: (1) the culture and values in which
the program exists, (2) psychosocial influences, (3) the technology and resources available, (4)
structural components, and (5) management and leadership within the program. This
examination is done in the context of the ever-present environmental forces that influence
programmatic and organizational decisions.

Managerial / Leadership
The leadership subsystem is the command and control subsystem in the organization.
This subsystem plans, directs, and controls the processes of the organization and relates
the organization to the environment. The subsystem integrates all other subsystems in
such a manner that they are unified to achieve organizational goals.

Culture and values

The culture and values subsystem speaks directly to the organization’s customary beliefs,
rituals, artifacts, rules of behavior, social norms, goals and values that define the
organizational culture, and to a large extent considers those basic assumptions that go
unspoken but are attached to the organization’s identity.

Psychosocial
The psychosocial subsystem directs our attention to the social system in the organization.
This subsystem considers the important individual characteristics, roles, relationships,
and the powerful informal relationships that affect organizational behaviors. It speaks to
the organizational climate in which employees and managers act out their roles.

© 2009, Janice Noga and Margaret Hargreaves

 Logic Models and Systems Models 6

Technology
The technology subsystem defines the manner in which tasks are completed and specific
goals are accomplished with special sets of skills and knowledge. This subsystem relates
to specific resources, techniques, physical structure, and equipment that are needed in the
organization to produce the services and products.

Structure
The structure subsystem addresses the formal structures that divide and define how the
tasks are completed and the goals are met. This subsystem is defined by the
organizational charts, committees, job descriptions, policies and procedures, and patterns
of authority.

Envi ronm ent

Cul ture and Struc ture

Val ues
Manage ri al

Psyc hosocial Technolo gy

Envi ronm ent

© 2009, Janice Noga and Margaret Hargreaves

 Logic Models and Systems Models 7

Sample Systems Model: 4Cs Model for Systemic Change

Senge (2000) has defined educational systems as holistic and ecological, consisting of
interdependent elements that work separately and together toward a common purpose. He
defines a system as a ”perceived whole whose elements ‘hang together’ because they continually
affect each other over time and operate toward a common purpose” (Wagner, 2006, p. 97).

The Change Leadership Group at Harvard University’s Graduate School of Education, Wagner
et al (2006) extended Senge’s ideas to develop a systemic model for thinking about change in
schools across four interdependent, overlapping domains: (1) competencies, skills, and
knowledge of adults; (2) conditions of time, space, and resources that support learning; (3)
culture and conditions of leadership; and (4) the larger environmental and policy context in
which schooling takes place.

Context
understanding global, state, and
community realities

Culture Conditions
of classrooms, of learning and
schools, and teaching
districts
Student
learning

Competencies
of adults in schools

© 2009, Janice Noga and Margaret Hargreaves

 Logic Models and Systems Models 8

Competencies:
• The repertoire of skills and knowledge of adults that impacts and influences
student learning
• Building these competencies may engage other parts of the system at different
times and in different ways
• Not limited to classroom teachers

Conditions:
• The external architecture surrounding student learning.
• Represents visible arrangements and allocations of time, space, and resources
that support or hinder teaching and learning.
• May also include explicit expectations around roles and responsibilities,
student outcomes, laws and policies, and contracts.

Culture:
• The invisible meanings and mindsets that are held individually and
collectively throughout the system.
• The shared beliefs, values, assumptions, expectations, and behaviors related
to:
o Students and learning
o Teachers and teaching
o Instructional leadership
• The quality of relationships within and beyond the school.

Context:
• The social, historical, and economic context in which the work of schools
takes place.
• The worlds from which students come and those for which they must be
prepared.
• The demands and expectations, formal and informal, of the larger
organizational systems within educators work.

© 2009, Janice Noga and Margaret Hargreaves

 Logic Models and Systems Models 9

Sample Logic Model: Integrating A Systemic Perspective using Program Logic Models

In 2005 the Ohio Department of Education created a Technical Advisory Panel (TAP) to
recommend a set of performance measures for its Regional School Improvement Teams (RSITs),
contractors who provide school improvement services to schools and districts. The Department’s
purposes were to clarify the RSITs’ responsibilities for professional development and technical
assistance, and to ensure accountability for services as well as documentation of the impact of
services.

University-based evaluators were contracted to assist the TAP over a period of four months in
the creation of these measures. The lion’s share of the effort was turned to developing common
understanding across team members of the work of school improvement. After identifying
intended outcomes and impacts of school improvement, four logic models were created using a
framework based on the Balanced Scorecard for accountability, one each for externally focused
processes, internally focused processes, continuous improvement processes, and services
rendered. Implicit and explicit assumptions, as well as facilitators and barriers, were identified.
Finally, indicators of success were identified for the outputs in the logic models, leading to a set
of agreed upon performance measures.

As the Department and the regional teams worked to implement the performance measures for
school improvement, they found, in addition to accountability, that they were better able to
clarify roles, the contingencies and interdependencies of their work, and alignment of timelines,
budgets, and products. They also found that the work on performance measurement helped
inform understanding of the intersections among various centers in the Department and new
pathways toward vertical and horizontal integration and alignment.

© 2009, Janice Noga and Margaret Hargreaves

 Logic Models and Systems Models 10

Field Relations Regional Accountability Systems Project

External Operational Perspective

 How do our funders view us?
 To whom are we accountable and for
what?

[Budgets, revenue, contractual

accountability]

Service Perspective Internal Operational Perspective

 What must we be doing?
 How do our customers see us?
 How should we be doing it?
 Are we meeting their needs?
[Accurate and timely delivery of
[Responsiveness, timeliness, service
service; effective systems and
quality]
processes, sustainability]

Continuous Improvement Perspective

 How can we continue to improve and

provide value?

[Support for innovation and learning; staff

skills and training, use of technology,
continuous improvement]

Sources: Brown (1998). Accountability and performance measurement. http://www.accglobal.com/publications.;

Kaplan and Norton (1992). The balanced scorecard. Harvard Business Review (Jan/Feb).

© 2009, Janice Noga and Margaret Hargreaves

 Logic Models and Systems Models 11

Field Relations Regional Accountability Systems Project

Program Logic Model
Inputs Activities/ Outcomes
Outputs Impacts
Priorities Resources Strategies Short-term Long-term

• High quality
Regional PD ODE services that Priority districts & 1. Curricula in schools 1. Students
and TA • Performance reflect district and community schools are aligned with receive high
agreement Service regional needs, are able to use academic content quality
coordinators will Perspective
• Initiatives priorities, and data to: standards instruction
provide the
• Communication timelines aligned with
following to their 2. Educators develop academic
infrastructure 1. Build and maintain
regions: • Professional capacity to plan and use effective content
development • Fiscal decisions strategically teaching and standards
1. School • Technology reflect regional leadership
improvement infrastructure needs and 2. Implement school strategies aligned 2. Students have
• Legislation External priorities improvement with academic the right
support
Perspective • Service providers processes and content standards conditions and
through:
meet assurances research-based motivation for
• Delivery of OFR coordinating
for identified practices 3. Schools provide learning
professional team members & services
development effective
regional service • Timely delivery of intervention 3. Students
• Technical 3. Function more
providers: services that programs demonstrate
assistance effectively/
• Resources Internal reflect regional high levels of
efficiently for
• Staffing Perspective needs and achievement
2. State increased student
• Partnerships priorities performance
initiatives • Skills and
• Professional expertise • OFR coord. team
development members and
• Mandates providers have
• Other policy necessary skills &
directives Continuous
Improvement expertise
Perspective • Data- based
decision making
guides all
processes

© 2009, Janice Noga and Margaret Hargreaves

 Logic Models and Systems Models 12

Field Relations Regional Accountability Systems Project Program Logic Model: External Perspective

Inputs Activities/Strategies Outputs

ODE
• ODE funding by service; contractual
performance agreements
• State Initiatives
Review fiscal allocation processes for alignment with Regional input into the fiscal allocation
regional needs and priorities process that considers regional needs
OFR coordinating team members and and priorities
regional service providers:
• Resources
• Partnerships

Enter into performance agreement with OFR Signed performance agreement

ODE
• OFR facilitates communication across
RSITs and between ODE program Knowledge base of resources and
offices and RSITs Broker resources and services with providers to
service providers that meet regional
• Performance agreements address regional needs
needs

OFR coordinating team members and Develop partnerships with providers to address
Informal and formal agreements that
regional needs
regional service providers: outline services to be provide
• Performance contracts
• Partnerships

Processes are in place to identify and

Develop process and communication structure for
communicate assurances for service
accountability for service delivery
delivery

OFR coordinating team members and

regional service providers: Develop systematic processes for strategic planning, Protocols are developed for planning,
• Resources monitoring, and reporting results/impact monitoring, and reporting
• Partnerships

Note: Solid lines denote direct influence or connection between elements; broken lines denote an indirect influence

© 2009, Janice Noga and Margaret Hargreaves

 Logic Models and Systems Models 13

Field Relations Regional Accountability Systems Project Program Logic Model: Internal Perspective

Inputs Activities/Strategies Outputs

ODE
• Facilitate development of systematic Develop a systematic process for regional needs
process for needs assessment Process for regional needs assessment
assessment

Conduct regional needs assessment and develop

Regional plan identifies common needs
regional plan to deliver services that address regional
that drive services within region
needs and priorities

Shared understanding among OFR’s

coordinating team members and
OFR coordinating team members and providers around purpose, roles, and
regional service providers: responsibilities
• Resources Identify core work processes, structures, and functions
• Service partnerships and determine capacity needed to sustain them Processes, structures, and functions in
• Skills and expertise place to build capacity within OFR’s
coordinating team members and
ODE regional services providers to
• Consistency of direction and information accomplish core work processes
• Coherence in school improvement model
through alignment of programs and
initiatives
• Standards defining efficient and effective
service Articulated process for allocating
resources and services
Develop systems and processes to allocate and
manage resources for the provision of services and to
assure that there is adequate capacity to efficiently and
effectively deliver services
Accountability identified for carrying out
individual components of regional
service plan

Note: Solid lines denote direct influence or connection between elements; broken lines denote an indirect influence

© 2009, Janice Noga and Margaret Hargreaves

 Logic Models and Systems Models 14

Field Relations Regional Accountability Systems Project Program Logic Model: Continuous Improvement
Perspective

Inputs Activities/Strategies Outputs

OFR coordinating team members and Work force demonstrates knowledge and
regional service providers: Work with OFR to ensure work force knowledge in
skills in school improvement focal
• Resources areas*
school improvement processes is present and is
• Service partnerships
aligned with capacity needs
• Skills and expertise Skill set of OFR’s coordinating team
members matches regional needs

ODE
• Establish quality criteria for workforce
knowledge and skills
• Facilitate ongoing professional
development that meets identified
service provider needs
• Provide access to analyzed data, Assessment, evaluation, and performance
evaluations Create processes for data-based decision making,
measurement are ongoing and
including systematic, ongoing assessment and
embedded in daily practice and guide
evaluation of OFR coordinating team members and
processes, service, and strategic
providers’ knowledge, skills and practice
planning
ODE
• Initiate supporting infrastructure to
facilitate systematic, ongoing
assessment and evaluation of processes
and services

ODE
• Initiate statewide and regional
technology infrastructure to support core Specific opportunities are created for
Create opportunities for collaboration and networking
school improvement services sharing knowledge about research-
around research-based practices for school
• Provide leadership in developing based practices for school
improvement
collaborations and networking around improvement
research-based practices for school
improvement

∗ School improvement focal areas: data analysis, research-based practices, focused planning, resource management, monitoring and implementation, and high quality professional
development delivered to school/district site

© 2009, Janice Noga and Margaret Hargreaves

 Logic Models and Systems Models 15

Field Relations Regional Accountability Systems Project Program Logic Model: Service Perspective

Inputs Activities/Strategies Outputs

Approved district needs assessments

Assist priority districts to review existing district needs
assessments

OFR coordinating team members &

regional service providers:
• Resources
• Partnerships Completion of partnership agreements for
Develop partnership agreements with priority districts service that are aligned with needs
and community schools with services customized to
district base on needs assessment Collaborative relationships between
priority districts and regional service
providers established

Deliver information, resource and referral services to Districts receive information, resource, and
districts referral that meets their needs

OFR coordinating team members &

regional service providers:
• Resources
• Partnerships
• Skills and expertise

ODE
• Statewide and regional technology
infrastructure to support core business Evidence of progress made toward the
Provide technical assistance to priority districts for
technical assistance/ improvement
school improvement planning as identified in the
goals identified in the district
partnership agreements and regional plan
partnership agreements; customer
satisfaction

© 2009, Janice Noga and Margaret Hargreaves

 Logic Models and Systems Models 16

Field Relations Regional Accountability Systems Project Program Logic Model: Service Perspective

Inputs Activities/Strategies Outputs

Development and delivery of professional

ODE & Developers development aligned with academic
content standards, research-based
• Initiatives developed in timely manner
Create and/or provide professional development school improvement strategies, and
• Development of integrated/aligned
opportunities in priority districts that reflect research- identified district needs
timelines for deployment of state
based quality standards and are aligned with district
initiatives
needs, timelines, and priorities as identified in Educators in priority districts receive PD
• Development of content that meets
regional and district partnership agreements aligned with academic content
criteria for high quality professional
standards, research-based teaching
development
and school improvement strategies, and
• Coordination of initiatives across
identified district needs
program areas

ODE & OFR Coordinating Team

Members Priority districts have access to state
• Provide definition for how an activity or initiatives
program becomes a state initiative
• Define process for deployment of state Deploy state initiatives to priority districts in manner that
initiatives that identifies key roles and meets predefined ODE criteria for deployment of
responsibilities and defines each specific initiative
accountability accordingly Deployment meets contractual obligations
for each specific initiative

© 2009, Janice Noga and Margaret Hargreaves