0% found this document useful (0 votes)
95 viewsA Practical Guide For Policy Analysis
Uploaded by
IrinaCopyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
0% found this document useful (0 votes)
95 viewsA Practical Guide For Policy Analysis
Uploaded by
IrinaCopyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
/ 114
A Practical Guide for Policy Analysis:
The Eightfold Path to More Effective
Problem Solving
Eugene Bardach
Richard and Rhoda Goldman School of Public Policy
University of California at Berkeley
tote
Soptewhasr 2OOG
Laxtu yes
CHATHAM HOUSE PUBLISHERS
SEVEN BRIDGES PRESS, LLO
NEW YORK + LONDON Seven Bridges Press, LLC
195 Fifth Avenue
New York, NY 10010
Copyright © 2000 by Chatham House Publishers of Seven Bridges Press, LLC
Al rights reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted
in any form or by any means, electronic, mechanical,
Photocopying, recording, or otherwise, without the prior
permission of the publisher.
Publisher: Robert J. Gormley
Managing Edivor: Katharine Miller
Gover and Interior Design: Andrea Batash Design
Composition: Bytheway Publishing Services
Printing and Binding: Versa Press, Inc.
Library of Congress Cataloging-in-Publication Data
Bardach, Eugene.
A practical guide for policy analysis : the eightfold path to more
effective problem solving / Eugene Bardach.
P. cm,
Includes bibliographical references,
ISBN 1-889119-29-6
* Policy sciences. 2. Decision making, 3. Problem solving, 1.
Tide :
H97.B37 2000
320’.6—dcor 99-050621
Manufactured in the United States of America
0987654352 Contents
Preface
Introduction
Policy Analysis: More Art than Science
The Eightfold Path
Iteration Is Continual
Some of the Guidelines Are Practical, but Most Are Conceptual
The Concepts Come Embedded in Concrete Particulars
Your Final Product
The Spirit of the Eightfold Path
Part i. The Eightfold Path
4. Define the Problem
Think of Deficits and Excesses
The Definition Should Be Evaluative
Quantify If Possible
Conditions That Cause Problems Are Also Problems
Missing an Opportunity Is a Problem
Common Pitfalls in Problem Definition
Defining the solution into the “problem”
Be skeptical about the causal claims implicit in diagnostic problem
definitions
Hterate
2, Assemble Some Evidence
Think Before You Collect
The value of evidence
Self-control
Do a Literature Review
xi
xiii
xiv
xiv
xiv
xv
xv
xv
xvi
WU e NN
N
10 A PRACTICAL GUIDE FOR POLICY ANALYSIS
Survey “Best Practice”
Use Analogies
Start Early
Touching Base, Gaining Credit
Freeing the Captive Mind
ty, Brokering Consensus
3. Construct the Alternatives
Start Comprehensive, End Up Focused
Model the System in Which the Problem Is Located
Reduce and Simplify the List of Alternatives
Design Problems
A Linguistic Pitfall
4, Select the Criteria
Apply Evaluative Criteria to Judging Outcomes, Not Alternatives
Criteria Selection Builds on Problem Definitions—and Continues
Evaluative Criteria Commonly Used in Policy Analysis
Efficiency
Equality, equity, faimess, “Justice”
Freedom, community, and other ideas
Weighting Conflicting Evaluative Criteria
The political process takes care of it
The analyst imposes a solution
Practical Criteria
Legality
Political Acceptability
Robustness and improvability
Criteria in Optimization Models
Linear programming
Improving linguistic clarity
5. Project the Outcomes
Projection = Model + Evidence
Attach Magnitude Estimates
Break-Even Estimates
The Optimism Problem
Scenario writing
The other-guy’s-shoes heuristic
Undesirable side effects
The ethical costs of optimism
The Outcomes Matrix
Linguistic Pitfalls
6. Confront the Trade-Offs
Commensurability
10
14
Bot
at
12
12
12
14
7
19
19
19
20
20
23
24
26
27
28
29
29
32
35
36
37
38 CONTENTS
Break-even analysis revisited
Without Projecting Outcomes, There’s Nothing to Trade Off
Simplify the Comparison Process
7. Decide!
The Twenty-Dollar-Bill Test
8, Tell Your Story
The New York Taxi Driver Test
You, Your Client, and Your Audiences
What Medium to Use?
Your Story Should Have a Logical Narrative Flow
Some Common Pitfalls
Following the Eightfold Path
Compulsive qualifying
Showing your work
Listing without explaining
Style
Report Format
Table format
References and sources
Memo Format
The Sound Bite and the Press Release
Part Il. Gathering Data for Policy Research
in the Beginning
Locating Relevant Sources
Documents and People
People lead to people
People tead to documents
Documents lead to documents
Documents lead to people
Secondhand Information
Multiple Sources of Firsthand Information
The Search for Sources and the Search for Knowledge
Gaining Access and Engaging Assistance
Getting an Appointment
Cultivating Access
Exhausting Access
Acquiring and Using Leverage
Energy Plus Direction Equals information
The Defensive Informant
vii
39
39
40
41
41
AL
42
42
43
44
47
50
50 vill A PRACTICAL GUIDE FOR POLICY ANALYSIS
Protecting Credibility
Policy Research Aims at Approximate Truths
The Risks of Premature Exposure
Strategic Dilemmas of Policy Research
Part fll. “Smart (Best) Practices” Research:
Understanding and Making Use of What Look like
Good Ideas from Somewhere Else
Have Realistic Expectations
“Smart Practices”
Free Lunches
Breaking Loose from Conventions and Assumptions
Characterizing and Observing the “Practice”
Characterizing the Elements of a Practice
Distinguish between elements that are “essential” and those that
are “supportive”
Distinguish between “essential” and “optional” elements
Distinguish between the functions each of the elements performs
and the methods used to perform them
Allowing for Variation and Complexity
Characterization should be generic and flexible, not prescriptive
and overly precise
Characterization of the essential elements of a practice is not
necessarily simple; it could be complex
Specimens of a smart practice in the real world look rather
different from one another and require careful interpretation
Generic Vulnerabilities
But Will tt Work Here?
The Target Context
The Source Contexts
Back to the Eightfold Path
Appendix A. Things Governments Do
|. Taxes
A. What You Might Do
B. Why You Might Do It
H Regulation
A. What You Might Do
B. Why You Might Do it
Ml. Subsidies and Grants
A. What You Might Do
67
68
69
71
7
72
72
76
77
de
79
81
82
84
87
87
87
88
88
88
89
89 CONTENTS
B. Why You Might Do It
incentive effects
Wealth effects
Some design problems
WV, Provide a Service
A. What You Might Do
8, Why You Might Do it
V. Agency Budgets
A. What You Might Do
8, Why You Might Do It
Vi, information
‘A. What You Might Do
8, Why You Might Do It
VII. Modify Structure of Private Rights
A. What You Might Modify or Create
B, Why You Might Do It
Vill. Modify Framework of Economic Activity
A. What You Might Do
B. Why You Might Do It
Supporting more government intervention
Supporting less government intervention
1X. Education and Consultation
‘A. What You Might Do
8. Why You Might Do It
X. Financing and Contracting
‘A. What You Might Do
B. Why You Might Do It
XI, Bureaucratic and Political Reforms
‘A. What You Might Do
B. Why You Might Do It
Appendix B. Semantic Tips: A Summary
Defining the Problem
Assembling the Evidence
Constructing the Alternatives
Selecting Criteria
Projecting Outcomes
Confronting Trade-offs
Doing Smart Practices Research
89
90
go
90
90
90
91
91
91
91
g1
91
92
92
92
93
93
93
93
94
94
94
94
94
94
97
97
98
98
99
100
101
101 Preface
THIS IS. A HANDBOOK of concepts and methods for working your way through
a policy analysis. The presumed user is a beginning practitioner preparing to
undertake a policy analysis. I have developed the general approach and many of
the specific suggestions over twenty-five years of teaching policy analysis workshops
to first- and second-year graduate students at the Richard and Rhoda Goldman
School of Public Policy, University of California, Berkeley. I have also found this
handbook useful in teaching an undergraduate introduction to public policy and
for executive education groups.
The handbook assumes a familiarity with basic economic concepts, including
those having to do with market failures (including market imperfections). Ic is
not meant to stand alone but should be used in conjunction with other sources.
Five of the best textbooks in policy analysis, which amplify points in this handbook,
are
Edith Stokey and Richard Zeckhauser, A Primer for Policy Analysis (New
York: Norton, 1978).
David L. Weimer and Aidan R. Vining, Policy Analysis: Concepts and
Practice (Upper Saddle River, N.J.: Prentice Hall, 1999).
Robert D. Behn and James W. Vaupel, Quick Analysis for Busy Decision-
Makers (New York: Basic Books, 1982).
Lee S, Friedman, Microeconomic Policy Analysis (New York: McGraw-
Hill, 1984)—revised edition forthcoming,
Duncan MacRae Jr., and Dale Whittington, Expert Advice for Policy
Choice: Analysis and Discourse (Washington, D.C.: Georgetown Univer-
sity Press, 1997).
I wish to acknowledge the patience of and the helpful reactions from all the
students and friends who have made use of this handbook, especially those who
put up with its earlier versions. Special thanks are due to Robert Behn, Sandford xii A PRACTICAL GUIDE FOR POLICY ANALYSIS
Borins, Jose Canela-Cacho, Hank Dempsey, David Dery, John Ellwood, Lee
Friedman, David Garcia-Junco Machado, Nina Goldman, David Kirp, Jake Lavin,
Leo Levenson, Martin A. Levin, Duncan MacRae, Carolyn Marzke, John Mende.
loff, Michael O'Hare, Steven Page, Bric Patashnik, Beryl Radin, Andres Roemer,
Larry Rosenthal, Mark Sabean, Eugene Smolensky, David Weimer, and Mare
Zegans. Introduction
POLICY ANALYSIS is a social and political activity. True, you take personal
moral and intellectual responsibility for the quality of your policy-analytic work.
But policy analysis goes beyond personal decision making. First, the subject matter
concerns the lives and well-being of large numbers of our fellow citizens. Second,
the process and results of policy analysis usually involve other professionals and
interested parties: it is often done in teams or officewide settings; the immediate
consumer is a “client” of some sort like a hierarchical superior; and the ultimate
audience will include diverse subgroups of politically attuned supporters and
opponents of your work. All of these facts condition the nature of policy-analytic
work and have a bearing on the nature of what is meant by quality work,
‘A policy analyst can work in any number of positions. Once upon a time,
the term implied someone rather wonkish who worked in a large government
bureaucracy serving up very technical projections of possible policy impacts for
one or more policy alternatives to some undersecretary of planning. No longer.
Policy analysts help in planning, budgeting, program evaluation, program design,
program management, public relations, and other functions. They work alone,
in teams, and in loose networks that cut across organizations. They work in
the public, nonprofit, and for-profit spheres. Although their work is ideally
distinguished by transparency of method and interpretation, the analysts them-
selves might explicitly bring to their jobs the values and passions of advocacy
groups as well of “neutral” civil servants. The professional networks in which
they work might contain—in most cases, do contain—professionals drawn from
law, engineering, accounting, and so on, and in those settings the policy-analytic
point of view has to struggle for the right to counter—or better yet, synthesize—the
viewpoints of the other professionals, Although policy-analytic work products
typically involve written reports, they may also include briefings, slide presenta-
tions, magazine articles, and TV interviews. The recipients of these products may
be broad and diffuse audiences as well as narrowly construed paying clients or
employers. The advice in this handbook is directed both to policy analysts in xIv A PRACTICAL GUIDE FOR POLICY ANALYSIS
practice and to students and others who, for whatever reasons, are attempting to
ook at the world through the eyes of a practitioner,
Policy Analysis: More Art than Science
Policy analysis is more art than science, It draws on intuition as much as method,
Nevertheless, given the choice between advice that imposes too much strucere
on the problem-solving process or too little, most beginning practitioners quite
reasonably prefer too much. I have therefore developed an approach I ell the
Eightfold Path. The primary utility of this steuctured approach is that it reminds
you of important tasks and choices that otherwise might slip your mind; its
primary drawback is that, taken by itself, it can be mechanistic
The Eightfold Path
Define the Problem
Assemble Some Evidence
Construct the Alternatives
Select the Criteria
Project the Outcomes
Confront the Trade-Offs
Decide!
Tell Your Story
These steps are not necessarily taken in precisely this order, nor are all of
them necessarily significant in every problem. However, an effort to define the
Problem is usually the right starting place, and telling the story is almost inevitably
the ending point. Constructing alternatives and selecting criteria for evaluating
them muse surely come toward the beginning of the process Assembling some
evidence is actually a step chat recurs throughout the entire process and it applies
Particularly to efforts to define the problem and to project the oweoomes of the
alternatives being considered,
Iteration Is Continual
The problem-solving process—being a process of trial and error-—is iterative, so
that you usually must repeat each of these steps, sometimes more than un INTRODUCTION xv
The spirit in which you take any one of these steps, especially in the earliest
phases of your project, should be highly tentative. As you move through the
problem-solving process, you will probably keep changing your problem defini-
tion, your menu of alternatives, your set of evaluative criteria, your sense of what
evidence bears on the problem, and so on. With each successive iteration you
will become a bit more confident that you are on the right track, that you are
focusing on the right question, and so on, This can be a frustrating process, but
it can also be rewarding—provided you can learn to enjoy the challenge of search,
discovery, and invention.
Some of the Guidelines Are Practical, but Most Are Conceptual
Most of the concepts used will seem obvious, However, there are exceptions.
First, technical terms are sometimes employed. Second, some commonsense terms
may be used in a special way that strips them of certain connotations and pethaps
imports others. For the most part, all these concepts will become intelligible
through experience and practice.
The Concepts Come Embedded in Concrete Particulars
In real life, policy problems appear as a confusing welter of details: personalities,
interest groups, rhetorical demands, budget figures, legal rules and interpretations,
bureaucratic routines, citizen attitudes, and so on. Yet the concepts described in
this handbook are formulated in the abstract. You therefore need to learn to “see”
the analytic concepts in the concrete manifestations of everyday life.
Your Final Product
So what will your final product look like? Here is a very rough sketch of a typical
written policy-analytic report: In a coherent narrative style you will describe some
problem that needs to be mitigated or solved. You will lay out a few alternative
courses of action that might be taken. To each course of action you will attach
a set of projected outcomes that you think your client or audience would care
about, suggesting the evidentiary grounds for your projections. If no alternative
dominates all other alternatives with respect to all the evaluative criteria of interest,
you will indicate the nature and magnitude of the trade-off implicit in different
policy choices. Depending on the client’s expectations, you might state your own
recommendation as to which alternative should be chosen. xvi A PRACTICAL GUIDE FOR POLICY ANALYSIS
The Spirit of the Eightfold Path
The spirit of the Eightfold Path is, I hope, an economizing and uplifting spiric,
Analyzing public policy problems is a complex activity. I is easy to get lost, waste
alot of ime, become demoralized. Other manuals and textbooks in policy analysis
are very concerned that you get the analysis “right,” in some sense, I hope this
one will help in that respect too, But, even more, I hope that this one will help
you get it done with reasonable efficiency as well. PART |
The Eightfold Path
4. Define the Problem
YOUR FIRST PROBLEM DEFINITION is a crucial step. It gives you (1) a
reason for doing all the work necessary to complete the project and (2) a sense
of direction for your evidence-gathering activity. And in the last phases of the
policy analysis, your final problem definition will probably help you structure
how you éell your story.
Usually, the raw material for your initial problem definition comes from
your client and derives from the ordinary language of debate and discussion in
the client’s political environment, language I call generically issue rhetoric. This
shetoric may be narrowly confined to a seemingly technical problem or broadly
located in a controversy of wide social interest. In either case, you have to get
beneath the rhetoric to define a problem that is analytically manageable and that
makes sense in light of the political and institutional means available for mitigat-
ing it.
Use the raw material of issue rhetoric with care. It often points to some
condition of the world that people don’t like or consider “bad” in some sense, like
“teenage pregnancy,” “media violence,” or “global warming,” These evaluations do
not necessarily need to be taken at face value. You will sometimes wish to explore
the philosophical and empirical grounds on which you, your client, or others in
your eventual audience should or should not consider the alleged condition
“bad.” Furthermore, issue thetoric may point to some alleged—but not necessarily
real—cause of the troubling condition, for example, “welfare” or “human wasteful-
ness.” You want not simply to echo the issue rhetoric in your problem definition,
but to use it as raw material for a provisional problem definition that you hope
will prove analytically useful.
Some issues may connote more than one problem. Depending on the audi-
ence, for example, “teenage pregnancy” might connote sexual immorality, the
blighting of young people’s and their children’s life chances, exploitation of 2 A PRACTICAL GUIDE FOR POLICY ANALYSIS
taxpayers, and social disintegration. Usually you will want to determine a primary
problem focus; otherwise you will find that the analysis gets out of hand, Bur if
the problems aren’t too complicated, you may feel you are willing to define more
than one.
Think of Deficits and Excesses
Tt often—but not always—helps to think in terms of deficit and excess. For
instance:
® “There are too many homeless people in the United States.”
° “The demand for agricultural water is growing faster than out ability to
supply it at an acceptable financial and environmental cost.”
* “California's population of school-age children is growing at 140,000 pet
year, and our ability to develop the physical facilities in which to educate
them is not growing nearly as fast.”
Kc often helps to include the word t00 in the definition (e.., “too big,” “too
small.” “growing too slowly,” “growing too fast”). These last two phrases (abous
“growing”) remind us that problems deserving our attention don’t necessarily
exist “today” but are (at least potentially) in prospect for the future, whether near
or distant,
However, it does not help to think in terms of deficit and excess when your
broblem is an already well-structured decision problem, for example, “Dump the
dredging spoils cither in the Bay or somewhere out in the Pacific Ocean,” Nos
docs it help if your challenge is to invent any way to accomplish some defined
objective, for example, “Find some grant funds to close the anticipated gap between
revenues and expenditures.” These decision- and invention-type “problems” are
problems for the policy analyst but are not the sort of problems I am addressing
in this section.
The Definition Shoutd Be Evaluative
Remember the idea of a “problem” usually means that people think there is
something wrong with the world. But note that wrong is a very debatable term,
Not everyone will chink that the facts you (or others) have defined as a problem
ate really a problem, for each person may apply a different evaluative framework
to these facts. Unfortunately, there are no obvious or accepted ways to resolve
philosophical differences of this type.
‘common philosophical as well as practical question is “What private troubles r
THE EIGHTFOLD PATH 3
wartant definition as public problems and thereby legitimately raise claims for
amelioration by public resources?” It is usually helpful to view the situation
through the market failure lens.’ In its simplest formulation, market failure occurs
when the technical properties of a good or service
© make it hard to collect payment from all the potential beneficiaries—for
instance, the large number of people who profit, albeit indirectly, from
advances in basic sciences
© make it hard to collect from the beneficiaries of consumption the true
economic cost of making use of the good or service—fiesh air that vehi-
cle owners use as a sink for their auto emissions;
@ make it hard for consumers (and sometimes suppliers) to know the true
qualities of the good or service they are acquiring—many repair-type ser-
vices, including those performed by physicians as well as those performed
by auto mechanics;
© make the cost of producing the marginal unit lower than the average cost
within the relevant range of demand—a magazine article distributed via
the Incernet.
It is hard to overestimate the importance of this point, for in most—though
not all—situations where no actual market failures can be identified, people's
private troubles cannor typically be ameliorated by even the most well-intentioned
governmental interventions. And even when some amelioration is possible, there
are usually many adverse side effects. In some cases, it may nevertheless be
worthwhile to pay the price of these side effects, but such calculations must be
done carefully and scrupulously.
Besides market failures, the main situations where private troubles can warrant
definition as public problems are
© breakdowns of systems, such as family relationships, that occur largely
outside markets;
® the concern of many citizens about low living standards that arise pre-
cisely because markets function well and do not reward individuals very
generously if they lack marketable assets;
1. See chapter 5 in David L. Weimar and Aidan R. Vining, Policy Analysis: Concepes and Prac-
tice (Upper Saddle River, N.J.: Prentice Hall, 1999). For a persuasive analysis of most tradi-
tional market failures in transaction cost terms, see Richard O. Zetbe Jr., and Howard E.
McCurdy, “The Failure of Marker Failure,” Journal of Policy Anabsis and Management 18,
no. 4 (1999): 558-78. Zerbe and McCurdy also emphasize the rich variety of interven-
tions besides those undertaken by government to remedy traditionally conceived “market
failures.” 4 A PRACTICAL GUIDE FOR POLICY ANALYSIS
© the existence of discrimination against racial and other minorities;
© the failure of government to function well in areas where it is tradition-
ally expected to act effectively (e.g., in providing public schools).
Quantify if Possible
‘The definition should, insofar as possible, include a quantitative feature, Assertions
of deficie or excess should come with magnitudes attached. How big is “too big”?
How small is “too small”? How about “too slowly”? How about “too fast”? In
the examples above, how many homeless people are there in the United States?
How many acre-feet of water are used now, and how does that amount compare
with the demand in some specified future year (given certain assumptions about
water pricing)? Exactly what is “our ability to develop physical facilities,” and
how do we expect it to grow, or shrink, over time?
If necessary, gather information to help you calibrate the relevant magnitudes.
See the discussion under “Assemble Some Evidence.”
In many or most cases, you will have to estimate—or “guesstimate,” more
likely—the magnitudes in question. Sometimes you should furnish a range as well
as a point estimate of magnitudes (e.g., “Our best guess of the number of homeless
persons in families is 250,000, although the truth could lie between 100,000 and
400,000”).
Conditions That Cause Problems Are Also Problems
Some problem conditions are not experienced as troublesome per se by citizens
but are perceived by them, or by analysts working on their behalf, to be causes
of troubles. It is sometimes useful to diagnose one ot more alleged causes of this
type and to define these as problems to be mitigated or removed, for instance,
“One of the problems in the air pollution area is that states have not been willing
to force motorists to keep their engines tuned up and their exhaust systems in
proper order.”
Note that this sort of problem definition is not metely descriptive but is also
diagnostic. It implicitly asserts that some condition, which may or may not be
troubling to people per se, is an important cause of some other condition that is
indeed troubling, Problem definitions that pretend to some diagnostic power can
be useful, but they can also be treacherous. Suppose, after all, that the causal
diagnosis is mistaken or misleading, for example, that states’ unwillingness to
enforce engine maintenance routines is nor in fact a vety important cause of air
pollution. Because definition in some contexts connotes legitimate arbitrariness
(‘Tl define ‘justice’ to mean . ..”) the causal claims implicit in diagnostic problem :
L
THE EIGHTFOLD PATH 5
definitions can easily escape needed scrutiny. See “Project the Outcomes” for
further discussion.
Missing an Opportunity Is a Problem
‘A special case of “a problem” is an opportunity missed. Is it not rather small-
minded to think of policy analysis as devoted merely to the amelioration of
problems”? May policy analysis not rise above the tedious and uninspiring
business of patching and fixing? Can we not aspire to a world in which we can
identify opportunities to do creative—not to say wonderful—things? “If it ain’t
broke, don’t fix it” is a confining idea, and certainly policy analysts, policymakers,
and public managers ought not to allow the “problem” focus to restrict the search
for plausible opportunities. Unfortunately, the working agenda of most policy
professionals is set by complaints, threats, worries, and troubles. There is often
little time or energy left over to think about improvements that no one has
identified as missing. Still, if latent opportunities are really lying around, it would
be a pity to ignore them.
“Where do we find opportunities for creative policy improvements that haven’t
first been identified by complaints, threats, and so on? Little academic or technical
theory is available to answer this question, But Box LI (p. 6) contains a list that
is suggestive.
Common Pitfalls in Problem Definition
Problem definition is a deceptively simple step. It is a step beset by at least two
dangerous pitfalls.
Defining the solution into the “problem.” Your problem definition should
not include an implicit solution introduced by semantic carelessness. Projected
solutions must be evaluated empirically and not legitimated merely by definition.
‘Therefore, keep the problem definition stripped down to a mere description, and
leave open where you will look for solutions.
© Don’t say: “There is too little shelter for homeless families.” This formula-
tion might inadvertently imply that “more shelte:” is che best solution
and might inhibit you from thinking about ways to prevent families
from becoming homeless in the first place. Try instead: “There are too
many homeless families.”
Don’t say: “New schools are being built too slowly.” This formulation
could imply “more schools” as the solution and could inhibit you from
thinking about ways to use existing facilities more efficiently. Try instead: 6 A PRACTICAL GUIDE FOR POLICY ANALYSIS
BOX |. SOME GENERIC OPPORTUNITIES FOR SOCIAL IMPROVEMENT THAT
OFTEN GO UNNOTICED
Byrproducts of personal aspirations. It is possible to structure new incentives
or create new opportunities for petsonal advantage or satisfaction that can indi
rectly result in social benefit (e.g. offering to share the benefits of cost-reducing
innovations with public sector employees who conceive them and implement
them).
Complementarity. Two ot mote activities can be joined so that each makes
the other more productive (eg., public works construction and combating unem-
ployment).
Development. A sequence of activities or operations may have the potential
to be arranged to take advantage of a developmental process (eg., assessing wel-
fare clients for employability and vocational interest before, rather than after,
sending them out on a job search),
Exchange. There are untealized
possibilities for exchange that would increase
social value. We ty
ically design policies to exploit those that simulate market-
like arrangements (e.,, pollution permit auctions, and arrangements to rein,
burse an agency for services it renders another agency's clients or customen),
Multiple functions. \ system can be designed so that one feature can be used
to perform two or more functions (e.g., when a tax administrator dramatizes an
enforcement case in such a way as both to deter potential violators and to reas,
sure nonviolators thar they are not being made into suckers for their honesty),
Nontradivional participants, Line-level employees of public agencies often
have knowledge of potential program improvements that could usefully be incor-
Porated into the agencies’ policies and operations. The same is true of the agen-
cies! customers or clients ot the parties that they regulate
Rationalization, Purely technical rationalization of a system is possible (¢,,
shortening queues by deliberate spacing of arrival times, or creating contracts to
solidify informal agreements that are vulnerable to decay and misunderstanding).
Rummaging. By rummaging mentally, one might discover novel uses in
seemingly improbable but readily available materials (eg., using the automobile
registration system as a vehicle for carrying oue voter registration as well
Underusilized capacity. An example, in many communities, is school facili-
fies that are utilized for relatively limited purposes for only part of the day and
for only pare of the year—although school officials would be quick to warn that
using this capacity without harming school functions is not always easy. THE EIGHTFOLD PATH 7
“There are too many schoolchildren relative to the currently available
classroom space.”
A tipoff that you're probably smuggling an implicit solution into the problem
definition is to hear yourself saying, “Aha, but that’s not the real problem; the
real problem is...” While there are better and worse ways to conceptualize a
problem, or to solve a problem, ie stretches ordinary usage too much to say that
one problem could be “more (less) real” than another.
Be skeptical about the causal claims implicit in diagnostic problem defi-
nitions. I said above that “conditions that cause problems are also problems.”
However, the causes must be real, not merely assumed. You have to evaluate the
causal chain that goes from the situation itself to the bad things it is alleged to
cause, and to convince yourself that the causal chain is real. For instance, for
some people, “cocaine use” is not a problem in itself, buc it might be a problem
if it Leads to crime, poor health, family disintegration, and so on. But does it lead
to these outcomes, and to what degree? The evidence on this question should be
evaluated very carefully before you decide it’s okay to work with a problem
definition involving “too much cocaine use.”
iterate
Problem definition is a crucial step. But because it is hard to get it right, you
might take that same step again and again. Over the course of your analytic work,
your empirical and conceptual understanding will evolve. Also, as you begin to
tule out alternative approaches to solving or mitigating your problem, you will
probably want to sculpt the problem definition so that, in the end, you and the
political system will have some chance of attacking the problem successfully.
Finally, if you are working in an office or agency context, you will implicitly be
negotiating a mutually acceptable problem definition with your analyst colleagues
and your hierarchical superiors.”
2. Assemble Some Evidence
All of your time doing a policy analysis is spent in two activities: thinking
(sometimes aloud and sometimes with others) and hustling data that can be
2. Some analysts also claim thae it is simply not worthwhile co define as “problems” condi-
tions that cannot be ameliorated: “Problems are better treated as opportunities for improve-
ment; defined problems, as problems of choice between alternative means to realize a given
opportunity. ‘The process of problem definition would then be one of search, creation, and
initial examination of ideas for solucion uncil a problem of choice is reached.” David Dery,
Problem Definivion in Policy Anabsis (Lawrence: University Press of Kansas, 1984), 27. 8 A PRACTICAL GUIDE FOR POLICY ANALYSIS
curned into evidence. Of these ewo activities, thinking is by far the more important,
but hustling data takes much more time: reading documents, hunting in libraries,
poring over studies and statistics, interviewing people, traveling to interviews and
waiting for appointments, and so on,
The real-world settings in which policy analysis is done rarely afford the time
for a research effort that would please a careful academic researcher, In fact, time
Pressure is probably almost as dangerous an enemy of high-quality policy analysis
as politically motivated bias, if not more so. Therefore, it is essential to economize
on your data collection activities. The key to economizing is this: Try to collect
only those data that can be tutned into “information” that, in turn, can be
converted into “evidence” that has some beating on your problem.
For the logically minded, here are some definitions: Data are facts-—ot, some
might say, representations of facts—about the world. Data include all sorts of
statistics but go well beyond statistics, too. Data also include, for instance, facts
about an agency manager's ability to deal constructively with the press. Information
is data that has “meaning,” in the sense chae it can help you sort the world into
different logical or empirical categories. The prevalence of cigarette smoking in
five different countries is data, but these data become information when you
decide it is interesting to array the countries comparatively (e.g., from lowest to
highest prevalence). Evidence is information that affects the existing beliefs of
important people (including yourself) about significant features of the problem
You are studying and how it might be solved or mitigated. Differential prevalence
of smoking, for instance, can become evidence beating on hypotheses about .
differential levels of concern about personal health across counties.
You need evidence for three principal purposes. One purpose is to assess the
nature and extent of the problem(s) you are trying to define. A second is to assess
the particular features of the concrete policy situation you are engaged in studying,
For instance, you may need to know—or guess—about agency workloads, recent
budger figures, demographic changes in a service area, the political ideology of
the agency chief, the competency of the middle-level managers in the agency,
and the current attitudes of some other agency that nominally cooperates with
this one on some problem. The third purpose is to assess policies that have been
thought, by atleast some people, to have worked effectively in situations apparently
similar to your own, in other jutisdictions, perhaps, or at other times. All three
Purposes are relevant to the goal of producing realistic projections of possible
policy outcomes.
Because each of these purposes becomes salient in different phases of the
3. Sometimes these situations will have been evaluated statistically, and sometimes not: see
Part IIT, “‘Smart (Best) Practices’ Research.” rt,
Ss
id
Pee ee ee eee ae
THE EIGHTFOLD PATH 9
policy analysis process, this “Assemble Some Evidence” step on the Eightfold
Path will be taken more than once, but with a different focus each time.
Think Before You Collect
Thinking and collecting data are complementary activities: you can be a much
more efficient collector of data if you think, and keep on thinking, about what you
do and don’t need (or want) to know, and why. The principal—and exceedingly
common—mistake made by beginners and veterans alike is to spend time collecting
data that have little or no potential to be developed into evidence concerning
anything you actually care about. People often do this because running around
collecting data looks and feels productive whereas first-rate thinking is hard and
frustrating, Also, the people paying for your work tend to be reassured that, when
they see you busily collecting data, somehow they are getting their money's worth.
The value of evidence. Since most evidence is costly to produce, you must
weigh its likely cost against its likely value. How is its likely value to be estimated?
The answer may be cast in a decision-analytic framework (decision trees), though
you should remember that the process of making a decision involves a great many
elements prior to the moment of actual choice, such as defining a useful problem,
thinking up better candidate solutions, and selecting a useful model. In general,
the value of any piece of evidence depends on
© the likelihood that it will cause you to substitute some better decision for
whatever decision you would have made without it (which might have
been an “acceptable” decision in and of itself);
© the likelihood that the substituted decision will, directly or indirectly, pro-
duce a better policy outcome than the outcome that would have been
produced by the original decision;
© the magnitude of the difference in value between the likely-to-be im-
proved outcome and the original outcome.
Self-control. Ic is surprising how well you can do in many cases by gathering
no evidence at all but simply sitting down and thinking something through and
then making some serious educated guesses. There is nothing shameful about
acting on such guesstimates and thereby conserving your data-collecting time and
enetgies for answering questions for which good evidence is really necessary. See
Part Il, “Gathering Data for Policy Research.”
A helpful check on yourself, to prevent yourself from collecting useless data,
is to ask yourself the following questions before embarking on some data collection
venture: a0 A PRACTICAL GUIDE FOR POLICY ANALYSIS
© “Suppose the data turn out to look like so-and-so as opposed to thus-
and-such. What implication would that have for my understanding of
how to solve this problem?”
© “Compared to my best guess about how the data will look once I’ve got
them, how much different might they look if I actually took the trouble
to get them?”
© “How much is it worth to me to confirm the actual difference between
what I can guess and what I can lean about the world by really getting
the data?”
Ic is this sort of critical attitude about the value of expensive data collection
that often leads good and experienced policy analysts to make do with back-of-
the-envelope estimates. However, none of the above is meant to be an excuse for
shirking the job of getting good data—and sometimes lots of them, at huge costs
in time and money—when you've convinced yourself that the investment really
would pay off. There’s an obvious and critical difference between justifiable and
unjustifiable guesstimates.
Do a Literature Review
‘There is hardly a problem without some academic discipline or professional
association doing research on its causes and solutions. It is easy to find journals
and various professional publications disseminating research results, theories, case
studies, the musings of experienced practitioners, and so on. The Internet brings
much of this to your desktop, but some of the best vehicles ate better accessed
by browsing the periodical shelves of university or government libraries.
Advocacy organizations often publish a great deal of interesting work and
may take special pains to disseminate their work on the Internet. However, there
is a danger of relying too much on such soutces just because they are readily
available,
Survey “Best Practice”
The chances are that the problem you are studying is not unique, and that
policymakers and public managers in other jurisdictions, pethaps not very different
from the one you are studying, have dealt with it in some fashion. See if you
can track down some of these past solutions and see if you can extrapolate them
to the situation you are studying. The extrapolation process is complicated,
though. See Part III, “‘Smart (Best) Practices’ Research: Understanding and
Making Use of What Look like Good Ideas from Somewhere Else.” THE EIGHTFOLD PATH = 44
Use Analogies
Sometimes it pays to gather data about things that are, on the surface, quite
unlike the problem you are studying but that, under the surface, show instructive
similarities. For instance, your understanding of how a merit pay plan for compen-
sating managers in the public sector might work could perhaps be improved by
seeing how similar schemes work in the private sector. Or if you are working on
the problem of how a state can discipline, and perhaps disbar, incompetent
attorneys, you might usefully spend a good deal of your time learning about how
the medical profession handles problems of physician incompetence. If you are
working on how to reduce neighborhoods’ resistance to accepting low-income
housing projects, you could usefully look into the literarure on community resis-
tance to accepting solid-waste incinerators.
‘As these examples suggest, some analogies are casicr to perccive, and to make
sense of, than others. It takes a little imagination to see instructive analogies and,
occasionally, a little daring to try to convince others to see both the usefulness
of the analogy and its inevitable limitations.
Start Early
You are often dependent on the busy schedules of other very busy people whom
you ask to furnish you with information or opportunities for interviews. It is
extremely important to put in requests for information, especially interviews, well
in advance of when you expect to want to have completed the data collection.‘
Touching Base, Gaining Credibility, Brokering Consensus
The process of assembling evidence inevitably has a political as well as a purely
analytical purpose. Sometimes it entails touching base with potential critics of
your work so that they will not be able to complain that you ignored their
perspectives. By making yourself known to potential supporters of your work,
you may also be able to create a cadre of defenders.
‘A more complex objective, where appropriate, might be to blend policy
analysis with the process of improving a policy idea or decision during the course
of implementation. (See the following discussion of “improvability” as a criterion.)
This entails obtaining “feedback” from participants, usually in an iterative process,
4, For a useful description of how to conduct literature reviews, library searches, phone inter-
views, and personal interviews, sce Weimer and Vining, Policy Analysis, chap. 10A. See also
Part Il, “Gathering Data for Policy Research.” 12 A PRACTICAL GUIDE FOR POLICY ANALYSIS
and sharing some of your own reactions with them. You become more of a
partner in the process than an outside observer and diagnostician. An even mote
complex and challenging role is for you to become a particular type of “partner,”
* facilitator and broker, whether by acting as a transmission belt from one person
to another or by convening meetings and other gatherings.
Freeing the Captive Mind
In exchange for access to data and a ready-made worldview, researchers sometimes
uunctitically accept problem definitions and preferred solutions from kindly infor-
mants (not to mention from paying clients or employers). To counter such
temptations, be sure to make contact with individuals or factions whom you would
expect to disagree—the more sharply the berrer—with your kindly informants. A
time-saving, but only partial, substitute is to ask your kindly informant, “Who
might object strongly to your point of view about this, and why might they
do so?”
3. Construct the Alternatives
By alternatives I mean something like “policy options,” or “alternative courses
of action,” or “alternative strategies of intervention to solve or mitigate the
problem.”
Start Comprehensive, End Up Focused
In the last stages of your analysis, you won't want to be assessing more than three
or four principal alternatives. But in the beginning, you should ext on the side
of comprehensiveness: Make alist ofall the alternatives you might wish to consider
in the course of your analysis. Later on you will discard some obvious losers,
Combine others, and reorganize still others into a single “basic” alternative with
ne or mote subsidiary “variants.” For your initial list, though, where should you
turn for ideas?
« Note the alternatives that key political actors are actively proposing or
seem to have on their minds. These may include people’s pet ideas, insti-
tutions’ inventories of “off-the-shelf” proposals that simply await a win-
dow of opportunity, and prepackaged proposals that political ideologues
are perennially advocating, 2
THE EIGHTFOLD PATH = 43,
© Try to invent alternatives that might prove to be superior to the alterna-
tives currently being discussed by the key political actors. It’s good to
brainstorm, to try to be creative—but don’t expect that you'll necessarily
produce much better ideas than other people have already thought about.
One way to coax your creativity is to refer to the checklist in Appendix
‘A, “Things Governments Do.” For each entry on the list, ask yourself,
“Might it make sense to try some version of this generic strategy to help
mitigate this problem?” Because it is a long and comprehensive list, the
answer with respect to any single strategy will usually be negative. Going
through the list systematically is worthwhile, however. Because the list
is not very long, with experience you will need to spend only a few
minutes to decide whether any ideas there might be worth considering
further.
‘Always include in your first approach to the problem the alternative “Let
present trends continue undisturbed.” You need to do this because the
world is full of naturally occurring change, and some of these ongoing
changes might mitigate the problem on which you are working. (Note
that T am not characterizing this alternative as “Do Nothing.” It is not
possible to “do nothing.” Most of the trends in motion will probably per-
sist and alter the problem, whether for better or for worse.)
°
In most cases, however, this “let-present-trends-continue” option will drop
out of your final analysis. This happens because, if you do your problem definition
work well, you will end up with an important problem in your sights that in
most cases can be mitigated to some degree by some affirmative action.
Inspect the most common sources of “natural” change in the public policy
environment to see if any will affect the scope of the problem:
1. Political changes following elections, as well as changes induced by the
prospects of having to contest an election
Changes in unémployment and inflation rates that accompany the busi-
ness cycle
. The changing “tightness” or “looseness” of agency budgets caused by
overall taxing and spending policies
. Demographic changes, such as population migration patterns and popula-
tion “bulges” moving through certain ages.
Rp
+
. See also the very valuable discussion on generic policy instruments in Weimer and Vining,
Policy Analysis, chap. 9- 44 APRACTICAL GUIDE FOR POLICY ANALYSIS
Model the System in Which the Problem Is Located
“We often think about alrernative approaches to the problem 2s possible interven
tions in the system that holds the problem in place or Keeps ie going. Logically,
iets not necessary to model the causes of a problem in order to cure it—pharmacet-
tical manufaccurers can testify that many of their successful products work by
vraknown causal routes on conditions whose causes afe not at all understood. But
= good causal model is often quite useful for suggesting possible “intervention
points.” This is especially ruc when the problem is embedded in a complex
system of interacting forces, incentives, and constraints—which is usually the case.
Consider, for instance, a system that produces “too much traffic congestion” at
some choke point like a bridge or a tunnel. A skerch of the relevant causal model
vend include the demand for travel along che relevant route, the available modes
we ravel, che amount of roadway capacity, and the price r0 users of roadway
capacity. An efficient and simple—but usually politically unpopular—incervention
might be to increase the price 10 users 90 a8 10 reflect the degree to which each
eer contributes to congestion and increased travel times.
‘ous, and self-conscious should your causal model be?
How elaborate, rigor
Many social scientists who devote themselves £0 policy analysis would hold, “The
nore the better.” 1 say, “Yes, but...” Self-consciousness is highly desirable.
Flzborateness (or comprehensiveness, in this case a near synonym) is desirable
because it decreases the risk of missing important causal connections. But it can
blar the analytic focus and blunt creativity in designing intervention strategies.
Rigor is desirable if it prevents you from relying on unarticulated and false
ascumptions. But its down sie is that you might exclude factors that are impor-
‘unt—for instance, the personalities of certain actors—because you don’t know
how to model their effect rigorously and/or because you have only hunches
regarding the navure of the relevant personalities.
Many models are best thought of as elaborations of some fundamental meta-
phor. Some elaborations can be mathematical and quite precise, while others are
peal and evocative. Some commonly used metaphors that are the bases for
vrodels of particular value in designing alternatives are discussed here:
«A marker where disaggregated suppliers exchange goods or services with disag-
dqregated demanders. Nove that market models can apply to unpriced
foods and services, The main idea behind the market model is really
equilibration through exchange. Flence, the market model can be applied
to many phenomena other than the production and allocation of text-
book goods like widgets or apples.
To inatance, you might try to understand the flow of patients into @
stare mental hospital system in rerms of supply and demand: there is a THE EIGHTFOLD PATH 45,
short-run “supply” of available beds in state hospitals and a per-diem
charge for each, while there is a complex “demand? for their use gener-
ated by police departments, county psychiatric emergency units, judges,
members of the public, and so on.
A standard intervention strategy for improving markets that are not
working as well as they might be is to find some way to raise or lower
the prices faced by either suppliers or demanders.
* Production models. Unfortunately, there is not much of an academic litera-
ture about the operating logics of the common types of production sys-
tems found in public policy (e.g., command-and-control regulation, the
provision of information, and all those other “Things Governments Do”
briefly described in Appendix A). In any case, the main concern in un-
derstanding production systems should be to identify the parameters
whose values, when they move out of a certain range, make the systems
most vulnerable to breakdown, fraud and abuse, egregious diseconomies,
and the distortion of intended purpose. It is also helpful to know about
those parameters that matter most when we try to upgrade a production
system from mere adequacy to performance levels we might think of as
“excellent.” (See Pare III on smart practices.)
‘Another way to look at production models is through optimization
lenses. Operations research models (¢.g., queuing, inventory management,
Markov processes) are relevant here.”
© Evolutionary models, constructed of three important subprocesses: varia-
tion among competitors, selection, and retention. Suppose, for instance,
that, in an agency enforcing health-related standards in the workplace,
the complaints disproportionately concerned visible and annoying prob-
lems which were not, however, as hazardous to worker health as less visi-
ble and annoying problems. In this case, the evolutionary model suggests
several plausible intervention points. The agency might try to educate
workers to detect and complain about more serious problems, and try
thereby to swamp the less serious problems (thus changing the pool of
“competitors”). It might start screening the complaints for their likeli-
hood of being associated with more fruitful targets (thus changing the se-
6. However, see chapter 9 on “generic policies” in Weimer and Vining, Policy Anabsis. Also
sce Lester Salamon, ed., Beyond Privatization: The Tools of Government Action (Washing-
ton, D.C.: Urban Insticute Press, 1989).
7, For a good, brief discussion, see Stokey and Zeckhauser, A Primer, and Andres G, Vic-
torio, Applied Models in Public Policy (Manila: Ateneo de Manila University, 1995). Also
see the models, particularly that of case management, in Stephen R. Rosenthal, Managing
Government Operations (Boston: Little, Brown, 1982). 16 APRACTICAL GUIDE FOR POLICY ANALYSIS
lection mechanism). Or it might attempt to persuade workers, and pet~
haps their union representatives, (0 reduce their propensity to complain
bout matters the agency wishes to hear less about (thus changing the
“retention mechanism,” workers’ artirudes).”
Reduce and Simplify the List of Alternatives
“The final lise of alternatives—the one you include in your presentation 10 your
dthene and other audiences-—will almost certainly look quite different from the
fone you started with. Not only will you ave thrown some out that just don't
ioe geod but oul ae a work to conceptualize and
simplify alvernatives.
“The key to conceptualization is to try t0 sum UP the basic strategic thrust
ofan alternative in a simple sentence of even 2 phrase. This is difficule bu wsually
worth the effort. Ic usually helps t use very plain, short phrases stripped of
jargon. When the Environmental Protection Agency (EPA) was created, the, first
rernistraror confronted (a partial list of) aleernatives that might have been
vTeceribed thus: “Let the states do the works ler the feds give them the money”;
“Remove impediments to firms cooperating on antipollution research”; and “Sue
the bastards” (meaning the large, visible polluting firms and industries, the prosecu-
ion of which would help build political support for the new agency).
The key to simplification is to distinguish between a basic alternative and
jes variants, The basic clement in many policy alvernatives is an intervention
cerategy, such as regulatory enforcement oF 3 subsidy or a tax incentive, that
causes people of institutions to change their conduct in some way.” But no
enendion strategy can stand alone; i¢ must Be implemented by some agency
or constellation of agencies (peshaps including nonprofit organizations), and it
oer bave a source of financing. Usually the variants oF the basic strategy are
defined by different methods of implementation ‘and different methods of fi-
nancing.
“Thos distinction between a basic strategy and variants based on implementation
details is especially helpful when you have a lot of possible solutions to consider
and you need to reduce the complexity involved in comparing them. Making
__
5. For other ideas and an excellent discussion of the vses of models generally, see Charles A.
re een James G. March, Ar Insroduation #0 Models in ‘the Social Sciences (New York:
Harper and Row, 1975).
9, Often, though not always, the basic clement something like a smart practice, that is, am
Satemencion straregy that attempts co cake advantage of come qualitative opportunity co
