WHO/MSD/MSB 00.

2j

Facilitator’s
workshop guide
Evaluation of costs and effects of
psychoactive substance use treatment

Facilitator’s workshop guide 1

WHO/MSD/MSB 00.2j

c World Health Organization, 2000

WHO
World Health Organization

UNDCP
United Nations International Drug Control Programme

EMCDDA
European Monitoring Center on Drugs and Drug Addiction

This document is not a formal publication of the World Health Organization (WHO) and all rights are reserved by
the Organization. The document may, however, be freely reviewed, abstracted, reproduced and translated, in
part or in whole but not for sale nor for use in conjunction with commercial purposes. The views expressed in
documents by named authors are solely the responsibility of those authors.

2 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Acknowledgements

The World Health Organization

gratefully acknowledges the contri-
butions of the numerous individuals
involved in the preparation of this
workshop and workbook series, in-
cluding the experts who provided
useful comments throughout its
preparation for the Substance Abuse
Department, directed by Dr. Mary
Jansen. JoAnne Epping-Jordan
(Switzerland) wrote this Facilitator's
Workshop Guide and designed the
5-day workshop from which the
guide is based. Maristela Monteiro
(WHO’s Substance Abuse Depart-
ment) also contributed to its devel-
opment. Brian Rush of Virgo Plan-
ning and Evaluation Consultants
(Canada) contributed materials for
creating Programme Logic Models,
and provided general input on the
workshop agenda.

Financial support for this project

was provided by the United Nations
Drug Control Programme (UNDCP),
the European Monitoring Centre for
Drugs and Drug Abuse (EMCDDA),
and the Swiss Federal Office of
Public Health.

Facilitator’s workshop guide 3

WHO/MSD/MSB 00.2j

Table of contents
Background and goals of overall project 7
Introduction to the workshop 7
Structure of this guide 7
Structure of the overall project 8
Summary of the workbook component 9
Summary of the workshop component 10
Summary of the workshop follow-up component 10
Summary of the project evaluation component 11

Pre-workshop planning 12
1. Preparing participants 12
Selecting participants 12
Working productively with participants 12
Giving participants advance information 13
2. Preparing for facilitation 13
Qualities of a good facilitator 14
Working with your team to maximise effectiveness 14
Working cross-culturally 15
During the workshop 15
A final word... About participant ambivalence 16
3. Room structure, equipment, and materials 18
Room structure 17
Equipment and materials 17
Overhead transparencies 17
4. Suggested workshop schedule 18

Day 1 20
Opening activities 20
Presentation: introduction of workbooks 21
Exercise: fears and expectations 21
Site presentations of local treatment 22
Presentation: why is treatment evaluation important? 23
Exercise: the pros and cons of evaluation for me 23

4 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Day 2 24
Presentation: the 8 steps of planning an evaluation 24
Presentation and exercise: identify and
prioritise evaluation needs 25
Presentation: describe your programme for evaluation 25
Exercise: describe your programme for evaluation 28

Day 3 29
Presentation and exercise: define your evaluation questions 29
Presentation and exercise: determine your evaluation measures 30

Day 4 32
Presentation and exercise: prepare a data collection plan 32
Presentation and exercise: ensure that
your resources are sufficient 33
Presentation: the 6 steps of implementing an evaluation 34
Exercise: prepare your site presentation 35

Day 5 36
Presentation: summary 36
Site presentations of evaluation plans 36
Presentation: writing an evaluation plan 37
Closing activities 38

Problems and solutions 39

Annexes
Annex 1: How to describe your local treatment 42
Annex 2: Questionnaires 44
Annex 3: Written evaluation plan criteria and checklists 74
Annex 4: Overhead prototypes 82

Facilitator’s workshop guide 5

WHO/MSD/MSB 00.2j

6 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Background and goals

of the overall project
Introduction to the workshop
This facilitator’s workshop guide provides the The immediate goals for the workshop are:
essential information to organise and imple-
ment a training workshop on the Costs and • To introduce participants to the workbook
Effects of Treatment for Psychoactive Sub- series
stance Use Disorders. During the workshop,
participants will be guided by facilitators to • To guide participants to plan their own
plan an evaluation of their substance use treatment evaluation
treatment programme or network. Facilitators
assist participants to design an evaluation that Secondary or longer-term goals of the work-
suits their unique resources and needs. The shop are:
proposed training workshop is fundamentally
integrated with a complementary nine-volume • To enhance participants’ attitudes towards
workbook series that explains how to plan and treatment evaluation
implement treatment evaluations. Many presen-
tations and exercises are taken directly from the • To enhance participants’ general abilities
workbook series. The workbooks are described to plan and implement treatment evalua-
in greater detail on page 9 of this guide. tions

The workshop schedule is planned for five As a facilitator, it is important to familiarise

days. Experience suggests that five days are yourself with this guide, and the workbook
sufficient, but not excessive, to plan evalua- series from which it is derived, well in ad-
tions. Participants are introduced to the gen- vance of the workshop. Begin your prepa-
eral framework for evaluation on the first day. ration for the workshop by reading this
The bulk of their evaluation planning happens guide. After you have finished reading, you
during the second through fourth days. On can begin specific preparations with your
the fifth and final day, participants present their facilitation team and review the workbook
evaluation plans and discuss next steps. series.

Structure of this guide

This guide is divided into several sections. presents suggestions for managing common
“Pre-Workshop Planning” covers the essen- problems that arise during workshops. The
tial information you need to prepare for the annexes provide supplementary forms for
workshop. Detailed explanations of the proposed your use before, during, and after the work-
presentations and exercises, divided into five shop. Each of the annexes is explained within
days, are presented next. This is followed by a the main body of this guide. Refer to the Table
section entitled “Problems and Solutions,” which of Contents for more specific information.

Facilitator’s workshop guide 7

WHO/MSD/MSB 00.2j

Structure of the
overall project
The programme logic model illustrates follow-up, and project evaluation.
the overall project structure and goals. Implementation objectives, short-term
Project components include the work- goals, and long-term goals are listed un-
books, the training workshop, workshop der each component.

Project
Workshop
components Workbook Workshop follow -up Evaluation

Implementation
objectives • To transfer • To introduce • To support sites • To assess sites
needed workbooks to to effectively that receive
knowledge and sites implement their the workbooks
planning skills treatment and/or attend
• To enhance evaluations the workshop
• To provide a sites’ abilities to
clear and use workbooks • To gather
detailed guide effectively information
about the
• To guide sites to workshop’s
plan a local and
treatment workbooks’
evaluation effectiveness

Short-term
goals • To improve
• To prepare local evaluation plans in
workbooks and
collaboration with relevant stakeholders
workshop

Long-term
• To enhance sites’ capacities to conduct high quality treatment evaluations
goals

• To integrate evaluation systematically with treatment decision making

• To improve quality, coverage, and cost-effectiveness of treatment

8 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Summary of the workbook component

The first component of the project is the workbook series. The first three
manuals are introductory and foundation workbooks, whereas the next
six manuals are specialised workbooks.

Introductory Workbook
Framework Workbook

Foundation Workbooks
Workbook 1: Planning Evaluations
Workbook 2: Implementing Evaluations

Specialised Workbooks
Workbook 3: Needs Assessment Evaluations
Workbook 4: Process Evaluations
Workbook 5: Cost Evaluations
Workbook 6: Client Satisfaction Evaluations
Workbook 7: Outcome Evaluations
Workbook 8: Economic Evaluations

The workbooks are designed to be used in All workbooks contain “It’s your turn” par-
conjunction with the training workshop de- ticipatory exercises. These exercises are de-
scribed in this facilitator guide. The workbooks signed to assist participants to apply work-
also can be used without workshop training, book information to their own evaluations.
but better outcomes are expected when “It’s your turn” exercises are incorporated into
participants attend a training workshop in ad- the suggested workshop curriculum.
dition to reading the workbooks. This is be-
cause training workshops provide participants Most workbooks contain case examples,
with individualised attention through the evalu- which are real life accounts of treatment
ation planning process. Data are being col- evaluations conducted around the world.
lected currently regarding whether this hy- The cases can be useful learning tools.
pothesis is supported. They show how evaluators have conducted
different kinds of evaluations, logistical
Facilitators should keep these points in mind: problems they overcame, methods they
used, and how results were applied to im-
• The foundation workbooks are the main prove treatment services. Editorial com-
focus of the workshop, and should be used ments about each of the cases also are pro-
by all participants. vided. An “It’s your turn” exercise after
each case example will help you to reflect
• The specialised workbooks provide more them critically.
detailed information about specific types of
evaluation. Most participants will use only You should review all workbooks, including
one or two specialised workbooks during the “It’s your turn” exercises and case examples,
week. The choice of specialised workbook in advance of the workshop.
will depend on the evaluation question.

Facilitator’s workshop guide 9

WHO/MSD/MSB 00.2j

Summary of the workshop component

The second component of the overall project Secondary or longer-term goals of the work-
is the training workshop. The workshop is de- shop are:
signed to be conducted over the course of
five days by a facilitation team (for a sug- • To enhance participants’ attitudes towards
gested schedule, see page 23 of this guide). treatment evaluation
Detailed information about workshop prepa-
ration and curriculum is presented later in this • To enhance participants’ general abilities
facilitator’s guide. to plan and implement treatment evalua-
tions
The immediate goals for the workshop
week are: The workshop is fundamentally integrated
with the workbook series. Many presenta-
• To introduce participants to the workbook tions and exercises are taken directly from
series the workbooks. Participants should have the
workbook series available throughout the
• To guide participants to plan their own workshop, and should be instructed to re-
treatment evaluation fer to them as appropriate during the pre-
sentations.

Summary of workshop
follow-up component
The third component of the overall project is After planning is complete, sites also might
workshop follow-up. The main goal of the need assistance to implement their evaluations.
workshop follow-up is to support participat- Questions or problems could arise with data
ing sites in implementing their treatment evalu- collection strategies, data analysis, or using
ations. results effectively. Facilitators and/or other
consultants can be quite helpful in many of
Frequently, participating sites are not finished these situations.
completely with evaluation planning by the end
of the workshop. They may need additional Follow-up can occur via telephone, email, fax,
time and/or to consult with others before or post. Site visits also are possible, depend-
finalising their plans. Follow-up and assistance ing on available funding.
during this planning phase are often useful.

10 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Summary of the project

evaluation component
This project is currently being evaluated by There are pre- and post-workshop question-
WHO/ UNDCP/ EMCDDA. Results from naires, designed to be completed at the be-
the project evaluation will be used to improve ginning and end of the workshop. The pre-
the workbooks and workshop curriculum, with workshop evaluation questionnaire can be sent
the goal of creating written materials for gen- to participants in advance of the workshop,
eral dissemination with established utility and or completed during opening activities. The
effectiveness. Results also will be used to advantage of sending questionnaires in ad-
clarify whether the workshop produces en- vance (and asking for their return in advance)
hanced outcomes, relative to those who re- is that they can provide valuable information
ceive only the workbook series. about the participants that can be useful for
workshop planning. If participants have for-
Local or regional evaluation of the workshop gotten to complete the questionnaire, ask them
and workbook series also is important. to finish it during opening activities. For par-
Through your own evaluation of the work- ticipants who receive the workbook series,
shop and workbook series, specific problems but do not attend a training workshop, there
for your region, language, or culture can be is a “Pre-Workbook Questionnaire” on page
identified. Local evaluation also can assess 91 of this guide. This questionnaire is designed
whether your participants are satisfied with to be completed and returned by participants
various components of the project, as well before they receive the workbooks. In addi-
as whether their participation helps them tion, there are follow up questionnaires about
to plan and implement meaningful treatment the workbook series, designed to be com-
evaluations. pleted by all participants six months and 12
months following participation in the work-
Follow the basic steps for planning and imple- shop and/or receipt of the workbook series.
menting treatment evaluations (as outlined in These follow-up questionnaires are located
the foundation workbooks) when designing starting on page 96 of this guide.
your local evaluation of the workshop and
workbook series. Make sure that you under- Comments from organisers and facilitators to
stand your evaluation questions, have a spe- the World Health Organization are most wel-
cific plan for data collection and data man- come, and should be addressed to:
agement, and know how you will use your
results to improve the workshop and/or work- Dr. Maristela Monteiro
book series. Substance Abuse Department
World Health Organization
The participant questionnaires being used by 20 Avenue Appia
WHO/ UNDCP/ EMCDDA to evaluate the 1211 Geneva 27 · Switzerland
overall project are reproduced in this guide Fax: ++41 22 791 48 51
(pages 79 to 121) for your convenience. E-mail: [email protected]

Facilitator’s workshop guide 11

WHO/MSD/MSB 00.2j

Pre-workshop planning

1. Preparing participants
Selecting participants
Workshop participants should be chosen by participants should hold positions in which they
site with the assistance of selected criteria. have decision making authority.
A site could be a country, a defined region
within a country, or a specific treatment ser- Group definitions for this facilitator guide:
vice. An important criterion for selection is
that each site should have three individuals Individual level: each participant works
prepared to commit to the evaluation project alone (1 person)
and work together — one person involved in
planning and funding treatment programmes, Site level: participants work with the other
another involved in direct service delivery, and participants from their treatment site (usually
the third with technical and research skills. 3 people)
Each site should be asked to bring three par-
ticipants, representing these different sectors, Triplet level: three sites work together (usu-
with the purpose of discussing and reaching ally 9 people)
agreement on the evaluation to be undertaken
and going through the planning process dur- Group level: all participants meet together
ing the workshop. Regardless of background, (everyone)

Working productively with participants

Prior experience indicates that grouping sites create high quality evaluation plans. Facilita-
into larger groups of three sites each, or trip- tors should foster cross-site discussion and
lets, is beneficial for exchange between sites support within the triplets. By doing so, they
and monitoring by facilitators. Throughout the remove themselves from the role of expert
workshop, triplets meet with assigned facili- and rely instead on participants to help each
tators to exchange ideas and receive feed- other. Over the course of the workshop,
back about their evaluation plans. Triplets cre- cross-site collaboration usually grows stron-
ate the opportunity for cross-site peer ger, and lasting collaborations are frequently
education, and allow participants to draw from established.
each other’s knowledge and experience to

12 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Ideally, one facilitator should be assigned to guiding their sites through the exercises and
each triplet (three sites) for the duration of providing ongoing, individualised feedback
the workshop. Triplet formations and facilita- during evaluation planning. Facilitators should
tor assignments should be made by facilita- see their role less as expert and more as
tors in advance of the workshop. assistant.

As a general guideline, minimal time should Most participants will become fatigued occa-
be spent on lectures. The presentations sionally during the workshop week. Be alert
should be short and focused on orienting par- for signs of tiredness and take breaks as
ticipants to the exercises to be completed. needed.
Assigned facilitators should be responsible for

Giving participants advance information

It is usually helpful to provide participants with local treatment in preparation for the work-
basic information about the workshop before shop. A copy of detailed written instruc-
the actual training. Accordingly, in advance tions for completing this task, entitled “How
of the workshop, it is suggested that you send to describe your local treatment”, is located
participants copies of the Framework Work- on page 74 of this guide. Once sites finish
book, which presents the general model for the written summary, they should return it
treatment evaluation and some introductory to the organising facilitator. Upon receipt
information about the workbook series. Also of the summary, the organiser should re-
send the workshop agenda. Participants can quest that the site also prepare a 15 minute
read this information in a relaxed manner, and oral presentation of the summary informa-
in doing so, begin to orient themselves towards tion, to be presented during Day 1 of the
the task of evaluation planning. workshop. The thought of an oral presen-
tation may create a considerable amount of
Participants also should be encouraged to anxiety for some participants, so it is sug-
begin thinking about their local treatment gested that the organiser wait to request
service or network in a systematic way. One this task until the written summary is com-
way to accomplish this goal is to ask each plete. The organiser also should emphasise
site to create a written summary of their that the oral presentation can be informal.

2. Preparing for facilitation

This workshop is designed to be facilitated Ideally, the facilitation team should represent
by a team of qualified individuals. Why a a diversity of professional and cultural back-
team? Team facilitation increases the over- grounds. Diversity increases the overall
all knowledge and skill base available to par- knowledge and skill base of the team, brings
ticipants. It also allows facilitators to give multiple perspectives to the process, and re-
individualised attention to participants as flects the varying backgrounds of partici-
they plan their evaluations. Ideally, there pants. Also, many participants will feel
should be at least one facilitator for every more comfortable with facilitators from
three participating sites (or, stated differ- their own cultural or professional back-
ently, one facilitator per triplet), or a mini- grounds. For similar reasons, inclusion of
mum of two facilitators if there are less than women (both facilitators and participants)
six participating sites. also should be encouraged.

Facilitator’s workshop guide 13

WHO/MSD/MSB 00.2j

Qualities of a good facilitator

• Trust in others and their capacities (both • Familiarity with the culture, needs,
co-facilitators and participants) strengths, and limitations of participants,
and respect for individual differences
• Good listening skills
• Organisation: equipment, supplies, time
• Good presenting skills schedule

• Confidence without arrogance • Ability to strike the delicate balance be-

tween being directive and non-directive,
• Ability to create an atmosphere of confi- and to know when each stance is needed
dence among participants at different points during the workshop

• Respect for the opinion of others (not im- In addition to these qualities, facilitators also
posing ideas) should have relevant knowledge and experi-
ence in evaluation planning and implementa-
• Flexibility in changing methods and se- tion, and/or a good understanding of typical
quences as needed barriers and benefits of treatment evaluation.
Formal scientific training is advantageous for
• Knowledge of group process, including the certain types of evaluation, but is no guaran-
ability to sense the group’s mood at any tee of good facilitation.
given time and adjust the programme ac-
cordingly

Working with your team to

maximise effectiveness
Co-operation is essential among co-facilita- comes apparent that there is a gap in the
tors: mutual trust; recognition and acceptance knowledge base of the co-facilitators, efforts
of individual strengths and weaknesses; and should be made to bring a complementary
an attitude of reciprocal learning and assis- facilitator to the workshop (even for only part
tance. The success of the workshop does not of the workshop). In the absence of this op-
depend on individual performance, but rather, tion, facilitators can make note of questions
on how well the co-facilitators work together that arise during the workshop and have them
as a team to enhance participants’ evaluation answered by an expert consultant after the
planning. If problems arise with participants workshop ends.
during the workshop, or if you feel like you
need help, tell your co-facilitators. The team Throughout the workshop, facilitators should
should be able to support one another through meet for about one hour at the end of each
all difficulties. day. During this time, they should discuss any
problems or issues that have arisen, as well
Well ahead of the workshop, facilitators as their general impressions about how well
should familiarise themselves with the sub- participants are progressing with their plan-
ject matter and meet with their team to plan ning. All problems or issues (among partici-
the workshop in detail. Facilitators should dis- pants or facilitators) should be resolved as
cuss their backgrounds and professional quickly as possible. A general discussion and
strengths at this time. Based on this informa- debriefing among facilitators after the work-
tion, specific facilitators can be assigned to shop ends also are recommended.
specific presentations and exercises. If it be-

14 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Working cross-culturally
During many workshops, participants and fa- Facilitators also should be aware of the im-
cilitators will represent a range of cultural pact of their own cultural preferences. It is
backgrounds, which are not restricted to na- natural to be interested in people who share
tionality. Cultural differences also can include your opinions, native language, or ways of
differences in work environments (university, doing things. However, when you are a fa-
government, or treatment settings), status cilitator, it is important to be even-handed with
within organisations (director or employee), everyone, and to avoid any appearance of
roles within organisations (physicians or ad- preferential regard. Throughout the workshop,
ministrators), and perspectives on gathering you should consider whether you are being
information (quantitative or qualitative ap- fair to all concerned.
proaches).
Finally, language barriers might present sig-
As a facilitator, it is important to respect and nificant problems for facilitators and partici-
work with the values of participants from dif- pants from diverse backgrounds. In these situ-
ferent cultures. The key for overcoming cul- ations, remember to speak slowly, and avoid
tural barriers during the workshop is to seek complicated sentence structure and/or vo-
background information on the cultural prac- cabulary. For foreign languages, most people
tices of participants, to anticipate potential read better than they understand oral com-
problems, and to be willing to adjust your ap- munication. Use the written word to reinforce
proach as needed. While it is impossible to key concepts via overhead transparencies,
change some of your personal characteris- chalk boards, flip charts, and the workbooks
tics, such as your age or nationality, modifi- themselves. Pictures or diagrams also can be
cations to clothing, presentation style, and fa- useful.
cilitation techniques can be made as
appropriate.

During the workshop

Before the workshop begins each day, facili- summaries should document each site’s
tators should review the suggested curricu- progress, and can be shared with workshop
lum and make modifications if needed (e.g., organisers, future site consultants, and project
in the case that some sites are working funders. The summary should be about 20
through the material more quickly or slowly lines (more or less) for each site. Components
than anticipated). should include:

Facilitators should consult informally with each • site background information

other throughout the day, to ensure that ev-
erything is progressing smoothly and/or to • prior evaluation experience of participants
troubleshoot any problems that might arise.
• other skills participants bring to evaluation
Each facilitator should take notes throughout planning and implementation
the workshop about their sites’ progress in
evaluation planning. Notes provide useful ref- • barriers to evaluation
erences for discussing progress or problems
with other facilitators, and also serve as a good • type of programme logic model (national,
memory tool for oneself. regional, local)

At the end of the workshop, it is useful for • main evaluation question(s) and how that
facilitators to compile their notes into a writ- related to their programme logic model
ten summary about each of their sites. These

Facilitator’s workshop guide 15

WHO/MSD/MSB 00.2j

• major issues that sites struggled with • take time to converse informally with par-
during the workshop, and how (or if) these ticipants
issues were resolved
• attempt to solve any problems that par-
The site summaries should focus on objec- ticipants have, even if they are not related
tive information rather than facilitators? sub- directly to the workshop itself
jective opinions. Limit your comments to facts
about what happened during the workshop • organise after hours social events for ev-
week. Avoid judgmental statements, compari- eryone to attend. If participants have trav-
sons between groups, and your personal im- elled a long distance to attend the work-
pressions. shop, or are not familiar with the region,
they may be particularly grateful to attend
Finally, remember that facilitators have a spe- organised events.
cial role during the workshop: to help every-
one feel comfortable and valued. There are
several ways to accomplish this goal:

A final word...
About participant ambivalence
Participants may have mixed feelings about nections for participants. For example, a site
aspects of this project. Although possible con- may report limited funding for services as a
cerns are unlimited in nature, participants may major problem in its setting, and securing new
worry about the viability of doing treatment funding as a major goal. In this situation, you
evaluations locally, have scepticism about fa- can point out how evaluation results can be
cilitators understanding or supporting them, used to make programmes more efficient and
and/or feel too under-educated to be success- secure new funding in some cases. Other
ful in planning and implementing evaluations. common benefits to evaluation are: finding out
Others may feel they are over-qualified and whether programmes are working as hoped,
there is no point in participating. obtaining information that can improve
programmes, and improving local knowledge
It is important for facilitators to be aware that and experience about how to conduct evalu-
ambivalence of one sort or another is com- ations. The key is to personalise motivational
mon among participants, but that many times messages to particular sites as much as pos-
this ambivalence remains unexpressed. It sible.
is the role of the facilitator to be alert for pos-
sible ambivalence, and to have strategies for It also is important to understand possible in-
dealing with it effectively. hibiting factors (or barriers) to doing evalua-
tions. Barriers could include systemic prob-
One useful method for dealing with ambiva- lems, or involve negative attitudes about
lence is to increase participants’ motivation treatment evaluations. For example, partici-
for doing treatment evaluations by helping pants might think that they are too busy or
them to see how evaluations help meet their too inexperienced to conduct evaluations.
goals. One way to do this is to ask partici- Others might not really want to know if the
pants to discuss their treatment services and programme is working as hoped, because they
objectives. By listening carefully to their re- fear change or discontinuation of funding.
sponses, you can gain insight into possible These factors can be discussed and, if pos-
motivating factors and help to draw new con- sible, problem-solved during the workshop.

16 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

3. Room structure,
equipment, and materials
Room structure
The workshop site should have plenty of three participating sites, consider adjoining
space for sites to be able to spread out and rooms for each site triplet. If you use mul-
work separately without disturbing each tiple rooms, one room should be large
other. Auditorium settings are not advisable, enough to hold everyone for group presen-
however, because of the participatory na- tations. Also, rooms should not be far apart
ture of the workshop. If you have more than from one another.

Equipment and materials

The suggested equipment (listed below) should Participant equipment
be prepared in advance of the workshop.
While the recommended equipment is ideal • complete copy of workbook series - 1 set
for the various activities, modifications can per participant
be made if needed. For example, chalk boards
could be used in place of an overhead projec- • note pads - 1 pad per participant
tor or flip charts.
• pens or pencils
Facilitator equipment
• flip chart and paper - 1 chart per site
• overhead projector
• coloured construction paper - at least 2
• acetate sheets for overhead projector
different colours (for writing hopes and
• multicoloured pens for overhead projec- fears; louder and slower signs: see pages
tor 27 and 29) - about 8 sheets per participant
• flip chart (make sure you have plenty of
flip chart paper available) • multicoloured marking pens - 1 set per site

• multicoloured pens for flip charts • post-it notes or small pieces of paper
• note pads backed with tape (for creating programme
logic model) - 1 package per site
• pens or pencils
• name tags • name tags

Overhead transparencies
A set of overhead transparency prototypes is for your use throughout the workshop. Please
located at the back of this facilitator guide. If note that you should review the prototypes in
you plan to use an overhead projector, these advance of the workshop, and choose those
prototypes can be made into transparencies that best suit your needs.

Facilitator’s workshop guide 17

WHO/MSD/MSB 00.2j

4. Suggested
workshop schedule
This workshop schedule is based on the The workshop format combines brief presen-
premise that each participating site will send tations with participatory exercises. Partici-
three individuals to participate. During the pants listen to formal presentations based on
workshop, participants will be guided, step by workbook material, then meet with other par-
step, through planning a local evaluation of ticipants from their site to put the information
their substance use treatment services or net- to work immediately in planning their own
work. In the process of planning, they will be evaluations. Facilitators work closely with
introduced to the workbook series, and learn each site throughout the planning exercises.
to use it as a resource during and after the Participants also provide and receive feed-
workshop. back with other sites via triplet exercises.

The workshop is designed to tailor evaluation The schedule is planned for five days. Prior
plans to participating sites’ different levels of experience indicates that this amount of time
resources and/or expertise. Evaluation plans is sufficient, but not excessive, to accomplish
could be as simple as a needs assessment evaluation planning. Suggested time durations
survey, or as complex as a full economic are listed for each activity. Lunch and other
evaluation. The key for facilitators is to assist breaks can be inserted between activities as
in tailoring evaluation planning to each site’s needed. The schedule can be modified for
unique needs and capacity for evaluation. those who have more or less time available,
although it is recommended that you do not
alter the sequence of activities.

18 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Day 1 • opening activities (2 hours)

• presentation: introduction of workbooks (15 minutes)

• exercise: fears and expectations (30 minutes)

• site presentations of local treatment (15 minutes per site)

• presentation: why is treatment evaluation important? (15 minutes)

• exercise: The pros and cons of evaluation for me (1 hour)

Day 2 • presentation and exercise: the 8 steps of planning an evaluation

(1 hour)

• presentation and exercise: identify and prioritise evaluation needs

(1 hour)

• presentation: describe your programme for evaluation (1 hour)

• exercise: describe your programme for evaluation (4 hours)

Day 3 • presentation and exercise: define your evaluation questions (3-4

hours)

• presentation and exercise: determine your evaluation measures

(3-4 hours)

Day 4 • presentation and exercise: prepare a data collection plan (2 hours)

• presentation and exercise: ensure that your resources are suffi-

cient (1 hour)

• presentation: the 6 steps of implementing an evaluation (1 hour)

• exercise: prepare your site presentation (3 hours)

Day 5 • presentation: summary (15 minutes)

• site presentations of evaluation plans (15 minutes per site)

• presentation: writing an evaluation plan (15 minutes)

• presentation and exercise: overcoming barriers to evaluation (1

hour)

• closing activities (2 hours)

Facilitator’s workshop guide 19

WHO/MSD/MSB 00.2j

Day 1
The first day of the workshop has several one exercise that moves from the site to the
goals, including: triplet level. The major components of the
day’s suggested agenda are:
• making introductions and establishing rap-
port • opening activities

• orienting participants to the workshop • presentation: introduction of workbooks

agenda and goals
• exercise: fears and expectations
• introducing participants to the workbook
series • site presentations of local psychoactive
substance use treatment
• learning about each site’s background and
current situation • presentation: why is treatment evaluation
important?
• managing practical details of the workshop
week as needed (e.g., lodging, meals, • exercise: The pros and cons of evaluation
transportation, site funding) for me

During the first day, most activities happen at Each of these components is explained in
the group level. At the end of the day, there is greater detail below.

Opening activities
(group level)
* The pre-workshop It is a good idea to arrive at the workshop site • proposed time structure and agenda for
evaluation
early, to set up the room(s) and arrange your the week
questionnaire can be
equipment. Wear a name tag to identify your-
sent to participants in
advance of the self as a facilitator, and give participants their • assignment of sites to specific triplets and
workshop, or completed own name tags as they arrive. facilitators
during opening
activities. The The specific content of opening activities will • orientation to workshop site (e.g., location
advantage of sending vary by workshop. Suggested components in- of toilets, telephones, refreshments, etc.)
questionnaires in clude:
advance (and asking for • any outstanding practical issues (e.g., site
their return in advance) • opening statements by organisers and host funding, transportation, etc.)
is that they can provide
site representative
valuable information
• pre-workshop evaluation by participants
about the participants
that can be useful for • introduction of facilitators (questionnaire on page 80 of this guide)*
workshop planning. If
participants have not • introduction of participants (ask partici-
returned the pants to describe their professional role and
questionnaire, ask them list the skills that they bring to their evalu-
to finish it at this point. ation team)

20 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

If participants have different primary lan- ences, learn two words from each
guages, instruct participants to write the participant’s language, and facilitate useful
words slower and louder in their native communication.
languages on coloured paper (tell partici-
pants which colour to use for each word). If participants and facilitators share a com-
Then, at anytime during the workshop, par- mon primary language, the words louder and
ticipants should show the appropriate pa- slower can be depicted graphically by each
per if they would like the current speaker person. The same rules could apply: show the
to talk slower or louder. This is a fun appropriate paper when a speaker is talking
method to acknowledge cultural differ- too quickly or quietly.

Presentation: introduction
of workbooks
(group level)
Each participant should be provided with a formation about conducting evaluations,
set of workbooks. One facilitator should ex- whereas the specialised workbooks (Work-
plain the workbook series structure, and tell books 3 through 8) present detailed infor-
participants that the workbooks will be the mation for different types of evaluations.
basis for the upcoming workshop. Partici- Facilitators should give participants a
pants should be told that Workbooks 1 and minute or two to look through the workbooks
2 provide a solid foundation of general in- before proceeding.

Exercise: fears and expectations

(group level)
This exercise is a modification of question be read from a distance. To protect your pri-
3 of the Step 1A exercise located in Work- vacy, do not put your name on the cards.
book 1 (page 11). Participants should be given
two different colours of paper for writing their When finished, the cards should be collected
fears and expectations, and directed which by a facilitator and grouped by colour.
colour to use for each category.
Facilitators next should review briefly the
Give the following instructions (available as cards, and use these as a means to demon-
overhead prototype in the back of this guide): strate similarities and uniqueness of ideas. The
identity of card writers should not be asked
Working individually, list on the provided cards directly by facilitators. Instead, cards should
1) your expectations about the workshop, and be clustered and discussed at the group level.
then 2) your fears or concerns. Use different Following the presentation, cards should be
card colours for expectations vs. fears. Write taped to the wall and displayed for the dura-
legibly and use large print, so your cards can tion of the workshop.

Facilitator’s workshop guide 21

WHO/MSD/MSB 00.2j

In past workshops, the following themes have Expectations

been expressed:
• Transmitting knowledge to home region or
Fears country

• The workshop will not be relevant • Solving problem of addiction in home re-
gion or country
• Home treatment programme/network will
not be motivated to implement an evalua- • Increase skills
tion
• To get a simple method of evaluation
• There are insufficient funds to implement
an evaluation • To come back with practical projects

• There is insufficient time to cover all the • Learn something concrete

workbooks during the workshop
• Enhance future work
• Communication and language barriers
• Establish good contacts
• Personal and/or facilitator inexperience
• Break barriers of communication

Site presentations of psychoactive

substance use treatment
(group level)
Each site presents briefly (15 minutes) to the capacity for evaluation. Facilitators also
entire group an overview of its treatment ser- should use these presentations as an oppor-
vice or system. Presentations should be ar- tunity to listen for participant ambivalence and
ranged with sites in advance, and should be other potential barriers to evaluation. Feed-
based on the written summary created by each back from facilitators should be limited at this
site (see Preparing Participants on page 12 stage, however.
of this guide).
Following these presentations, facilitators
The purpose of the presentations is to obtain should be able to begin formulating possible
information about each site’s programmes and evaluation questions for each site.

22 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Presentation: why is treatment

evaluation important?
(group level)
The presentation from this section can be • the underpinning of the project is the de-
based on material from the Framework Work- velopment of a healthy culture of evalua-
book. Consider using the sections “Why tion, as part of ongoing collection of infor-
is treatment evaluation Important”, mation for decision making at the
“Treatment evaluation as feedback”, and programme, service, and system level;
“Levels of evaluation” for content. In-
clude these ideas: • participants should be ready to accept and
implement changes to their programmes
• historically, policy decisions regarding according to the results of the evaluations,
treatment have been made without con- regardless of their own views on specific
sidering evaluation results; treatments or approaches.

• evaluation is not a technology, nor a col-

lection of information or statistics, but an
attitude towards change;

Exercise: The pros and cons * Motivational

of evaluation for me
interviewing is an
approach based on
principles of experimental

(site to triplet level) social psychology, and

has the goal of increasing
intrinsic motivation to
change by helping to
Following the presentation, participants should The instructions are as follows (available as develop a discrepancy
be directed to complete the exercise described overhead prototype in back of this guide): between goals and
below. This exercise should be started at the current behavior.
site level, then move to the triplet level for 1 Each site should work separately and write Although Motivational
discussion. The purpose of this exercise is to on flip chart paper the pros and cons of Interviewing was
developed originally for
discover information about each site’s moti- doing treatment evaluation at its local site.
clinical use with people
vators and barriers to successful evaluation, Allow 15 minutes to complete this step.
who have alcohol
and secondly, to increase their motivation to problems, it is a highly
do evaluation in their setting. Those facilita- 2 Each site should meet with at the triplet effective tool for anyone
tors familiar with motivational interviewing level to discuss the pros and cons that they who is attempting to
techniques* may find them useful with par- wrote. negotiate behavior
ticipants at the latter point of this exercise. change in any setting
(such as facilitators of
this workshop).
For more information,
see Motivational
Interviewing: Preparing
People to Change
Addictive Behavior by
W.R. Miller and S.
Rollnick, Guilford Press,
1991.

Facilitator’s workshop guide 23

WHO/MSD/MSB 00.2j

Day 2
The goals for the second day include: activities also are included. The suggested
agenda includes:
• introducing the basic steps of evaluation
planning • presentation and exercise: the 8 steps of
planning an evaluation
• assisting sites to begin to identify and
prioritise their evaluation needs • presentation and exercise: identify and
prioritise evaluation needs
• teaching a structured method for describ-
ing programmes for evaluation • presentation: describe your programme for
(programme logic model) evaluation

• assisting sites to begin constructing their • exercise: describe your programme for
own programme logic models evaluation

Today, participants begin working intensively Each of the activities is described in detail on
at the site level. Some triplet and group level the next pages.

Presentation and exercise: the 8

steps of planning an evaluation
(group, site, and triplet levels)
This presentation should begin by over view- If organisers plan to give resources to sites
ing the 8 steps of evaluation planning (as out- for their evaluations, the sites should be made
lined on page 5 of Workbook 1). Follow- aware of the type and extent of resources
ing this, a more detailed presentation can be available. This will help them to evaluate their
made for Steps 1 and 2. situations correctly (see below), and may
eliminate some concerns about being under-
Step 1, Decide Who Will Be Involved In The resourced for evaluation efforts.
Evaluation, was covered to a large extent
during Day 1. Highlight the information pre- After you have finished your presentation,
sented on pages 10 to 11 of Workbook 1 direct participants to complete the “It’s your
for participants’ future reference. turn” Step 2 exercise, located on page
12 of Workbook 1. The exercise questions
Step 2, Assess Your Resources, is an impor- are designed to assist sites to evaluate accu-
tant but potentially intimidating task for par- rately their financial/material, expertise, and
ticipants. In your presentation, be sure to time resources for evaluation. Participants
emphasise that high quality evaluation CAN should work at the site level, and then move
be conducted with very few resources. The to the triplet level for feedback and exchange
key is to evaluate resources accurately, and if time permits. Facilitators should work
then choose a project that is practical with closely with sites and triplets during the exer-
the resources available. cise.

24 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Presentation and exercise: identify

and prioritise evaluation needs
(group, site, and triplet levels)
The following points might be useful to in- see the range of possibilities for evaluations,
clude in this presentation. It is natural for and perhaps to see some common trends in
site participants to have a diversity of ideas ideas. The next step (Step 3: Describe Your
about what to evaluate. For example, ad- Programme for Evaluation) will help to fur-
ministrators may be most interested in cost ther clarify needs and priorities.
efficiency, while clinicians may be inter-
ested in the benefits of a new treatment. It After the presentation, participants should
is important for all participants from each be directed to complete the “It’s your
site to understand one another’s perspec- turn” Step 4 exercise, questions 1 and
tives, then begin to come to a decision about 2, located on page 19 of Workbook 1.
their overall needs for evaluation. This step They should complete this exercise at the
of evaluation planning (Step 3: Evaluate and site level. If time permits, triplet discussion
Prioritize Evaluation Needs) is a first step of the exercise should follow. As with all
towards accomplishing this goal. Partici- exercises, facilitators should work with their
pants should be encouraged to explore sites and triplets to provide assistance and
freely different options for evaluation. The feedback as needed.
goal at this stage is to assist participants to

Presentation: describe your

programme for evaluation
(group level)
This is your opportunity to explain the pro- conduct an evaluation at the service level, then
cess and benefits of creating a programme the PLM should portray the service level, too.
logic model (PLM). This step of the work- In some cases, site participants will not know,
shop is particularly important, because the or disagree about, their desired level of evalu-
PLMs will form the basis for the evaluation ation. Discussion among participants to de-
plans created later in the week. The PLMs termine treatment evaluation level is the first
helps sites to achieve consensus on the evalu- priority in these cases.
ation questions that should be asked.
Refer to the information contained in Step 3
Before site participants create PLMs, they of Workbook 1 (starting on page 14) to
need to consider the treatment level at which structure the rest of your presentation.
they want to conduct their evaluation. Refer Overheads, which may help orient partici-
participants to pages 14 and 15 of the pants, are contained at the end of this guide.
Framework workbook for a reminder about
different treatment levels. In some cases, sites In addition to the information provided in the
will need to decide if they want to conduct workbook, consider including the following
their evaluations at the activity, service, questions as part of your explanation of how
agency, or system level. Whatever they de- to create programme logic models. (These
cide, their PLM level should match their evalu- questions are provided courtesy of Dr. Brian
ation level. For example, if a site wants to Rush, of Virgo Planning and Evaluation Con-

Facilitator’s workshop guide 25

WHO/MSD/MSB 00.2j

sultants, London, Ontario, Canada.) Ask par- your implementation (process) objectives
ticipants to answer the following questions in into components. Be careful not to get
reference to their own treatment programme: too detailed at this stage, but try to cap-
ture, in as specific terms as possible, the
1 In the long run, what should be different main elements of your program activities
in the community, or the “target popula- and services.
tion”, as a result of your programme be-
ing delivered? What are the changes you 4 The last stage is to show the connection
hope for, even recognising your between your various programme compo-
programme may only be playing a small nents and your short-term objectives (and
part in achieving these changes? These eventually your long-term objectives). This
changes are your long-term objectives. is done in a diagram connecting the vari-
Some of them may be quite general and ous activities and outcomes. Again, the
reflect broad goals or aspects of your mis- examples provided will illustrate how this
sion statement. is done. The most important point, how-
ever, is that there should be some logic or
2 In the shorter term, what changes do you rationale underlying these connections. In
hope will occur in the community or the other words, why should delivering these
“target population”, as a result of your activities or services produce these
programme being delivered? What short- changes?
term changes are needed in order to
achieve your longer term objectives 5 Once you have a draft of a diagram put-
(goals) identified above? How are these ting all of the above together, you should
objectives linked over time? In other expect to revise it several times. This is a
words, what changes lead to what other great tool for communicating with each
changes to eventually achieve your longer other about your programme. You can use
term objectives? Write down all the it to discuss how your program has
changes you think might happen as a re- evolved over time and should adjust in the
sult of your programme and then ask your- future. In other words, it is a planning and
self how they are inter-connected. What programme management tool as much
is the logic or rationale connecting one as an evaluation tool.
change to another?
Once you have a good draft you can start
3 Next, rather than focusing on the changes using it to plan your evaluation activities. Ba-
you hope to make, think about how you sically, your task is to use the logic model to
are trying to do this. What services or ac- prioritise those aspects of the programme to
tivities do you currently deliver or which be evaluated and to develop and select the
you plan on delivering in the near future? specific evaluation questions to be addressed.
These are your implementation (process The chart that accompanies the logic model
objectives). They reflect what you do, will help you do this by giving you a frame-
rather than the outcomes (changes) you work to translate each of your programme
expect to achieve. This is an important objectives (from the logic model) to an evalu-
distinction since delivering the service is ation strategy for getting feedback on the
only a means to an end; the services you achievement of that objective. It helps in this
provide are not themselves the out- process to have numbered each objective in
comes. Your activities or services prob- your logic model (or group of objectives) and
ably cluster into different program com- transfer them one at a time to this table using
ponents. There is not hard and fast rule the number to cross-reference.
as to how you cluster them — some-
times it's because a set of activities or What will emerge as you complete this table
services all relate to a particular group is a very large “shopping list” for your evalu-
of people (e.g., youth vs elderly) or they ation. Now, as a group, your task is to decide
all relate to a particular programme func- what is feasible for an evaluation plan giving
tion or role (e.g., smoking cessation vs. due consideration to the resources you have,
alcohol education). The examples pro- measurement and design issues, ethical and
vided give a flavour of how to organise confidentiality considerations, etc.

26 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Experience indicates that specific ex- programme in Workbook 1 and 2 is pre-

amples of programme logic models help sented below and in the overhead proto-
participants better understand the basic type section in the back of this guide.
principles. The programme logic model Consider referring to this example as you
from the fictional heroin detoxification make your presentation.

Main Diagnosis Withdrawal

Assessment & & Treatment
components management Referral
Intake planning & Treatment

Implementation • To confirm clients’ • To formulate • To monitor • To provide

objectives eligibility for the treatment plan withdrawal information
program symptoms about other
• To determine • To sign • To prescribe programs for
clients’ motivation therapeutic standard further drug-free
to engage in contract medication treatment
treatment • To provide a safe
• To motivate
• To determine and supportive
environment clients to
clients individual
continue
needs • To conduct labor-
treatment in one
• To obtain standard atory/other tests
of these
somatic, mental • To motivate client programs
and psychological to finish the
status of the client program
• To form therapeutic • To provide safe
alliance with the withdrawal
client management

Short-term • To collect the • To cope with withdrawal symptoms • To increase clients

outcome necessary knowledge about
• To stabilise mental and physical status
objectives information for further treatment
development of possibilities
• To maximise client satisfaction with the
adequate • To increase clients
program
treatment knowledge about
• To enhance • To reduce involvement in criminal consequences of
clients’ activities PSU and AIDS
motivation to • To maximise the
stay in the number of clients
detoxification who are referred
program to long-term
treatment

Long-term • To collect a • To reduce drug use behaviour

outcome database of
• To prevent consequences of drug use
objectives standardised
client information
• To improve mental and physical well-being of clients
• To improve
quality of care

Facilitator’s workshop guide 27

WHO/MSD/MSB 00.2j

Exercise: describe your

programme for evaluation
(site level)
After you have finished describing programme understanding the basic PLM structure, while
logic models, direct participants to move to others might disagree among themselves
the site level and complete the “It’s your turn” about their activities or goals. Regardless, it
Step 3 (A and B), located on pages 15 is especially important for facilitators to pro-
and 17 of Workbook 1. The majority of time vide intensive assistance and feedback dur-
should be spent on exercise 4B. Post-it notes ing this exercise. Listen to sites’ discussions,
(small pieces of note paper with adhesive on provide advice or examples if needed, and
one side) are useful for creating programme make yourself available for questions as they
logic models: if each component of the model arise. Expect that most sites will need sev-
(e.g. a single activity or objective) is written eral hours and multiple versions before being
on one post-it note, transformation and edit- satisfied with their PLMs.
ing of the model is easy to perform. If post-it
notes are not available, small pieces of paper It is a good idea to move sites to the triplet
backed with tape can be used instead. Sites level at least once during this exercise. In-
should be given adequate space to create their put from other sites can spur new thinking
models (e.g. a portion of an empty wall, or a or resolve dilemmas. Do not wait until the
large flip chart). end of the exercise to get feedback at the
triplet level; move sites to their triplets well
Prior experience indicates that for most sites, in advance of completion of the PLMs.
creating a programme logic model is the most After receiving triplet level feedback, par-
challenging and thought-provoking task of the ticipants can continue working on their
workshop. Some sites might have difficulty models at the site level.

28 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Day 3
The goals for the third day include: Today, participants will continue to spend the
majority of time at the site level, although some
• using the programme logic model that was time will be spent at group level presentations,
developed during Day 2, sites will define and with triplet level feedback. Specific ac-
specific evaluation questions tivities for the day include:
• participants will be instructed how to use
• Presentation and exercise: define your
the foundation and specialised workbooks
evaluation questions
together
• Presentation and exercise: determine your
• participants will learn guidelines for choos-
evaluation measures
ing evaluation measures
• sites will begin to choose their evaluation Each of the activities is described in detail
measures below.

Presentation and exercise:

define your evaluation questions
(group, site, and triplet levels)
The presentation and exercise should be based this approach, and then assist the sites to nar-
on Step 5 of Workbook 1 (pages 20 to row the questions to a feasible number and
21). The presentation should be given at the scope given their time, experience, and other
group level. It can be brief (15 minutes), but resources.
should include some general remarks about
how to narrow the scope of possible evalua- In addition to feasibility, sites should be en-
tion questions (see below for specific content couraged to create evaluation questions that
suggestions). Following the presentation, sites are meaningful to the treatment programme
should be given several hours to formulate or service. That is, questions should be linked
and discuss their evaluation questions. clearly to a specific component or goal of the
programme logic model, and have clear im-
A general structure for moving from the plications for improving treatment services.
programme logic model to evaluation ques- Sites might want to ask themselves: How will
tions and data collection methods is presented the results from the proposed evaluation ques-
in the overhead prototype entitled “Basic For- tion be used to improve our treatment ser-
mat for Moving from Logic Model to More vices? If there is no clear answer, then evalu-
Detailed Evaluation Planning” located in the ation questions should be reformulated.
back of this guide. Consider showing this form
to participants at the beginning of the presen- At this point in the workshop, participants do
tation and making copies for their use during not need to choose specific evaluation mea-
this phase of evaluation planning. sures. (They will be chosen later in the day.)
Participants should be instructed to focus on
Experience shows that many sites will want formulating general evaluation questions, en-
to “answer it all” with their first evaluation. suring that they are feasible and meaningful.
This approach usually ends unfavourably. It Following the presentation, participants should
is essential that the facilitators caution against be directed to move to the site level and com-

Facilitator’s workshop guide 29

WHO/MSD/MSB 00.2j

plete questions 1 through 3 of the “It’s your During the discussion, provide constructive
turn” Step 5 exercise, located on page 21 commentary and guidance to sites on their
of Workbook 1. Participants will not be able to evaluation questions. Ideally, these two dis-
do question 4 at this point, as it involves consul- cussion points should unfold seamlessly and
tation with expected users of the results. simultaneously.

At a middle point of the exercise, sites should Experience indicates that some participants
be moved to the triplet level, where they can will confuse process and outcome evaluation
present possible evaluation questions to com- questions. In other words, they might attempt
ment and feedback from other participants to answer an evaluation question about out-
and the assigned facilitator. Triplet feedback come by measuring process, or vice versa.
serves a dual purpose: to give participants and As you listen to possible evaluation questions,
facilitators an opportunity to provide commen- ask yourself whether your sites are making
tary to sites on the feasibility and appropri- this common error. In preparation for this
ateness of evaluation questions, and for fa- eventuality, consult Workbooks 4 and 7 for
cilitators to teach participants about different more information about the differences be-
evaluation types. As sites are discussing their tween process and outcome evaluations.
evaluation questions, the facilitator should take Briefly, process evaluations seek to understand
notes. Once every site has finished, the pos- the extent to which a treatment programme
sible questions should be used to explain dif- is operating as planned. The evaluation ques-
ferent evaluation types (see Framework tions do not ask about changes in clients that
workbook, pages 18 to 27): result from services being provided. Rather,
the questions concern coverage and process.
• Needs Assessment Outcome evaluations measure how clients and
• Process their circumstances have changed, and second,
• Costs attempt to show that treatment has been a fac-
• Client satisfaction tor in causing this change. Typically, process
• Outcome evaluations should be conducted in advance of
• Economic outcome evaluations. Why? Evaluators must
first establish that treatment is happening in a
Use the site evaluation questions to highlight consistent and desirable manner before evalu-
key differences in each type of evaluation. ating whether it helps clients.

Presentation and exercise:

determine your evaluation measures
(group, site and triplet levels)
Before beginning the main presentation, the foundation workbooks for general informa-
presenting facilitator should make a few com- tion. An overhead prototype listing the
ments about how to use the specialised work- specialised workbooks content is located in
books in conjunction with the foundation work- the back of this guide.
books. Participants should be reminded that
Workbooks 1 and 2 provide a solid founda- The rest of the presentation should cover key
tion of general information about conducting material from Step 6 of Workbook 1 (pages
evaluations, whereas the specialised work- 22 to 41). You should tailor this presentation
books (Workbooks 3 through 8) present de- to match participants’ prior knowledge of mea-
tailed information for different types of evalu- surement issues and evaluation questions. If
ation. Sites should begin consulting the participants are relative novices concerning
specialised workbook that is applicable for measurement issues, a discussion of basic
their evaluation question, while still using the concepts may be needed. In other cases,

30 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

higher-level discussions may be appropriate. be made aware that any revisions such as
Regardless, limit your comments to 45 - 60 rewording, eliminating, adding, or reordering
minutes maximum. items might diminish the validity and reliabil-
ity of an established instrument.
Possible areas for presentation include:
Finally, tell participants that sometimes they
• The language (meaning) of measurement cannot rely on existing instruments, scales,
• Why bother with indicators? forms, or interview schedules to create their
data collection instruments. If they must de-
• Quantitative and qualitative measurement
sign a new indicator or question, they should
• Reliability and validity use the following checklist for constructing it
• Types of measures (reproduced on page 60 of Workbook 1 and as
• Tips for selecting or preparing a data col- an overhead prototype in the back of this guide):
lection instrument
1 Are the words simple, direct and familiar
Overhead prototypes for each of these top- to all?
ics are located in the back of this guide. 2 Is the question as clear and specific as
possible?
Regardless of participants’ prior experience, 3 Is it a double question?
remind them during the presentation that if
4 Does the question have a double negative?
they don’t take measurement issues seriously
during programme evaluation, any claims they 5 Is the question too demanding?
make about the programme will always be 6 Are the questions leading or biased?
open to criticism. Someone could always say: 7 Is the question applicable to all respon-
“This sounds nice, but how do you actually dents?
know your programme accomplishes that?”
8 Is the question objectionable?
By using good measures and data collection
techniques, participants can provide a better 9 Will the answers be influenced by response
response to this question. For this reason, sys- styles?
tematic measurement in programme evalua- 10 Have you exhausted the response alter-
tion is the best tool for convincing people about natives?
what a programme does, how it functions,
what outcomes are achieved and what has The exercise following the presentation (It’s
been done to improve it. Your Turn Step 6F from Workbook 1:
page 41) is designed to assist sites to select
Caution participants to limit their evaluation their own data collection methods. Note that
*A review of the
measures to those that are necessary to an- sites will not be able to answer question 3 at Addiction Research
swer the evaluation questions. Some partici- this time, because it requires consultation with Foundation Outcome
pants might want to collect as much data as expected users of the results. If sites used Measures Directory
possible, for example measuring client depres- the form, “Basic Format for Moving from (undated) is highly
sion using three different scales. This ap- Logic Model to More Detailed Evaluation recommended. This
proach leads to unnecessary burdens upon Planning,” during the morning session, they Directory contains
evaluation participants, higher drop-out rates, should continue to use it during this exercise. many potentially
Facilitators should provide guidance as useful instruments
and more cumbersome data management.
needed. In many cases, sites will need spe- and discusses
Limiting measures to those that are absolutely
reliability, validity,
needed to answer the evaluation questions can cific suggestions about measures to use.
and practical issues in
avoid these kinds of problems. Many measures are presented in the work- administration.
books and case examples, so be sure to Information about this
Also advise participants that there are many familiarise yourself with these materials in publication can be
advantages to using existing instruments that advance of the workshop. In particular, see obtained from:
have been proven to be valid and reliable. the tables on page 11 of Workbook 4 for Addiction Research
They need to be sure, however, that the mea- a comprehensive listing of relevant mea- Foundation, 100
sures collect the right kind of data, and that sures. In addition to the workbooks, it is a Collip Circle, Suite
good idea to have copies of other common 200, London, Ontario,
they are applicable for their specific culture
data collection measures available.* Canada, N6G 4X8.
and/or setting. In addition, participants should

Facilitator’s workshop guide 31

WHO/MSD/MSB 00.2j

Day 4
Main goals for Day 4 include: This day covers a large amount of workbook
material: the remainder of Workbook 1 and
• Review how to prepare for data collec- Workbook 2. Presentations based on Work-
tion, including choosing a time frame and book 1 are fairly thorough, while Workbook 2
sampling strategy. Assist sites to develop presentations just introduce key concepts and
their own data collection strategies. recommend further reading. The final part of
the day is spent preparing for Day 5's site
• Ensure that sites’ evaluation plans are re- presentations.
alistic given the time and resources they • presentation and exercise: prepare a data
have available. collection plan
• presentation and exercise: ensure that your
• Introduce the 6 steps of implementing
resources are sufficient
evaluations,
• presentation: the 6 steps of implementing
• Give sites a clear structure for presenting an evaluation
their evaluation plans during Day 5. • exercise: prepare your site presentation

Presentation and exercise:

prepare a data collection plan
(group, site, and triplet levels)
This group level presentation should corre- points. In terms of the number of times to
spond to the information presented in Step 7 collect data, more is not always better. Why?
of Workbook 1 (pages 42 to 47). Key con- Frequently repeated data collection (e.g.,
cepts include a) setting a time frame for data weekly during treatment) results in increased
collection, and b) determining the sample to data collection resources and more compli-
be used. In other words, sites must decide cated data analysis. And in many cases, fre-
about the “when?” and “who?” of data col- quently repeated data collection is not neces-
lection. Their decision-making should depend sary to answer evaluation questions. Many
on their evaluation questions, limitations on evaluations require one data collection point
resources, and other practical constraints. only. Even for more complicated outcome
evaluations, one pre-treatment and one post-
When setting a time frame, sites should con- treatment data collection point (and perhaps
sider type of evaluation that they are doing one additional follow-up data collection) is
and their resources for collecting and man- usually sufficient. If multiple data points will
aging data. They need to decide two things: be collected, decisions about time intervals
the number of times to collect data, and the between data points also should be driven by
amount of time between the data collection evaluation questions and practical constraints

32 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

on resources. In any case, sites should be di- Following the presentation, direct participants
rected to consult the appropriate specialised to move to the site level and complete the
workbook for specific suggestions. two “It’s your turn” exercises for Step
7, located on pages 44 and 47. Please note
Sampling is an important issue, yet one that that participants will not be able to answer
novice researchers often downplay or over- each exercise’s question 3 at this point, be-
look while conducting evaluations. Beginning cause each question requires input from ex-
evaluators may see no harm in sampling the pected users of results. Facilitators should
most “convenient” group of people without assist sites in their decision making as needed.
considering the group’s representativeness. After sites have discussed their plans, move
Consequently, results can be biased and non- them to the triplet level for feedback. By this
generalisable. Participants should be point, triplets should be working well together,
familiarised with the important concept of allowing for meaningful cross-site education
representativeness, and with different meth- and support.
ods for sampling populations (see page 45
of Workbook 1).

Presentation and exercise: ensure

that your resources are sufficient
(group and site levels)
This group level presentation is based on Step able, well-organised plan that also is feasible
8 of Workbook 1 (pages 48 to 49). The to complete successfully. If facilitators have
goal of this presentation and exercise is to been alert for potential problems throughout
ensure that each site’s resources (financial/ the planning process, then this exercise should
material, expertise, and time) are sufficient not create the need for major revisions.
for the evaluation plan that its participants have
created. The presentation itself should be brief In some cases, workshop organisers might be
and focused upon orienting participants to the providing funding and/or other support to sites.
site level exercise. If you have not already done so, this is the
time to negotiate any outstanding issues,
Ask participants to move to the site level and which might include:
complete the “It’s your turn” exercise lo-
cated on page 49 of Workbook 1. Because • the project time frame
some participants might be sensitive about
discussing their resources in the presence of • seed funding
other sites, it is suggested that participants
remain at the site level for the duration of this • availability of follow up consultants
exercise. Facilitators, however, should assist
sites to examine their resources realistically, • means of communication with the
and redirect them as needed. organisers following the workshop

The goal of this exercise is to have partici- • deadlines for evaluation plans and progress
pants review their evaluation planning and reports
decide whether they have created a reason-

Facilitator’s workshop guide 33

WHO/MSD/MSB 00.2j

Presentation: the 6 steps of

implementing an evaluation
(group level)
This group level presentation should overview Step 2...
the 6 steps of implementing an evaluation (as
outlined in Workbook 2). Because of time This presentation should be a brief overview
constraints, it is impossible to cover each of of salient issues for data collection. Because
these steps in detail during the workshop. participants will not yet be at this stage of
However, facilitators should mention key con- their evaluations, you can simply point out
cepts from each of the six steps (outlined important issues to consider, and direct them
below). to pages 22 to 23 of Workbook 2 for fu-
ture reference.
Step 1...
Step 3...
Preparing for data collection is an important
step in the process of conducting evaluations. Step 3, data analysis, can be a highly techni-
Key components include: cal area. Your comments should be tailored
to the participants’ levels of expertise and
Managing ethical issues evaluation questions.

• Informed consent Depending on your audience, key concepts

include:
• Manage risks of participation
• What can data analysis tell us?
• Confidentiality
• The importance of planning for data analy-
Developing a data management plan sis in advance

• Design a record keeping system • General factors to consider when choos-

ing a data analysis strategy
• Acquire and process data
• The importance of seeking technical help
• Train people who will collect data as needed

• Standardisation
Steps 4 to 6...
Conducting a pilot test
Remember to convey the important concept
that evaluation does not end with data analy-
• Identify flaws
sis. Reporting results, making use of what was
learned, and starting again are equally vital
• Revise data collection plan as needed
aspects of the evaluation process. The “Cre-
ating a Healthy Culture for Evaluation”
Writing an evaluation plan
graphic located on page 8 of the Frame-
work Workbook (and available as an over-
(this task will be covered in detail during Day
head prototype in this guide) can be used
5)
to structure this part of the presentation.
Participants should be directed to Step 1 of
Workbook 2 (pages 10 to 21) for more in-
formation about these topics.

34 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Exercise: prepare your

site presentation
(site level)
The final part of the day should be devoted to Briefly show your programme logic model
preparing for Day 5’s site presentations, dur- components. Indicate the part of the model
ing which time each site will be asked to that is the focus of the evaluation.
present briefly (15 minutes) an overview of
its evaluation plan. The primary purpose of Section II. Evaluation question, measures,
the presentations is to cross-inform partici- data collection
pating sites about their evaluation plans. The
secondary purposes of the presentations are Show the one or two most important evalua-
a) to provide a “positive reinforcement” op- tion questions and how you will answer them
portunity for each site’s evaluation planning (evaluation plan)
efforts, b) to build a sense of community be-
tween the sites, and c) to increase each site’s Section III. Using the results for decision-
commitment to complete its evaluation making
through public proclamation of its plan.
Describe who will receive the results from
Most sites will need to wait until they return the evaluation and how it will be used for de-
home and consult with others before finalising cision-making.
their evaluation plans. In these cases, sites
should do this exercise with the planning they If an overhead projector is available, each site
have completed at this point. should be instructed to create one overhead
transparency per section to structure their
Participants should be given the following evaluations. Photocopies of these transpar-
structure for preparing their site presentations encies can then be made for facilitators and
(available as an overhead prototype in the other sites to keep as references. If you do
back of this guide): not have an overhead projector, flip charts or
chalk boards also can be used.
Section I. Programme logic model and
programme description

Facilitator’s workshop guide 35

WHO/MSD/MSB 00.2j

Day 5
Goals for the fifth and final workshop day in- Like Day 1, today’s activities occur mostly at
clude: the group level. Participants are provided the
opportunity to reconvene and learn about one
• Each site will give a 15 minute presenta- another’s progress during the workshop, and
tion of its evaluation plan to give feedback to facilitators. The specific
activities include:
• Explain how to write an evaluation plan
• presentation: summary
• Assist each site develop a specific action
plan • site presentations of evaluation plans

• Address remaining questions or problems • presentation: writing an evaluation plan

• Obtain constructive feedback from par- • exercise: develop a specific action plan
ticipants about ways to improve the work-
shop • closing activities

Presentation: summary
(group level)
One facilitator should provide a brief sum- plicated issues may require individual atten-
mary presentation of the work that has been tion and/or modification of the day’s agenda.
accomplished so far, and the remaining items
for the day. The facilitator also should elicit Other facilitators should listen carefully to the
feedback from the group regarding any other summary and participant questions, identify
outstanding issues. Relatively simple questions gaps in the information provided by the pre-
or outstanding issues probably can be ad- senting facilitator, and interject as needed.
dressed during closing activities. More com-

Site presentations of evaluation plans

(group level)
During this interval, each site should present sentations are a) to provide a “positive re-
a brief overview (15 minutes) of its evalu- inforcement” opportunity for each site’s
ation plan. Presentations should follow the evaluation planning efforts, b) to build a
structure presented on page 35 of this sense of community between the sites, and
guide. As stated earlier, the primary pur- c) to increase each site’s commitment to
pose of the presentations is to cross-inform complete its evaluation through public proc-
participating sites about their evaluation lamation of its plan. In light of these pur-
plans. The secondary purposes of the pre- poses, it is important for facilitators to keep

36 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

the majority of their comments congratula- Depending on the time available, consider al-
tory and positive. lowing a few minutes at the end of each pre-
sentation for comments from other sites. This
Experience indicates that most sites will need is the only structured opportunity for partici-
assistance to keep their presentations within pants to learn about all other sites, plans, and
the 15 minute time frame. Consider appoint- cross-site feedback at this level can be very
ing a time keeper, who can watch the time useful.
and tell speakers when their time is close to
finished.

Presentation: writing an
evaluation plan (group level)
After the site presentations, a facilitator should Sites should be encouraged to complete their
present the rationale and general guidelines own written evaluation plans after they re-
for writing an evaluation plan. This informa- turn home and consult with expected users
tion is presented on pages 20 to 21 of Work- of the results. In some cases, these plans also
book 2, and summarised as an overhead will be forwarded to workshop organisers for
prototype in the back of this guide. In review.
some cases, workshop organisers might re-
quire that sites submit a written plan follow- Sites involved with the initial WHO/ UNDCP/
ing the workshop in order to receive feed- EMCDDA project were asked to submit writ-
back and/or financial support. In other cases, ten evaluation plans for review. A set of guide-
sites will be on their own to implement their lines, along with checklists for sites and re-
evaluations. Regardless, writing a structured viewers to complete, were created to
plan helps evaluators organise their thoughts encourage consistency in the writing and re-
clearly. It also serves as a reminder to every- view process. These guidelines and check-
one involved about the purpose of the evalu- lists are reproduced starting on page 74 of
ation and the questions and decisions the re- this guide for your convenience.
sults are intended to address.
Exercise:
The basic elements of an evaluation plan (ex-
plained in greater detail on page 18 of Work- After participants thought about their evalua-
book 2) are: tion plan, they should be encouraged to think
about a concrete action plan which will help
1) Background and general purpose them transfer their plan into action.

2) Programme logic model The facilitator should direct participants to

write a response to the following questions:
3) Evaluation team

4) Evaluation questions to be addressed Why is a concrete action plan helpful to ini-

tiate your evaluation?
5) Data collection strategy
Think about 2-3 concrete task that should be
6) Data management plan carried out at the beginning of your imple-
mentation?
7) Staff training
When should these task be completed? How
8) Pilot test much time will they occupy? Who will be in-
volved in completing these tasks?
9) Strategy for using results

Facilitator’s workshop guide 37

WHO/MSD/MSB 00.2j

Tell the participants to be as specific as pos- Reminder: do not ask the group who wrote
sible when thinking about these concrete specific responses. If a person wants to
steps. They also should think about feasible make himself/herself known, that is his/her
deadlines for these steps. Instruct participants choice, but the information should not be
to keep their responses anonymous. After “forced” by facilitators.
participants are finished writing, papers should
be collected and discussed.

Closing activities
This section should begin with closing com- Following the question and answer period,
ments for the workshop. Consider using the participants should be directed to complete
phrase “Creating a Healthy Culture for Evalu- the post-workshop evaluation questionnaire
ation” (see page 8 of the Framework (located on page 48 of this guide). After
Workbook; figure available as overhead the questionnaires are completed, facilitators
prototype in the back of this guide) as an should revisit the fears and expectations (writ-
organisational theme for the presentation. ten on Day 1 of the workshop). As a group,
review the fears, and determine to what ex-
This is your final opportunity to “tie up loose tent expectations were met. The outcome of
ends” with participating sites. Any outstand- this discussion can provide a qualitative evalu-
ing issues regarding continuing collaboration ation of the workshop’s success.
with sites, expectations for site evaluation re-
ports, financial support, etc., should be handled After the workshop is finished, remember to
now. conduct a final debriefing session with all fa-
cilitators (see “Preparing for Facilitation” on
This also is an opportunity for participants to Page 13). Also take the time to collect and
pose remaining questions to facilitators. Pos- organise facilitator notes for each site (see
sible questions could include general aspects page 15).
of evaluation planning or implementation, or
something specific to their site or evaluation
plan.

38 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Problems and solutions

Experience indicates that various kinds of
problems might arise during the workshop. A list of
common problems and suggested solutions are
outlined in the section below.

Problem: There are significant language barriers

Solution: In situations with significant lan- ful. If participants are still having trouble fol-
guage barriers, remember to speak slowly, and lowing key ideas, refer them to relevant sec-
avoid complicated sentence structure and/or tions of the workbooks, where they can read
vocabulary. Write important ideas using over- at their own pace. Finally, encourage partici-
head transparencies, chalk boards, and flip pants to use their “louder” and “slower” cards
charts. Pictures or diagrams also can be use- (see page 27) whenever needed.

Problem: Participants are not present for the entire workshop

Solution: Absences from the workshop schedule and triplet work as rapidly as pos-
hinder evaluation planning and should be sible, to foster cross-site collaboration and
avoided if possible. In situations where ab- collective learning. A designated facilitator
sences are unavoidable, it is better if at least should work intensively to cover material that
two people from a site are present, so that they missed in order to achieve this goal.
they can discuss during the workshop and “Mini-presentations” can be held while other
make presentations to others upon their re- sites are working on planning activities. As-
turn. Sites that arrive late to the workshop signments can be given for evening work to
should be moved forward into the mainstream help them catch up quickly to other sites.

Problem: Participants become too tired to work effectively.

Solution: The workshop schedule is demand- free to rest and relax. While in session, try to
ing, so it is no surprise that some participants keep the tone of the workshop cheerful. For
become tired after a few days. Remember to variety, move around the room while mak-
schedule breaks throughout each day. Even ing presentations, and make use of multiple
a few minutes to stand and/or walk around teaching tools (overhead projector, flip
the room can make a big difference. If tired- chart, chalk board). Remember to minimise
ness is a big problem, consider rescheduling “lectures”, and focus on involving partici-
your time so participants will have a half-day pants in discussions.

Facilitator’s workshop guide 39

WHO/MSD/MSB 00.2j

Problem: Facilitators are too tired to meet at the end of the day.
Solution: Like participants, facilitators’ feel- some drawbacks, though: morning meetings
ings of tiredness are often related to mental are frequently rushed, and by the next morning,
exertion. Yet, it is very important for facilita- facilitators will not remember as clearly what
tors to meet each day to discuss how their happened the day before. For these reasons,
sites are progressing, and to get suggestions immediate daily debriefing is preferable. Try to
from their team about the day ahead. For make the meeting as enjoyable as possible. Meet
these reasons, daily facilitator meetings make in a picturesque location, or consider going for a
a large difference in how well the workshop walk as you talk together. In addition to work-
proceeds. If you are too tired in the evenings, shop practicalities, discuss your frustrations,
morning meetings are an option. There are fears, or anything else that is concerning you.

Problem: Participants are not working well together at the site level.
Solution: “Not working well together” can together with fellow site participants. This
be defined in multiple ways. In some cases, facilitator could raise some hypothetical
participants will prefer to work separately problems (like those mentioned above) as
from one another, thus eliminating meaning- examples, then discuss how they interfere
ful site level discussions. In other cases, one with planning a meaningful evaluation. Once
participant will dominate with his/her ideas or the site exercises are underway, facilitators
opinions, while others remain silent. In yet should remain alert for possible problems,
other situations, site participants will disagree discuss them with their co-facilitators, and
strongly with one another about the best intervene as appropriate. If you need to in-
course of action. A certain amount of vari- tervene, consider using a straightforward
ability in how people relate to each other is to approach, such as “In my experience, sites
be expected, especially cross-culturally. Fa- develop better evaluation plans when all
cilitators should be respectful of cultural and participants discuss their views openly. Per-
personal differences, yet be aware of situa- haps we can try that now. X, what do you
tions that interfere with the goal of high qual- think about this question?” In other cases,
ity evaluation planning. Before site level ex- a more subtle approach will be more suit-
ercises begin, it might be helpful for one able, such as “X, I noticed that you haven’t
facilitator to make a few comments at the said anything about this question yet. What
group level about the importance of working are your views?”

Problem: Participants react negatively to a facilitator.

Solution: Your age, gender, nationality, ap- less of the specific issue, the key is to antici-
pearance, experience, and/or behaviour can pate these potential problems, to seek back-
sometimes be the focus of negative reactions ground information on the cultural practices
from participants. As a facilitator, it is impor- of participants’ countries in advance of the
tant to understand these potential pitfalls, and workshop, and to be willing to adjust your ap-
to adjust your approach accordingly. Regard- proach as needed.

Problem: Sites are not keeping up with the workshop schedule.

Solution: If most or all sites are behind (see above). Address these problems as
schedule, you should re-examine your needed. In any event, encourage sites to stay
agenda and consider whether you are mov- with the agenda as much as possible. Even if
ing through the material too quickly. More they are not finished completely with an ex-
typically, however, you will find that some ercise, they should move forward to the next
sites simply work through the material more steps as the agenda suggests. Final decisions
quickly than others. For sites that are be- will need to be taken after the workshop, and
hind schedule, assess whether participants unfinished exercises also can be completed
are having difficulties working well together at that time.

40 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Problem: Sites are moving through the material too quickly.

Solution: The potential issues and solutions to ing fully each topic. It is possible that these
this problem are similar to those listed above. If sites are being dominated by a single mem-
nearly all sites are working ahead of schedule, it ber, thus curtailing meaningful discussion. Al-
is possible that your agenda allows too much ternatively, all participants from a certain site
time. If, on the other hand, only certain sites might be treating the discussions too superficially.
are working ahead of schedule, you should As a facilitator, it is your role to assess these
assess whether these participants are discuss- issues and intervene appropriately.

Problem: Sites’ evaluation questions are not “meaningful”, or linked directly to

programme objectives listed in the programme logic model.
Solution: The determination of what is sites’ plans to your team, and get feedback
“meaningful” is highly subjective. But for the as to whether their evaluation questions are
purposes of this project, a meaningful evalu- sufficiently meaningful. If a question is not
ation question is defined as one that is linked thought to be meaningful, you should raise this
directly to assessing whether programme ob- issue with the site the next day. Tell the site
jectives are being met. As a facilitator, it is participants that you are confused about how
your role to ensure that sites have chosen their question(s) will improve their programme,
questions that are meaningful according to this and ask them to explain this to you. Their re-
definition. One way to help yourself is to meet sponse will help you determine whether you have
daily with your co-facilitators to discuss sites’ missed an important justification, or whether the
progress. In this context, you can present your site needs to adjust its plans.

Problem: Sites’ evaluation questions are not feasible.

Solution: In some cases, sites will create you should listen to their plans with their
evaluation plans that are not reasonable, given unique situation in mind. If you think that there
their resources (time, expertise, funding). They is a significant problem, you should raise this
might feel pressured to “answer it all” with with the site participants. It is often useful to
one evaluation, or might have an unrealistic take a non-threatening approach by telling
perception of their limitations. As a facilita- them that you are confused, and asking them
tor, you should assist sites to choose evalua- to explain to you how they will be able to ac-
tions that are feasible. The first step is to get complish their plan with their available re-
a good understanding of each site’s available sources. Once they start explaining, they will
resources (accomplished during Day 2). Later, usually see the problem for themselves.

Facilitator’s workshop guide 41

WHO/MSD/MSB 00.2j

Annex 1
How to describe your
local treatment
How to Describe your Local Treatment is a
structured format for workshop participants to use
in advance of the workshop. Please see page 15 of
this guide for more information about its use.

In preparation for the upcoming workshop, includes both specialised and generalist
we would like you to prepare a two to three services and agencies.
page summary of your current substance
use treatment activity, service, agency, or Depending on the size of your clinical ser-
network. The main purpose of the summary vices, you may or may not have a service,
is to help us gain a better sense of your agency, or system within your setting. You
clinical services, so that we can help you should focus your summary on the level of
design an evaluation plan that is well-suited care that is available, and also at the level
for your situation. you want to evaluate.

A treatment activity involves a particular To help guide the preparation of your sum-
treatment modality (e.g., cognitive- mary, please follow the guidelines below:
behavioural therapy).
• Make sure that you keep your summary
A treatment service (e.g., outpatient to three pages or less. If you have a large
care) can include more than one treatment treatment service or system, you will
activity or modality (e.g., cognitive- need to choose the most important in-
behavioural therapy, relapse prevention, formation to present. Remember that the
social skills training). purpose of the presentation is to
familiarise us with your overall situation.
A treatment agency may offer more than Undue details are not necessary.
one type of treatment service (e.g., detoxi-
fication, inpatient, outpatient and continu- • Your accuracy is important to us.
ing care). Please answer our questions as cor-
rectly as possible.
A community treatment system is com-
prised of many different services, agen- • Use the following questions to guide the
cies and treatment settings. It involves development of your summary.
people with substance use disorders and

42 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

General substance use issues structured and provided on an as-

needed basis?
1 What are the major substance use prob-
lems requiring treatment in your region? - If you have structured activities, please
describe them in terms of the target
2 What is the current drug policy of your patient population, length of
region? (Include pertinent information programme, and the typical course of
about criminal laws and treatment policies one patient through the programme.
for those who use illegal substances.)
2 Who are your typical patients?
3 What are the government’s and/or
community’s attitudes towards conducting - Age
treatment evaluation and improving sub-
stance use treatment services? - Gender

- How many served

Overview of clinical services
- Common substance(s) used
1 Briefly overview your local clinical ser-
vices. - Common types of substance-related
disorders/problems
2 Are the clinical services private or public?

3 How are your clinical services related to Goals of treatment

the national health system and/or general
medical services in your region? What are the overall goals of your clinical
services? (For example: reduce patients? sub-
4 How are your clinical services related to stance use, assist patients to obtain employ-
the general community? ment, prevent children from initiating sub-
stance use, etc.)

Resources
Current evaluation efforts
1 Who are your employees?
Is evaluation happening now? If yes, please
- educational background describe current evaluation efforts.
- roles
- how many
Costs and benefits of
evaluation
2 What kind of facilities are available (of-
fice space, hospital beds, etc.)? 1 What are the main barriers to doing treat-
ment evaluations in your setting?
3 What is the annual operating budget?
2 What are potential positive benefits for
4 Are waiting lists common? How long is doing treatment evaluations in your
the waiting list on average? setting?

Activities Optional questions

1 Describe the type(s) of clinical services 1 If you could change one thing about your
that you provide. clinical services, what would it be?

- Do you have structured substance 2 What are the largest challenges facing your
use activities, or is clinical care un- clinical services?

Facilitator’s workshop guide 43

WHO/MSD/MSB 00.2j

Annex 2
Questionnaires
The following pages contain five participant
questionnaires. The purpose of each is described
below. For more information about the evaluation
component of this project, please see page 11.
The pre-workshop evaluation question- by people who plan to use the workbooks,
naire (page 80) can be sent to participants but who are unable to attend a work-
in advance of the workshop, or completed shop. This questionnaire should be com-
on Day 1. The advantage of sending question- pleted and returned before the workbooks
naires in advance (and asking for their return are released.
in advance) is that they can provide valuable
information about the participants that can be The six month evaluation questionnaire
useful for workshop planning. (page 96) is designed to be completed six
months after the workshop and/or re-
The post-workshop evaluation question- ceipt of the workbooks.
naire (page 86) is designed for completion by
workshop participants on Day 5 of the workshop. The twelve month evaluation question-
naire (page 109) is designed to be com-
The pre-workbook evaluation question- pleted 12 months after the workshop and/
naire (page 91) is designed for completion or receipt of the workbooks.

Pre-workshop evaluation questionnaire

This workshop and workbook series are be- after using them in their own settings. Your
ing evaluated for their effectiveness in help- responses will help us to understand your
ing people like you to conduct your evalua- needs and whether our workshop and work-
tions of substance use treatment. Our goal is book series is helpful to you.
to make future versions as useful as possible,
and you are an important part of this process. In addition to this questionnaire, you will be
asked to complete a follow up questionnaire
All workshop participants are requested to about the workshop and its materials at
complete evaluations at the beginning and end the end of the week. Also, follow up ques-
of the training programme. In addition, we are tionnaires about the workbook series will
requesting that users of our workbook series be mailed to you in about six months and 12
complete questionnaires about the workbooks months.

44 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Background Information

Name

Title

Name of your
institution

Address

Telephone number

Fax number

E-mail address

Your professional
background

Please check ALL boxes that describe your employment activities:

researcher

clinician (direct patient care)

treatment service or network administrator

government planner or policy maker

other

Please check ALL boxes that describe your anticipated role(s) in local evaluation
activities:

evaluation planning

oversight of evaluation implementation

data collection

data analysis

preparation of evaluation report

implementation of changes based on evaluation results

other

Facilitator’s workshop guide 45

WHO/MSD/MSB 00.2j

Questions about workshop

Please answer the
following questions Date of workshop: Location of workshop:
as honestly and
accurately as you
can. Respond with
what you really think
and what you really How helpful do you think this workshop will be in assisting you to plan your evaluation
know, not with what of substance use treatment?
you “should” think
or “should” know. none very little somewhat very much extremely

How helpful do you think this workshop will be in assisting you to implement your
evaluation of substance use treatment?

none very little somewhat very much extremely

What are your concerns or worries about the workshop?

Questions about workbook series

How helpful do you think this workbook series will be in assisting you to plan your
evaluation of substance use treatment?

none very little somewhat very much extremely

How helpful do you think this workbook series will be in assisting you to implement
your evaluation of substance use treatment?

none very little somewhat very much extremely

What are your concerns or worries about the workbook series?

How would you describe your knowledge of substance use disorders and their
treatment?

none very little some a lot extensive

How would you describe your knowledge of how to plan treatment evaluations?

none very little some a lot extensive

How would you describe your knowledge of how to implement treatment evaluations?

none very little some a lot extensive

46 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Rate the extent to which you agree with the following statements:
Strongly Disagree Neutral Agree Strongly
disagree agree
• Treatment evaluations are not needed to know
if substance use treatments are effective.
• Most patients with substance use problems
will not change their behaviour, regardless
of the quality of the treatment programme.
• Treatment evaluations take away limited re-
sources from direct clinical care.
• Treatment evaluations frequently result in
cuts to programme funds.
• We don’t have the experience to do treat-
ment evaluations in our setting.
• We don’t have the money to do treatment
evaluations in our setting.
• We don’t have the time to do treatment
evaluations in our setting.

To what extent is your current work related to delivering substance use disorder
treatments?

none very little somewhat very much all the time

To what extent is your current work related to conducting treatment evaluations?

none very little somewhat very much all the time

Indicate whether you have ever personally completed the following:

No Ye s
• Planned any kind of research study

• Planned a treatment evaluation

• Collected data from any source

• Collected data from patients

• Entered data into a data book

• Entered data into a computer file

• Analysed data by hand

• Analysed data using a computer

• Written an evaluation report

• Published results in a scientific journal

• Presented results orally

Thank you!

Facilitator’s workshop guide 47

WHO/MSD/MSB 00.2j

Post-workshop evaluation
questionnaire
This workshop and its materials are being the training programme. Your responses will
evaluated for their effectiveness in helping help us to understand whether our workshop
people like you to conduct your evaluations was helpful to you, and how we can improve
of substance use treatment. Our goal is to in the future.
make future workshops as useful as possible,
and you are an important part of this process. Additional follow up questionnaires about the
workbook series will be mailed to you in
We are requesting that all workshop partici- about six months and 12 months.
pants complete questionnaires at the end of

Background information

Name

Title

Name of your
institution

Address

Telephone number

Fax number

E-mail address

Your professional
background

Questions about workshop

Date of workshop: Location of workshop:

Name of facilitator working most closely with you during workshop:

48 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Please answer the

questions on the What overall rating would you give to the workshop training programme?
following pages as
honestly and
1 2 3 4 5
accurately as you
can. Respond with Lowest Highest
what you really
think and feel, not
with what you Overall, the workshop training programme:
“should” think or Strongly Disagree Neutral Agree Strongly
disagree agree
“should” feel.
• Allocated time appropriately, given the to-
tal time available

• Covered the subject matter in adequate

detail

• Used effective teaching methods

• Presentations were helpful

• Exercises were helpful

• Helped us to plan a better evaluation

Would you recommend this workshop training programme to a colleague?

no yes

Why or why not?

Complete the following ratings about the facilitators.

Overall, how helpful were the facilitators?

none very little somewhat very much extremely

The facilitators:
Strongly Disagree Neutral Agree Strongly
disagree agree
• Had sufficient knowledge

• Had sufficient teaching skills

• Created a positive learning atmosphere

• Helped us to plan a better evaluation

Additional comments about the facilitators:

Facilitator’s workshop guide 49

WHO/MSD/MSB 00.2j

Please finish the following sentences.

• The most useful aspect of the workshop training programme was:

• The least useful aspect of the workshop training programme was:

• Unnecessary information was given about:

• I wish more information were provided about:

• Additional comments or suggestions:

Thank you!

Pre-workbook questionnaire
Thank you for your interest in the WHO/ and after using them in their own settings.
UNDCP/ EMCDDA workbook series project Your responses will help us to understand your
on the costs and effects of treatment for psy- needs and to evaluate whether our workbook
choactive substance use disorders. These series is helpful to you.
workbooks are being evaluated for their ef-
fectiveness in helping people like you to con- Follow up questionnaires will be mailed to you
duct your evaluations of substance use treat- six months and 12 months following your re-
ment. Our goal is to make future versions of ceipt of the workbook series.
these workbooks as useful as possible, and
you are an important part of this process. Please complete the following questions and
return this form to the address listed below.
We are requesting that all users of our work- We also welcome any other comments, ques-
book series complete questionnaires before tions, or suggestions that you might have.

50 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Background Information

Name

Title

Name of your
institution

Address

Telephone number

Fax number

E-mail address

Your professional
background

Please check ALL boxes that describe your employment activities:

researcher

clinician (direct patient care)

treatment service or network administrator

government planner or policy maker

other

Please check ALL boxes that describe your anticipated role(s) in local evaluation
activities:

evaluation planning

oversight of evaluation implementation

data collection

data analysis

preparation of evaluation report

implementation of changes based on evaluation results

other

Facilitator’s workshop guide 51

WHO/MSD/MSB 00.2j

Please answer the

questions on the
following pages as How helpful do you think this workbook series will be in assisting you to plan your
honestly and evaluation of substance use treatment?
accurately as you
can. Respond with
none very little somewhat very much extremely
what you really think
and what you really
know, not with what How helpful do you think this workbook series will be in assisting you to implement
you “should” think your evaluation of substance use treatment?
or “should” know.
none very little somewhat very much extremely

What are your concerns or worries about the workbook series?

How would you describe your knowledge of substance use disorders and their
treatment?

none very little some a lot extensive

How would you describe your knowledge of how to plan treatment evaluations?

none very little some a lot extensive

How would you describe your knowledge of how to implement treatment

evaluations?

none very little some a lot extensive

52 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Rate the extent to which you agree with the following statements:
Strongly Disagree Neutral Agree Strongly
disagree agree
• Treatment evaluations are not needed to know
if substance use treatments are effective.
• Most patients with substance use problems
will not change their behaviour, regardless
of the quality of the treatment programme.
• Treatment evaluations take away limited re-
sources from direct clinical care.
• Treatment evaluations frequently result in
cuts to programme funds.
• We don’t have the experience to do treat-
ment evaluations in our setting.
• We don’t have the money to do treatment
evaluations in our setting.
• We don’t have the time to do treatment
evaluations in our setting.

To what extent is your current work related to delivering substance use disorder
treatments?

none very little somewhat very much all the time

To what extent is your current work related to conducting treatment evaluations?

none very little somewhat very much all the time

Indicate whether you have ever personally completed the following:

No Ye s
• Planned any kind of research study

• Planned a treatment evaluation

• Collected data from any source

• Collected data from patients

• Entered data into a data book

• Entered data into a computer file

• Analysed data by hand

• Analysed data using a computer

• Written an evaluation report

Please return this
• Published results in a scientific journal questionnaire to the
address listed on
• Presented results orally the first page.
Thank you!

Facilitator’s workshop guide 53

WHO/MSD/MSB 00.2j

Six month follow-up questionnaire

Our records indicate that you received the ful as possible, and you are an important
workbook series on the costs and effects part of this process.
of treatment for psychoactive substance
use disorders six months ago. Accordingly, Please complete the following questions as
we are enclosing a questionnaire to help us soon as possible and return this form to the
evaluate whether our workbook series has address listed below. We also welcome any
been helpful to date. Our goal is to make other comments, questions, or suggestions
future versions of these workbooks as use- that you might have.

Background information

Name

Title

Name of your
institution

Has your contact information changed? If yes, complete the following:

Address

Telephone number

Fax number

E-mail address

Your professional
background

Did you attend a workbook series training workshop?

no yes

If no, how did you receive the workbook materials?

If yes, what was the date and location of the workshop you attended?

54 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Please check ALL boxes that describe your anticipated role(s) in local evaluation
activities:

evaluation planning

oversight of evaluation implementation

data collection

data analysis

preparation of evaluation report

implementation of changes based on evaluation results

other

Please answer the

following questions How would you describe your knowledge of substance use disorders and their
as honestly and treatment?
accurately as you
can. Respond with none very little some a lot extensive
what you really
think and what you
How would you describe your knowledge of how to plan treatment evaluations?
really know, not with
what you “should”
none very little some a lot extensive
think or “should”
know.
How would you describe your knowledge of how to implement treatment evaluations?

none very little some a lot extensive

Rate the extent to which you agree with the following statements:
Strongly Disagree Neutral Agree Strongly
disagree agree
• Treatment evaluations are not needed to know
if substance use treatments are effective.
• Most patients with substance use problems
will not change their behaviour, regardless
of the quality of the treatment programme.
• Treatment evaluations take away limited re-
sources from direct clinical care.
• Treatment evaluations frequently result in
cuts to programme funds.
• We don’t have the experience to do treat-
ment evaluations in our setting.
• We don’t have the money to do treatment
evaluations in our setting.
• We don’t have the time to do treatment
evaluations in our setting.

Facilitator’s workshop guide 55

WHO/MSD/MSB 00.2j

To what extent is your current work related to delivering substance use disorder
treatments?

none very little somewhat very much all the time

To what extent is your current work related to conducting treatment evaluations?

none very little somewhat very much all the time

Indicate whether you have ever (including the past 6 months) personally completed
the following:
No Ye s
• Planned any kind of research study

• Planned a treatment evaluation

• Collected data from any source

• Collected data from patients

• Entered data into a data book

• Entered data into a computer file

• Analysed data by hand

• Analysed data using a computer

• Written an evaluation report

• Published results in a scientific journal

• Presented results orally

Has the workbook series contributed to your completion of these activities?

no yes

If yes, in what way?

56 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Indicate whether you completed the following tasks within the last six months. For
each “no” response, please indicate the reason(s) you did not complete the task.

No Ye s Reason(s) for not completing

• Read the Framework workbook

• Read Workbook 1 (Planning Evalua-

tions)

• Read Workbook 2 (Implementing

Evaluations)

• Read any of the Specialised Workbooks

(Workbooks 3 - 8)

• Completed “It’s your turn” Exercises

• Formed a local evaluation planning

workgroup

• Used the workbook guidelines to plan

an evaluation

• Used the workbook guidelines to imple-

ment an evaluation

• Finished an evaluation

• Reported the evaluation findings

• Used the evaluation findings to guide

changes in your treatment services

• Planned a second evaluation based on

results from first evaluation

Have you engaged in any OTHER form of evaluation training activity (in addition to
this workbook series) over the past 6 months?

no yes

If yes, what type of activity?

Read book(s)

Attended training workshop(s)

Other

Facilitator’s workshop guide 57

WHO/MSD/MSB 00.2j

If you have planned and/or implemented an evaluation using the workbook series, indi-
cate the evaluation type:

Needs evaluation

Process evaluation

Cost evaluation

Client satisfaction evaluation

Outcome evaluation

Economic evaluation

Please describe your evaluation here:

Please base your responses to the following questions on your experiences with the
workbook series to date.

Overall, how helpful was this workbook series in assisting you to plan your evaluation
of substance use treatment?

none very little somewhat very much extremely

Overall, how helpful was this workbook series in assisting you to implement your
evaluation of substance use treatment?

none very little somewhat very much extremely

The workbook series: Strongly Disagree Neutral Agree Strongly

disagree agree
• was clear to understand

• was easy to use

• was useful for our local needs

• covered all the relevant information

we needed
• terminology was easy to understand

• writing style was clear

58 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Would you recommend this workbook series to a colleague?

no yes

Why or why not?

Complete the following ratings about the Introductory/Framework Workbook.

I read the Framework Workbook:

no yes

Overall, how helpful was the Framework Workbook?

none very little somewhat very much extremely

Overall, the Framework Workbook:

Strongly Disagree Neutral Agree Strongly
disagree agree
• was clear to understand

• was easy to use

• was useful for our local needs

• covered all the relevant information we

needed
• contained terminology that was easy to
understand
• used a clear writing style

• helped us to plan and/or implement a

better evaluation

Additional comments about the Framework Workbook:

Facilitator’s workshop guide 59

WHO/MSD/MSB 00.2j

Complete the following ratings about the Foundation Workbooks (Workbooks 1 and 2).

Check the workbook(s) that you read.

No Ye s
• Workbook 1(planning evaluations)

• Workbook 2 (implementing evaluations)

Overall, how helpful were the Foundation Workbooks (Workbooks 1 and 2)?

none very little somewhat very much extremely

Workbooks 1 and 2: Strongly Disagree Neutral Agree Strongly

disagree agree
• were clear to understand

• were easy to use

• were useful for our local needs

• covered all the relevant information we

needed
• contained terminology that was easy to un-
derstand
• used a clear writing style

• helped us to plan and/or implement a bet-

ter evaluation

Additional comments about the Foundation Workbooks:

Complete the following ratings about the Specialised Workbooks (Workbooks 3 - 8).

Check the workbook(s) that you read.

No Ye s
• Workbook 3 (needs evaluations)

• Workbook 4 (process evaluations)

• Workbook 5 (cost evaluations)

• Workbook 6 (client satisfaction evaluations)

• Workbook 7 (outcome evaluations)

• Workbook 8 (economic evaluations)

60 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Overall, how helpful were the Specialised Workbooks that you read?

none very little somewhat very much extremely

The Specialised workbooks that I read:

Strongly Disagree Neutral Agree Strongly
disagree agree
• were clear to understand

• were easy to use

• were useful for our local needs

• covered all the relevant information we

needed
• contained terminology that was easy to
understand
• used a clear writing style

• helped us to plan and/or implement a

better evaluation

Additional comments about the Specialised Workbooks:

Complete the following ratings about the “It’s your turn” exercises.

Check the workbook(s) in which you completed “It’s your turn” exercises.

No Ye s
• Framework Workbook

• Workbook 1 (planning evaluations)

• Workbook 2 (implementing evaluations)

• Workbook 3 (needs evaluations)

• Workbook 4 (process evaluations)

• Workbook 5 (cost evaluations)

• Workbook 6 (client satisfaction evaluations)

• Workbook 7 (outcome evaluations)

• Workbook 8 (economic evaluations)

Facilitator’s workshop guide 61

WHO/MSD/MSB 00.2j

Overall, how helpful were the “It’s your turn” exercises?

none very little somewhat very much extremely

The exercises that I completed:

Strongly Disagree Neutral Agree Strongly
disagree agree
• were clear to understand

• were easy to use

• were useful for our local needs

• enhanced my understanding of the

workbook text
• helped our evaluation group work
together more effectively
• helped us plan and/or implement a better
evaluation

Additional comments about the “It’s your turn” exercises:

Complete the following ratings about the case examples (located in the back of most of
the workbooks).

Check the workbook(s) in which you read the case example(s).

No Ye s
• Workbook 1 (planning evaluations)

• Workbook 3 (needs evaluations)

• Workbook 4 (process evaluations)

• Workbook 5 (cost evaluations)

• Workbook 6 (client satisfaction evaluations)

• Workbook 7 (outcome evaluations)

• Workbook 8 (economic evaluations)

Overall, how helpful were the case examples?

none very little somewhat very much extremely

62 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

The case examples that I read:

Strongly Disagree Neutral Agree Strongly
disagree agree
• were clear to understand

• were easy to use

• were useful for our local needs

• enhanced my understanding of the

workbook text
• terminology was easy to understand

• writing style was clear

• helped us to plan and/or implement a

better evaluation

Additional comments about the case examples:

Please finish the following sentences.

The most useful aspect of the workbook series was:

The least useful aspect of the workbook series was:

Unnecessary information was given about:

I wish more information were provided about:

Additional comments or suggestions:

Thank you! Please return this questionnaire to the address listed on the first page.

Facilitator’s workshop guide 63

WHO/MSD/MSB 00.2j

Twelve month follow-up questionnaire

Our records indicate that you received the as useful as possible, and you are an im-
workbook series on the costs and effects portant part of this process.
of treatment for psychoactive substance
use disorders twelve months ago. Accord- Please complete the following questions as
ingly, we are enclosing a questionnaire to soon as possible and return this form to the
help us evaluate whether our workbook address listed below. We also welcome any
series has been helpful to you. Our goal is other comments, questions, or suggestions
to make future versions of these workbooks that you might have.

Background information

Name

Title

Name of your
institution

Has your contact information changed? If yes, complete the following:

Address

Telephone number

Fax number

E-mail address

Your professional
background

Did you attend a workbook series training workshop?

no yes

If no, how did you receive the workbook materials?

If yes, what was the date and location of the workshop you attended?

64 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Please check ALL boxes that describe your anticipated role(s) in local evaluation
activities:

evaluation planning

oversight of evaluation implementation

data collection

data analysis

preparation of evaluation report

implementation of changes based on evaluation results

other

Please answer the

following questions How would you describe your knowledge of substance use disorders and their
as honestly and treatment?
accurately as you
can. Respond with none very little some a lot extensive
what you really
think and what you
How would you describe your knowledge of how to plan treatment evaluations?
really know, not
with what you
none very little some a lot extensive
“should” think or
“should” know.
How would you describe your knowledge of how to implement treatment evaluations?

none very little some a lot extensive

Rate the extent to which you agree with the following statements:
Strongly Disagree Neutral Agree Strongly
disagree agree
• Treatment evaluations are not needed to know
if substance use treatments are effective.
• Most patients with substance use problems
will not change their behaviour, regardless
of the quality of the treatment programme.
• Treatment evaluations take away limited re-
sources from direct clinical care.
• Treatment evaluations frequently result in
cuts to programme funds.
• We don’t have the experience to do treat-
ment evaluations in our setting.
• We don’t have the money to do treatment
evaluations in our setting.
• We don’t have the time to do treatment
evaluations in our setting.

Facilitator’s workshop guide 65

WHO/MSD/MSB 00.2j

To what extent is your current work related to delivering substance use disorder
treatments?

none very little somewhat very much all the time

To what extent is your current work related to conducting treatment evaluations?

none very little somewhat very much all the time

Indicate whether you have ever (including the past 12 months) personally completed
the following:
No Ye s
• Planned any kind of research study

• Planned a treatment evaluation

• Collected data from any source

• Collected data from patients

• Entered data into a data book

• Entered data into a computer file

• Analysed data by hand

• Analysed data using a computer

• Written an evaluation report

• Published results in a scientific journal

• Presented results orally

Has the workbook series contributed to your completion of these activities?

no yes

If yes, in what way?

66 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Indicate whether you completed the following tasks within the last 12 months. For each
“no” response, please indicate the reason(s) you did not complete the task.

No Ye s Reason(s) for not completing

• Read the Framework workbook

• Read Workbook 1 (Planning Evalua-

tions)

• Read Workbook 2 (Implementing

Evaluations)

• Read any of the Specialised Workbooks

(Workbooks 3 - 8)

• Completed “It’s your turn” Exercises

• Formed a local evaluation planning

workgroup

• Used the workbook guidelines to plan

an evaluation

• Used the workbook guidelines to imple-

ment an evaluation

• Finished an evaluation

• Reported the evaluation findings

• Used the evaluation findings to guide

changes in your treatment services

• Planned a second evaluation based on

results from first evaluation

Have you engaged in any OTHER form of evaluation training activity (in addition to
this workbook series) over the past 12 months?

no yes

If yes, what type of activity?

Read book(s)

Attended training workshop(s)

Other

Facilitator’s workshop guide 67

WHO/MSD/MSB 00.2j

If you have planned and/or implemented an evaluation using the workbook series, indi-
cate the evaluation type:

Needs evaluation

Process evaluation

Cost evaluation

Client satisfaction evaluation

Outcome evaluation

Economic evaluation

Please describe your evaluation here:

Please base your responses to the following questions on your experiences with the
workbook series to date.

Overall, how helpful was this workbook series in assisting you to plan your evaluation
of substance use treatment?

none very little somewhat very much extremely

Overall, how helpful was this workbook series in assisting you to implement your
evaluation of substance use treatment?

none very little somewhat very much extremely

The workbook series: Strongly Disagree Neutral Agree Strongly

disagree agree
• was clear to understand

• was easy to use

• was useful for our local needs

• covered all the relevant information

we needed
• terminology was easy to understand

• writing style was clear

68 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Would you recommend this workbook series to a colleague?

no yes

Why or why not?

Complete the following ratings about the Introductory/Framework Workbook.

I read the Framework Workbook:

no yes

Overall, how helpful was the Framework Workbook?

none very little somewhat very much extremely

Overall, the Framework Workbook:

Strongly Disagree Neutral Agree Strongly
disagree agree
• was clear to understand

• was easy to use

• was useful for our local needs

• covered all the relevant information we

needed
• contained terminology that was easy to
understand
• used a clear writing style

• helped us to plan and/or implement a

better evaluation

Additional comments about the Framework Workbook:

Facilitator’s workshop guide 69

WHO/MSD/MSB 00.2j

Complete the following ratings about the Foundation Workbooks (Workbooks 1 and 2).

Check the workbook(s) that you read.

No Ye s
• Workbook 1(planning evaluations)

• Workbook 2 (implementing evaluations)

Overall, how helpful were the Foundation Workbooks (Workbooks 1 and 2)?

none very little somewhat very much extremely

Workbooks 1 and 2: Strongly Disagree Neutral Agree Strongly

disagree agree
• were clear to understand

• were easy to use

• were useful for our local needs

• covered all the relevant information we

needed
• contained terminology that was easy to un-
derstand
• used a clear writing style

• helped us to plan and/or implement a bet-

ter evaluation

Additional comments about the Foundation Workbooks:

Complete the following ratings about the Specialised Workbooks (Workbooks 3 - 8).

Check the workbook(s) that you read.

No Ye s
• Workbook 3 (needs evaluations)

• Workbook 4 (process evaluations)

• Workbook 5 (cost evaluations)

• Workbook 6 (client satisfaction evaluations)

• Workbook 7 (outcome evaluations)

• Workbook 8 (economic evaluations)

70 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Overall, how helpful were the Specialised Workbooks that you read?

none very little somewhat very much extremely

The Specialised workbooks that I read:

Strongly Disagree Neutral Agree Strongly
disagree agree
• were clear to understand

• were easy to use

• were useful for our local needs

• covered all the relevant information we

needed
• contained terminology that was easy to
understand
• used a clear writing style

• helped us to plan and/or implement a

better evaluation

Additional comments about the Specialised Workbooks:

Complete the following ratings about the “It’s your turn” exercises.

Check the workbook(s) in which you completed “It’s your turn” exercises.

No Ye s
• Framework Workbook

• Workbook 1 (planning evaluations)

• Workbook 2 (implementing evaluations)

• Workbook 3 (needs evaluations)

• Workbook 4 (process evaluations)

• Workbook 5 (cost evaluations)

• Workbook 6 (client satisfaction evaluations)

• Workbook 7 (outcome evaluations)

• Workbook 8 (economic evaluations)

Facilitator’s workshop guide 71

WHO/MSD/MSB 00.2j

Overall, how helpful were the “It’s your turn” exercises?

none very little somewhat very much extremely

The exercises that I completed:

Strongly Disagree Neutral Agree Strongly
disagree agree
• were clear to understand

• were easy to use

• were useful for our local needs

• enhanced my understanding of the

workbook text
• helped our evaluation group work
together more effectively
• helped us plan and/or implement a better
evaluation

Additional comments about the “It’s your turn” exercises:

Complete the following ratings about the case examples (located in the back of most of
the workbooks).

Check the workbook(s) in which you read the case example(s).

No Ye s
• Workbook 1 (planning evaluations)

• Workbook 3 (needs evaluations)

• Workbook 4 (process evaluations)

• Workbook 5 (cost evaluations)

• Workbook 6 (client satisfaction evaluations)

• Workbook 7 (outcome evaluations)

• Workbook 8 (economic evaluations)

Overall, how helpful were the case examples?

none very little somewhat very much extremely

72 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

The case examples that I read:

Strongly Disagree Neutral Agree Strongly
disagree agree
• were clear to understand

• were easy to use

• were useful for our local needs

• enhanced my understanding of the

workbook text
• terminology was easy to understand

• writing style was clear

• helped us to plan and/or implement a

better evaluation

Additional comments about the case examples:

Please finish the following sentences.

The most useful aspect of the workbook series was:

The least useful aspect of the workbook series was:

Unnecessary information was given about:

I wish more information were provided about:

Additional comments or suggestions:

Thank you! Please return this questionnaire to the address listed on the first page.

Facilitator’s workshop guide 73

WHO/MSD/MSB 00.2j

Annex 3
Written evaluation plan
criteria and checklists

General criteria for sites

To help ensure that you are covering all im- and priorities. A meaningful evaluation plan
portant areas in your written evaluation is one that is logically related to the
plan, we have developed a specific struc- programme logic model. A feasible evalua-
ture for what to include. Following this tion plan is one that is simple, reasonable for
structure will help you to organise your the budget and other available resources, and
thoughts clearly. It also will demonstrate to achievable within the one-year time frame.
us that you have thought through the im-
portant aspects of your proposed evalua- As you prepare your written evaluation
tion, and that you are ready and able to com- plan, use the workbooks that were given to
plete this particular evaluation. you at the workshop. The workbooks are
designed to assist you to independently de-
Remember, we want to assist you to de- velop your evaluation. They provide infor-
sign an evaluation plan that is meaningful mation and suggestions on all aspects of
and feasible, given your unique situation planning.

Structured outline for written evaluation plans

Organise your written evaluation plan according to these sections:

Section 1. Background and general purpose (approximately one-half page)

This is a summary of your evaluation plan. It sults, and how results will be used. Do not
should include a brief description of the pur- present specific details in this section; simply
pose of the evaluation, how you plan to con- provide the reader with a brief overview.
duct the evaluation, who will receive the re-

74 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Section 2. Programme logic model (one page)

This section describes the programme, ser- The model should describe:
vice, or system that you plan to evaluate in • The type(s) of services that you provide
the form of a programme logic model. Refer • Your process or implementation objectives
to Step 3 of Workbook 1 for more informa- • The short-term and long-term goals of
tion about this description. various parts of your programme.

Section 3. Evaluation team (approximately one page)

This section lists the people who will be in- Important: Indicate the person (one only,
volved with your evaluation project. For each please) who is responsible for communicat-
person, give the following information: ing with us throughout your evaluation. Pro-
vide that person’s mailing address, telephone
• Name number, and if available, fax number and email
• Professional title address.
• Educational background
• Type of experience (if any) conducting
evaluations - brief summary only
• Anticipated roles in the proposed evaluation

Section 4. Evaluation question(s) (approximately one page)

This section lists your specific evaluation ques- b) your evaluation questions should be writ-
tions. Refer to Step 5 of Workbook 1 for some ten in the form of a question, not a state-
examples of well-phrased evaluation questions. ment.

Important: c) for each evaluation question, you should

indicate what type of evaluation you are
a) your evaluation questions should follow conducting through addressing this
logically from your programme logic question.
model. In other words, each of your ques-
tions should be linked to one of your Here are two examples of how to do this:
programme’s implementation objectives or
goals in a specific way.

Programme objective or goal Evaluation question Evaluation type

Implementation objective: To What percentage of patients Process evaluation

provide initial assessments to receive initial assessments (Workbook 4)
patients in a timely manner. within 2 weeks of contacting
our programme?

Long-term goal: To reduce Is problem drinking reduced Outcome evalua-

problem drinking among pa- following participation in our tion (Workbook 7)
tients. programme?

Write your evaluation questions in this sec- Once you have identified your evaluation
tion using the same format. This will show type(s), refer to the appropriate specialised
us how your evaluation questions are linked workbook for information and examples of
to your programme logic model, and what how to complete that kind of evaluation.
type of evaluation you are conducting.

Facilitator’s workshop guide 75

WHO/MSD/MSB 00.2j

Do not phrase your question in the form of a stance use problems in our community” is not
statement. For example, “We are going to an acceptable evaluation question (because
assess the total number of people with sub- it is not a question at all).

Section 5. Data collection strategy (approximately two pages of text, not including
your data collection instruments)

This section is very important, because it is • Whether you will develop your own data
your road map for how you are going to get collection instrument. If you plan to use
from your evaluation questions to meaningful your own instrument, you need to be very
results. Many evaluators make the mistake specific about exactly what you plan to
of writing this section too vaguely, assuming ask. Include the specific questions and
that details can be decided as the evaluation scoring criteria you plan to use. Note: you
progresses. This attitude places the entire may refine your data collection instrument
evaluation in jeopardy, because details are after pilot testing, but it is still important to
forgotten frequently and/or procedures are not present your plan for specific questions
followed in a standardised manner through- and scoring criteria here.
out the evaluation.
• How you will collect the data (self-report
Follow this guideline: could an outside con- questionnaire, interview, focus group, re-
sultant read your data collection strategy sec- view of records, etc.)
tion and then be able to implement your evalu-
ation exactly as you are planning? If not, you • Who will administer the data collection
need to be more specific here. instrument(s)

Refer to Steps 6 and 7 of Workbook 1 for • How many participants you will assess
information and suggestions about creating
your data collection strategy. For each evalu- • Your sampling strategy: how you will
ation question you listed in section 4 (above), choose participants for your evaluation
you should indicate:
• Your time line for data collection (how
• Whether you will use a standardised data many weeks/months)
collection instrument. If so, indicate which
one(s), and state the specific variables
(subscales or other summary scores) you
plan to use.

Section 6. Ethical concerns (approximately one-half page of text, not including your
consent form)

This section covers relevant ethical issues. clude it here. (Some evaluations, such as
Most important, you should provide docu- those using record reviews, may not require
mentation that you have received approval a consent form.)
to conduct your evaluation from your local
ethics committee. You also should present Refer to Step 1-A of Workbook 2 for more
your plan for how you will keep participant information and suggestions for how to man-
information confidential. If you will use a age ethical issues during your evaluation. A
consent form with your participants, in- sample consent form also is presented.

Section 7. Data management plan (approximately one page)

You should explain your record keeping sys- Refer to Step 1-B of Workbook 2 for more
tem, including how you will store your data information and suggestions for how to de-
and maintain central records. velop a data management plan.

76 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Section 8. Staff training (approximately one-half page)

In this section, explain if any additional staff train all evaluation staff (data collectors,
will be needed for the evaluation. State record keepers) to conduct their jobs in a
whether you anticipate needing outside con- standardised way.
sultants, and if so, for how long and in what
capacity. Refer to Step 1-B of Workbook 2 for more
information and suggestions for how to en-
Regardless of whether new staff or consult- sure standardisation of your procedures.
ants will be needed, describe how you will

Section 9. Pilot test (approximately one-half page)

This section should describe your specific should plan to include it as part of your pilot
plans for conducting a pilot test of your evalu- testing, and then make changes to it as needed
ation. What type of a pilot test will you con- after the pilot testing.
duct, and how many people will participate in
each portion of it? If you are using a newly Refer to Step 1-C of Workbook 2 for more
developed data collection instrument, you information and suggestions.

Section 10. Data analysis (approximately one page)

This section presents an overview for how Selected information regarding data analysis
you plan to analyse your data. You should is presented in Step 3 of Workbook 2. If you
indicate the specific statistical tests you plan think that you will need further assistance to
to use to answer each of your evaluation choose specific statistical tests and/or con-
questions (section 4). State the specific duct the data analysis, state this clearly here.
analysis you are planning for each evalua- Skip descriptions of specific statistical tests if
tion question. you are unsure.

Section 11. Reporting results (approximately one-half page)

Describe how you will present your results, sual aids (e.g., bar graphs) will be used to
and the format you will use to present them. summarise results? Refer to Step 4 of
Will you present the information in oral form Workbook 2 for more information on report-
and/or written form, and what kind of vi- ing results.

Section 12. Strategy for using results (approximately one page)

This section is very important, because it ex- sults. We do not expect that this single
plains how your evaluation results can be ap- evaluation will be the only step needed to
plied to the long-term goal of improving sub- improve treatment in your setting. In real-
stance use treatment. Explain who will receive ity, multiple evaluations may be needed be-
results, and how this information could be used fore large changes can be made. However,
to make changes to improve treatment. you should explain clearly how these initial
evaluation results can serve as a beginning
Remember, evaluation is an ongoing pro- step to enhance treatment.
cess of asking questions and applying re-

Facilitator’s workshop guide 77

WHO/MSD/MSB 00.2j

Section 13. Budget (approximately one page)

If you will be receiving payment to complete • Have you indicated clearly how your re-
your evaluation, the first instalment will be sults will be used to improve substance use
authorised upon successful completion of the treatment?
written evaluation plan. “Successful comple-
tion” depends upon reviewers’ assessment of • Can your team complete the proposed
these factors: evaluation with available resources?
“Available Resources” includes money,
• Have you followed the structured outline necessary equipment and facilities, and
for written evaluation plans? Are all sec- availability of outside consultants if
tions present and complete? (If not, your needed.
plan will be returned to you for revision.)
• Can the proposed evaluation be com-
• Is the accompanying checklist completed pleted within the expected time frame?
and included with the plan?
• Are ethical concerns adequately ad-
• Are your evaluation questions related dressed?
meaningfully to your programme logic
model? • Other impressions

• Is your proposed data collection strategy It is likely that sites will need to revise their
sufficiently detailed? written evaluation plans at least one time
after reviewers have read them.
• Are your proposed data collection and data
analysis strategies adequate to answer your Given the amount of money available, de-
evaluation question(s)? scribe how you plan to allocate the funds
to complete the evaluation. If your site is
• Considering your plans for pilot testing, receiving additional funding for this evalu-
staff training, and record keeping, will your ation, include the relevant information in
data collection proceed smoothly and pro- your budget plan.
duce reliable information?

Section 14. Time line (approximately one page)

Present a time line for when you will accom- Total written evaluation plan page length:
plish each step of your evaluation. approximately 12.5 pages of text (programme
logic model, data collection instruments, and
consent form extra).

78 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Written evaluation plan checklist

This checklist is intended to assist site ation plan. Please complete this check-
participants to ensure that pertinent in- list and return it with your written evalu-
formation is included in the written evalu- ation plan.

Evaluators’ names and proposal title

Please check that the following components are included in your plan:

Section 1: Background and general purpose

Section 2: Programme logic model

• programme logic model included

Section 3: Evaluation team

• every evaluation team member included

• person responsible for communication is indicated

Section 4: Evaluation questions

• each evaluation question is linked to a specific component of the programme

logic model
• evaluation questions are written in the form of questions
• evaluation type is indicated for each evaluation question

Section 5: Data collection strategy for each evaluation question, state:

• data collection instrument (indicate whether standardised or newly developed)

• specific variables to be used
• specific questions and scoring criteria for all newly developed data collection
instruments
• method for data collection (interview, self-report, etc.)
• number of participants
• sampling strategy
• time line for data collection

Facilitator’s workshop guide 79

WHO/MSD/MSB 00.2j

Section 6: Ethical concerns

• documentation of ethics committee approval

• plan for keeping participant information confidential
• consent form for participation (if needed)

Section 7: Data management plan

• explanation of record keeping system

• explanation of data storage plan

Section 8: Staff training

• indication if additional staff will be needed

• indication if outside consults will be needed
• explanation of staff training procedures

Section 9: Pilot test

• type of pilot test

• number of participants for pilot test

Section 10: Data analysis

• specific statistical tests to be conducted, and/or

• indication of whether consultant will be needed to choose/conduct data analysis

Section 11: Reporting results

• plan for reporting results

Section 12: Strategy for using results

• who will receive results

• specific changes that could be made using results

Section 13: Budget

Section 14: Time line

80 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Written evaluation plan checklist

reviewer version
This checklist is intended to assist re- as a guide while reviewing site plans and
viewers to thoroughly assess sites’ writ- formulating recommendations.
ten evaluation plans. Use this checklist

Evaluators’ names and proposal title

Check if “yes”:
Ye s
• Is the site checklist completed and included with the plan?

• Are all 14 sections included in the plan as requested?

• Are the evaluation questions linked meaningfully to the programme logic

model?

• Is the proposed data collection strategy sufficiently detailed?

• Is the proposed data collection strategy adequate to answer the evaluation

questions?

• Considering proposed plans for pilot testing, staff training, and record keeping,
will data collection proceed smoothly and produce reliable information?

• Is the proposed data analysis strategy adequate to answer the evaluation

questions?

• Has the site indicated clearly how results will be used to improve treat-
ment?

• Can the site team complete the proposed evaluation with their available
resources? “Available Resources” includes money, necessary equipment
and facilities, qualifications and experience of evaluation team members,
and projected availability of outside consultants.

• Can the proposed evaluation be completed within the expected time frame?

• Are ethical issues adequately addressed?

Other impressions or concerns:

Facilitator’s workshop guide 81

WHO/MSD/MSB 00.2j

Annex 4
Overhead prototypes

Day 1
82 Evaluation of Psychoactive Substance Use Disorder Treatment
 WHO/MSD/MSB 00.2j

WHO/UNDCP/EMCDDA

Project on costs and

effects of treatment for
psychoactive substance
use disorders

Summary of the workbook series

Introductory Workbook
Framework Workbook
Foundation Workbooks
Workbook 1: Planning Evaluations
Workbook 2: Implementing Evaluations
Specialised Workbooks
Workbook 3: Needs Assessment Evaluations
Workbook 4: Process Evaluations
Workbook 5: Cost Evaluations
Workbook 6: Client Satisfaction Evaluations
Workbook 7: Outcome Evaluations
Workbook 8: Economic Evaluations

Facilitator’s workshop guide 83

WHO/MSD/MSB 00.2j

Fears and expectations

Working individually, list on the
provided cards:
1) your expectations about the workshop
2) your fears or concerns
Use different card colours for expectations vs. fears.
Write legibly and use large print, so your cards can
be read from a distance. Do not put your name on the
cards.

Why is treatment
evaluation important?
• Treatment services are growing
• Despite scarcity, resources are
given to ineffective treatments
• Treatment evaluation improves
quality of care and saves money for
services that are effective

84 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Pros and cons of evaluation

1. Each site should work separately and
write on flip chart paper the pros and
cons of doing treatment evaluation at its
local site. Allow 15 minutes to complete
this step.
2. Each site should meet with at the
triplet level to discuss the pros and cons
that they wrote.

Day 2
Facilitator’s workshop guide 85
WHO/MSD/MSB 00.2j

The 8 steps of planning

an evaluation
1. Decide who will be involved in the evaluation.
2. Assess your resources.
3. Identify and prioritise the evaluation needs.
4. Describe your programme for evaluation.
5. Define your evaluation questions.
6. Determine your evaluation measures.
7. Determine your evaluation design.
8. Ensure that your resources are sufficient.
If not, return to Step 3.

Possible evaluation partners

• patients/clients
• family members
• senior managers
• board members
• staff (therapists/clinicians)
• researchers
• funder(s)
• representatives of other programmes or
service systems in your community
• people living in your community

86 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Evaluation resources

• Financial/Material

• Expertise

• Time

Developing a
programme logic model
Types of objectives
IMPLEMENTATION

(Process) Concerned with the program itself. Level/quality of services. The means
whereby outcomes are to be achieved and the target group(s).

OUTCOME What you hope to change. The ends you pursue.

• Immediate Ü
• Short-term Ü Goals
• Medium term Ü
• Long-term Ü

Facilitator’s workshop guide 87

WHO/MSD/MSB 00.2j

Types of objectives (cont'd)

Implementation • to establish ............... for
• to provide .................. to
• to prescribe .............. for
• to teach ..................... to
• to give ........................ to

Outcome: • to improve
• to decrease/increase
• to change/modify

Basic program logic model

Main components
(e.g. assessment, detoxification, follow-up)
Ü

Implementation objectives
(e.g. to determine correct diagnosis, to
provide substance resistance skills, to
monitor health status)
Ü

Planned short-term
outcome objectives
(e.g. to increase motivation for further
treatment, to decrease the likelihood of
relapse)
Ü

Ü
Ü

Planned long-term
outcome objectives
(e.g. to decrease substance use, to
improve quality of life)

88 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Main Assessment & Diagnosis & Withdrawal Referral

Components Intake Treatment Management &
Planning Treatment

Ü
Implementation • To confirm clients’ • To formulate • To monitor withdrawal • To provide
Objectives eligibility for the treatment plan symptoms information
program about other
• To sign • To prescribe standard
programs for
• To determine clients’ therapeutic medication
further drug-free
motivation to engage contract
• To provide a safe and treatment
in treatment
supportive environment
• To motivate
• To determine clients
• To conduct clients to
individual needs
laboratory/other tests continue
• To obtain standard treatment in one
• To motivate client to
somatic, mental and of these
finish the program
psychological status of programs
the client • To provide safe
withdrawal management
• To form therapeutic
alliance with the client
Ü

Ü
Short-term • To collect the • To cope with withdrawal • To increase clients knowledge
Outcome necessary information symptoms about further treatment
Objectives for development of possibilities
• To stabilise mental and
adequate treatment
physical status • To increase clients knowledge
• To enhance clients’ about consequences of PSU
• To maximise client
motivation to stay in and AIDS
satisfaction with the
the detoxification
program • To maximise the number of
program
clients who are referred to long-
• To reduce involvement in
term treatment
criminal activities
Ü

Long-term • To collect a database • To reduce drug use behaviour

Outcome of standardised client
• To prevent consequences of drug use
Objectives information
• To improve mental and physical well-being of clients
• To improve quality of
care

Facilitator’s workshop guide 89

WHO/MSD/MSB 00.2j

Basic format for moving from logic model to

more detailed evaluation planning

Program Evaluation Evaluation Data

Objectives Questions Measures Collection
(from the Strategy
logic model)

Objective 1

Objective 2

Day 3
90 Evaluation of Psychoactive Substance Use Disorder Treatment
 WHO/MSD/MSB 00.2j

Evaluation types
• Needs assessment
• Process evaluation
• Cost evaluation
• Client satisfaction
• Outcome evaluation
• Economic evaluation

The specialised workbooks

If you are conducting a… Then you should review…

Needs Assessment Workbook 3

Process Evaluation Workbook 4
Cost Evaluation Workbook 5
Client Satisfaction Workbook 6
Outcome Evaluation Workbook 7
Economic Evaluation Workbook 8

Facilitator’s workshop guide 91

WHO/MSD/MSB 00.2j

What are indicators?

Program Client Assessment
Component

Ü
Implementation (1) (2) (3)
Objective Trained Comprehensive Treatment plan
assessment Assessment prepared and
worker is protocol client assigned
available completed to treatment
Ü

Ü
Outputs Indicator#1 Indicator#2 Indicators#3
(indicators of service
delivery and characteristics
of those served)

Why bother with indicators?

Claim Confidence Indicator
Level
“Our PS Very low I heard it through the grapevine

awareness is
Low I talked to John, who participated and he said it
effective” worked for him

Medium 40 out of 50 people in the programme checked off

“effective” in a satisfaction questionnaire completed
at the end of the programme

High Compared to pre-test scores, scores on an alcohol

knowledge test went up by an average of two points

Highest Compared to a control group of people who weren’t

exposed to the programme at all, participants in the
programme increased scores on seven out of eight
measures of PS awareness

92 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Reliability
refers to whether an indicator is
consistent across time and/or
observers

Validity
concerns the extend to which you
are actually measuring what you
intend to measure

Types of measures

• Observation
• Questionnaires
• Interviews
• Focus group discussions
• Examining routine records

Facilitator’s workshop guide 93

WHO/MSD/MSB 00.2j

Language of measurement
Variable [The abstract output or outcome of interest]

Indicator Indicator [Different types of measures of the output or outcome]

01 02 03 04 01 02 03 04 [The specific observations you make]

A trivial example...

Height [Variable]

Tape measure Mark on the wall [Two indicators]

You Me You Me [Data]

Quantitative approach:
measurement of variable
through numbers

Qualitative approach:
measurement of variable
through words

94 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Prepare for data collection

• Set a time frame

• Determine a sampling procedure

1) Representativeness
2) Randomness

Sampling procedures
1. Accidental
2. Reputational
3. Random
4. Stratified
5. Cluster
6. Quota

Facilitator’s workshop guide 95

WHO/MSD/MSB 00.2j

Writing good questions

1. Are the words simple, direct and familiar to all?
2. Is the question as clear and specific as possible?
3. Is it a double question?
4. Does the question have a double negative?
5. Is the question too demanding?
6. Are the questions leading or biased?
7. Is the question applicable to all respondents?
8. Is the question objectionable?
9. Will the answers be influenced by response styles?
10. Have you exhausted the response alternatives?

Day 4
96 Evaluation of Psychoactive Substance Use Disorder Treatment
 WHO/MSD/MSB 00.2j

The 6 steps of
implementing an evaluation
1. Prepare for data collection.
2. Collect data.
3. Analyse data.
4. Report results.
5. Make use of what was learned.
6. Start again.

Prepare for data collection

Manage ethical issues
• Informed consent
• Manage risks of participation
• Confidentiality

Develop a data management plan

• Design a record keeping system
• Acquire and process data
• Train people who will collect data
• Standardisation

Conduct a pilot test

• Identify flaws
• Revise data collection plan as needed

Write an evaluation plan

Facilitator’s workshop guide 97

WHO/MSD/MSB 00.2j

Descriptive vs.
Explanatory analysis
Descriptive analysis Explanatory analysis
Goal: Goal
To summarise the measurements To explain relationship between
for each relationships between variables and groups
variable

Examples: Examples:
How many clients reported…? Do male and female clients differ...?
How much time did staff spend...? Compared to clients assigned to a
How much variability was control group, were clients exposed
there in…? to... more likely to...?

Definitions
Frequency Number of responses in each category
of an indicator
Mode Score that occurs most frequently
Median Score that separates the upper half of scores
from the lower half of scores
Mean Average score
Range Lowest score to highest score
Variability Extent to which scores deviate from their
central tendency
Indices • Variability
• Standard deviation

98 Evaluation of Psychoactive Substance Use Disorder Treatment

 WHO/MSD/MSB 00.2j

Structure for site

Presentations
Section I. Programme logic model and programme description
Briefly show your programme logic model components. Indicate the
part of the model that is the focus of the evaluation.

Section II. Evaluation question, measures, data collection

Show the one or two most important evaluation questions and how
you will answer them (evaluation plan)

Section Ill. Using the results for decision-making

Describe who will receive the results from the evaluation and how it
will be used for decision making.

Day 5
Facilitator’s workshop guide 99
WHO/MSD/MSB 00.2j

Written evaluation plan

1. Background and general purpose
2. Programme logic model
3. Evaluation team
4. Evaluation questions to be addressed
5. Data collection strategy
6. Data management plan
7. Staff training
8. Pilot test
9. Strategy for using results

100 Evaluation of Psychoactive Substance Use Disorder Treatment