Part 2

Doing Health Policy and

Systems Research: Key steps
in the process
Lucy Gilson
University of Cape Town, South Africa and
London School of Hygiene and Tropical Medicine, United Kingdom of Great Britain
and Northern Ireland
This part of the Reader outlines the four critical steps in due to the influence of other health researchers or specific
developing a primary Health Policy and Systems Research interest groups. National research priority-setting pro-
(HPSR) study that should be addressed by all researchers: cesses are, therefore, important as a means of ensuring
1. identify the research focus and questions dialogue and engagement between researchers and
2. design the study health policy-makers and managers. The aims of such
engagement are to turn health system and policy pro-
3. ensure research quality and rigour
blems into researchable questions, identify priorities
4. apply ethical principles.
among them and, ultimately, support the uptake of
When assessing the quality of empirical HPSR work, research findings. Greater national funding for HPSR may
it is important to consider all steps, not only Step 3. be a further consequence (Green & Bennett, 2007).

Examples of international research priority-setting

Step 1: Identify the processes include those convened by the Alliance for
Health Policy and Systems Research in 2007-2008 which
research focus and identified priority topics for research in human resources,
questions financing and the role of the non-state sector (see
Table 3). At national level, the Essential National Health
The process of developing an HPSR study begins with Research approach has provided a framework for
identifying the topic of focus – the issue or problem you priority-setting that has been applied in various countries
want to investigate – and the related questions. There (Green & Bennett, 2007; Alliance for Health Policy
are two main reasons for this: and Systems Research, 2009). See also the work of
1. HPSR is defined by the topics and questions it the Council on Health Research for Development at
addresses rather than the disciplinary perspective http://www.cohred.org.
or the particular approach to data collection and Beyond networking with policy actors and other
analysis it adopts. researchers, identifying an HPSR topic and related
2. HPSR always aims to be policy relevant and to inform research questions should involve:
the decisions taken by those who influence how
n
thinking creatively, for example to identify new areas
health systems evolve and perform – the policy of work or different approaches to an investigation;
actors, from household to global levels. (Note that
n
exploring theory and conceptual understandings
policy relevance is a key criterion used to assess the relevant to HPSR generally, and the topic of focus;
ethical value of HPSR research, Henning, 2004). n
conducting a literature search to identify relevant
As policy relevance is always important to HPSR, those publications and research studies.
working in the field have paid increasing attention to the Finally, pragmatism is important when identifying
process of setting research priorities. A particular concern a research question. The research needs to be feasible,
has been the influence of global actors (conventional for example, the scope and size of the study must be
multilateral and bilateral research funders as well as considered relative to the resources and time available
global public–private initiatives) over priority-setting (Robson, 2002; Varkevisser, Pathmanathan & Brownlee,
within low- and middle-income countries. The priorities of 2003).
these global actors have often emphasized upstream
health research or commodity procurement, rather than
systems strengthening questions and initiatives. Even
amongst national research communities, HPSR questions
may receive less attention than other research questions

Part 2 - Doing Health Policy and Systems Research: Key steps in the process 43
 Table 3 Priority research questions in three health policy and systems areas, results
of international priority-setting processes (Source: Alliance for Health Policy and
Systems Research, 2009)

Human resources for health Health system financing Non-state sector

1st To what extent do financial and How do we develop and implement How can the government create a
non-financial incentives work in universal financial protection? better environment to foster non-
attracting and retaining qualified state providers in the achievement
health workers to under-serviced of health systems outcomes?
areas?

2nd What is the impact of dual practice What are the pros and cons of the What is the quality and/or coverage
(i.e. practice by a single health care different ways of identifying the of health care services provided by
worker in both the public and poor? the non-state sector for the poor?
the private sectors) and multiple
employment?
Are regulations on dual practice
required, and if so, how should they
be designed and implemented?

3rd How can financial and non-financial To what extent do health benefits What types of regulation can
incentives be used to optimize reach the poor? improve health systems outcomes,
efficiency and quality of health and under what conditions?
care?

4th What is the optimal mix of financial, What are the pros and cons of How best to capture data and
regulatory and non-financial policies implementing demand-side trends about private sector
to improve distribution and subsidies? providers on a routine basis?
retention of health workers?

5th What are the extent and effects of What is the equity impact of social What are the costs and affordability
the out-migration of health workers health insurance and how can it be of the non-state sector goods and
and what can be done to mitigate improved? services relative to the state sector?
problems of out-migration? And to whom?

Networking and creative

thinking
Engaging with policy actors and other researchers helps Networking can also help to stimulate creative thinking.
to ensure that the topic and research questions are In addition, exploring conceptual understandings and
policy relevant. Both groups, through their experience in theory can highlight new areas of work rarely considered
different settings, will have insights into the challenges in the past, or new ways of understanding how to
and opportunities that face health systems. The types of investigate a topic on which there is already some
questions that may interest national policy-makers are research.
shown in Box 3. Such questions focus on both policy
content and policy processes.

44 Gilson L, ed. (2012). Health Policy and Systems Research: A Methodology Reader
Alliance for Health Policy and Systems Research, World Health Organization
 Box 3: Broad research questions of interest to national
policy-makers

Policy formulation
• What is the nature and extent of problem X?
• What happened before in response to problem X, and what were the consequences? What were
the unexpected consequences?
• What are cost-effective responses to the problem?
• How long will it be before the impacts of response Y are seen? How can popular and political support
be sustained until the impacts are seen?
Policy implementation
• What happens in practice when policy Y is implemented, and why?
• Do policy implementors have the same understanding of the problem that the policy aims to address, and
the same policy goals, as the policy-makers? If not, how does that difference affect policy implementation?
• Is the organizational response adequate/sustained?
Policy evaluation
• Were the policy, or programme, objectives met?
• What were the unexpected outcomes?
• Did the policy objectives remain the same over time?
• Did the condition being addressed change over time?
• Was the programme[or policy?] implemented effectively?
Source: Rist 1998

Literature search
It is important to find out what relevant research has or developing new ideas on topics that have already
already been conducted in order to avoid unnecessary been considered.
duplication and to build on existing research.
The growth of interest in systematic reviews and
Although researchers can draw on their own knowledge syntheses of existing research reflects, in part, the
of a particular setting, it is always important that they concern that existing primary research is frequently
conduct more formal literature reviews of research not used as a basis for changing policy and practice,
previously conducted in other settings and not only in or for developing new research work. An important
the area with which they are familiar. While there is value resource for health policy and systems researchers is,
in replication studies (deliberately replicating work therefore, the Health Systems Evidence web site at
previously conducted in one setting in a new setting to http://www.healthsystemsevidence.org .
generate new insights, for example (Robson, 2002)), This is a continuously updated and searchable repository
the duplication of a research study simply because of of syntheses of research evidence about governance,
limited knowledge about existing research is a waste of financial and delivery arrangements within health
resources and so unethical (Emanuel et al., 2004). systems, and about implementation strategies that
can support change in health systems.
New studies must always offer value, that is they must
build on existing work, for example by addressing a
question not previously considered in a particular setting,

Part 2 - Doing Health Policy and Systems Research: Key steps in the process 45
 Key challenges
Two key challenges related to identifying appropriate conducted elsewhere. Similarly, managers of a
research questions are discussed below. particular health programme, be it HIV/AIDS, nutri-
1. Framing policy relevant and valuable HPSR questions tion or school health, tend to be most interested in
through networking with research users. research about how to strengthen their particular
programme and less interested in the systemic
A challenge of generating new research ideas
support needs across programmes. Yet, as discussed
through networking with policy actors is that the
earlier (see Part 1, Section 6), HPSR focuses on such
types of topics and questions identified as important
systemic needs rather than on programme-specific
will vary between policy actors, depending on their
needs.
roles and responsibilities within the health system
(as illustrated in Box 4). For example, policy actors Therefore, health policy and systems researchers need
working at lower levels of the health system have to think carefully about the fuzzy boundary between
particular operational needs which, while important, HPSR and management (see Part 1, Section 6) and
might limit the wider application of the work if other seek either to support managers to conduct their
policy actors do not see its relevance to them or own operational research, or to identify the wider
if it requires the duplication of research already value of the particular research question.

Box 4: The HPSR questions of different health policy

and systems actors

National policy-makers might ask:

• How can we prevent the HIV/AIDS programme from draining resources (time and staff) from other equally
important programmes?
• How can HIV/AIDS resources be used in ways that strengthen other areas of the health system?
• Should antiretrovirals be prescribed only by doctors or is prescription by nurses more cost-effective?
District managers might ask:
• Why are there more patient complaints about facility X than others in my district?
• Why are patient waiting times at clinics still very long, although we have already tried to reorganize
services to address the problem?
• How can we develop an integrated HIV/AIDS and tuberculosis service, in line with national policy?
Hospital managers might ask:
• How can we decrease the pharmacy waiting time?
• How can we reduce the average length of stay for chronically ill patients?
• Are ambulatory services available and adequate?
• Are patients coming late for treatment and why?
Patient groups might ask:
• Why do we have to wait so long to get care?
• Why do health workers treat us so rudely?

46 Gilson L, ed. (2012). Health Policy and Systems Research: A Methodology Reader
Alliance for Health Policy and Systems Research, World Health Organization
2. Identifying research questions that are relevant to a challenges. Therefore, the health policy and systems
range of policy actors and that add to the existing researcher can see the particular programme issue as
knowledge base. a case study of policy implementation.
The challenge for the health policy and systems All these approaches show how research around one
researcher is to identify policy-relevant and valuable programme can represent a tracer for understanding
research questions that not only directly address the and/or influencing health policy and system dynamics,
concerns of the main group of policy actors with as discussed earlier.
whom they seek to work and influence, but also have Overall, compared with research focused on a parti-
relevance to a wider range of actors and add to the cular disease programme, service area or clinical
existing HPSR knowledge base. treatment, HPSR requires the researcher to consider
For example, how can work on reducing a particular the system within which the specific service or
hospital’s pharmacy waiting time have relevance to treatment is nested. This means thinking:
other hospitals or to national managers concerned n
broad – beyond the disease or treatment of
with supporting all hospitals to reduce waiting times? focus;
Similarly, how can research linked to a particular n
up – above the programme or service to the
disease programme be undertaken in ways that offer facility, district, province etc.; and
policy and systems lessons that benefit other n
about the cross-cutting functions that underlie
service and programme delivery – the system
programmes as well? In both cases, it is important to
building blocks and interactions among them
see the specific focus of the research as an entry (Schneider, 2011).
point for considering an issue of relevance to a
broader range of actors and settings.
In terms of pharmacy waiting times, there could be
Identifying the purpose of
value in seeing the work in one hospital as a case the research
study of how to tackle such a problem. The case study
could generate ideas on processes and strategies that In developing research questions that will be policy
can initially be tested in other hospitals. Then, drawing relevant and valuable, it is also important to think about
on several experiences, this can become the basis for the overall purpose of the research, in particular:
compiling general insights into ways to address the n
What is the research trying to achieve? or Why is it
common problem of waiting times. This is an example being done?
of the process of analytic generalization and it n
To whom will it be useful?
provides the basis for the sort of policy learning in n
How will it be useful?
implementation discussed in Part 1, Section 7. n
How will it add to the existing knowledge base?
Another approach would be to see how work in
Thinking about such questions will also inform the
a particular programmatic area offers insights into
design of the research study (see Step 2).
a broader systems’ question of relevance across
programmes. For example, work on task shifting As research questions are developed, four dimensions
within an HIV/AIDS programme offers insights on can be considered:
the types of human resource development and 1. Whole field or specific policy: Will the research seek
management needs that must be addressed in any to focus on the field as a whole, and so expand
new policy initiative that involves an expansion of the knowledge of the nature and functioning of the key
scope of work of lower-level cadres; it also highlights elements of health policy and systems, or will it
the possible challenges to the political feasibility of seek to focus on a particular policy and support its
such an initiative and ways of managing those implementation?

Part 2 - Doing Health Policy and Systems Research: Key steps in the process 47
 2. Normative/evaluative or descriptive/explanatory can add to our general knowledge of policy
research questions: Will the research address norma- development and implementation (analysis of
tive or evaluative questions (which may involve value policy)? If so, this will generally demand longer
time frames, with a focus on the broader research
judgements) or descriptive or explanatory questions?
questions through which the complex and
Table 4 provides some examples of HPSR questions dynamic trajectories of policy experience, for
across dimensions 1 and 2. example, are more amenable to investigation.

3. Analysis ‘for’ or ‘of’ policy (Parsons, 1995) – where 4. Primary research purpose: Will the research primarily
focused on a particular policy: seek to explore an issue or phenomena in order to
describe it or to explain it? Or will it adopt a more
n
Will the research aim to support policy implemen-
tation in real time (analysis for policy)? (Whether critical stance in generating understanding, perhaps
considering the technical content of the policy or working with other people to bring about change
experience of the actors and processes engaged rather than focusing only on generating knowledge?
in its implementation.) If so, this may demand Although these research purposes often overlap in
shorter time frames and is likely to be focused on
practice, Robson (2002) identifies their different aims,
narrower research questions; or
see Box 5.
n
Will the research aim to generate, from that
policy’s experience, a broader understanding that

Table 4 Examples of HPSR questions (Source: Adapted from Potter and Subrahmanian, 1998)

Normative/evaluative questions Descriptive/explanatory questions

Questions about Cell 1 Cell 3

the policy itself Should this policy be adopted? Which agencies are stakeholders in this policy,
How does policy X impact on health seeking what positions do they take on the policy and
behaviour? why?
Which actor management strategies are How did policy X come about?
likely to be most useful in supporting Is there capacity to implement policy X?
implementation of policy x? How do front line providers understand policy X?

Questions about Cell 2 Cell 4

the field Which type of health system performs best? What are patterns of health seeking behaviour
What are the different approaches to actor and what influences that behaviour?
management that can be considered by those How is the health system organized at present?
seeking to manage policy change? What if a new provider was available, how
would health seeking behaviour change and
how would it affect the performance of the
system overall?
What influences how front line providers under-
stand policies, and how does their understanding
influence their implementation of the policy?

Note: The questions in Cell 1 are asked by those responsible for policy implementation, and essentially demand judgements,
at least some of which are likely to be informed by work addressing the questions proposed in Cell 4. The questions in Cell
2, meanwhile, address what people should do, and may be informed by the ‘what if’ questions included in Cell 4. Finally,
questions in Cell 3 encompass the areas of interest in health policy analysis, as outlined earlier: the context, history,
interests and organizations that shape a particular policy.

48 Gilson L, ed. (2012). Health Policy and Systems Research: A Methodology Reader
Alliance for Health Policy and Systems Research, World Health Organization
 Box 5: The purpose of different types of research
Exploratory research seeks to:
• find out what is happening, especially in little-understood situations
• generate new insights and ask questions
• assess phenomena in new light
• generate ideas and hypotheses for future research
Descriptive research seeks to:
• give an accurate profile of people, events, situations
Explanatory research seeks to:
• explain a situation or problem, traditionally, but not necessarily, in the form of a causal relationship
(evaluative research)
• explain patterns relating to the phenomenon being researched
• identify and explain relationships between aspects of phenomenon/phenomena
Emancipatory research seeks to:
• create opportunities and the will to engage in social action
m Critical research: Focuses on the lives and experiences of those traditionally marginalized, analysing how
and why inequities are reflected in power imbalances and examining how research into inequities leads
to political and social action
m Action research: seeks improvements in practices, understandings of practice and situations of practice,
and is undertaken by and with those who will take action
Source: Robson, 2002

The purpose of the research should reflect the current Relativists, however, are more likely to conduct forms of
state of knowledge about the topic. Exploratory work, for exploratory, descriptive and/or explanatory research that
example, is important when little is known about a topic aim to deepen our understanding of the phenomena
or when theory suggests a new way of examining of focus and the complex relationships among aspects
and understanding it; but descriptive research requires of those phenomena. Sheikh et al. (2011:5) have speci-
extensive knowledge of the situation in order to identify fically suggested that more HPSR work needs to adopt
what is useful to investigate. However, in empirical work this perspective and address the “fundamental, explora-
researchers often pursue more than one purpose at the tory and explanatory questions” that shape policy and
same time (see Figure 4). provide a platform for further research. For relativists,
emancipatory research also represents an important form
The purpose of the research will also reflect the resear-
of research – analysis for policy.
cher’s understanding of social and political reality (see
Part 1, Section 7). Positivists and critical realists tend to Box 6 shows how the different purposes of research
focus on evaluating causal relationships, based on translate into different basic forms of research questions.
particular forms of descriptive work. For them, therefore,
Finally, across these different research purposes, research
explanatory questions are the same, more or less, as the
might address one or more of the different levels of the
evaluative questions outlined in Table 4, Cell 1; perhaps
system (from micro, meso or macro level) and work with
also entailing forms of descriptive work and preceded by
different (conceptual) units of analysis such as individual
exploratory pilot studies, or accompanied by exploratory
behaviour, patient–provider relationships, the primary
work to support explanation.
health care system, the district hospital, etc.

Part 2 - Doing Health Policy and Systems Research: Key steps in the process 49
 Figure 4 Multiple research purposes

Action/Participatory research

Exploratory: Descriptive:
What ? Who, what, where,
(new insights) how many, how much?

Explanatory (Evaluative):
Impact? +
Why and how?

Box 6: Links between purpose and broad forms

of research questions

Exploratory/descriptive questions
‘What’ or ‘how many/much’, or ‘who’ or ‘where’ questions
• What is the experience of patients with new programme x?
• What is the experience of health workers in training programme x?
• What is the understanding of patient groups or health workers about a problem or a new programme?
• To what extent are family members involved in the programme?
• Who is exposed to condition x or health risk y?
Explanatory questions
Evaluation questions
• Does programme x lead to reduced health problems from the condition addressed?
• Is programme x more effective than programme y in treating this condition?
• For which group of patients is programme x most effective?
How and why questions:
• How does programme x generate these impacts?
• Why is programme x more effective than programme y?
• Why do health workers act unexpectedly when implementing the programme?
• How do policy actors’ values and beliefs influence their decision-making practices?
• Who supports and opposes new policy x, and why and how?

50 Gilson L, ed. (2012). Health Policy and Systems Research: A Methodology Reader
Alliance for Health Policy and Systems Research, World Health Organization
 n
health economics to understand the impacts achieved
Taking account of by a particular form of community accountability; and
multidisciplinarity n
an historical perspective to track the changing roles
of international organizations within global health
Within HPSR, different disciplinary perspectives generate policy.
different research questions on the same topic and
so generate varied policy-relevant insights on the issue
of focus. Therefore, on the one hand, it is important
Finalizing research
to consider the disciplinary perspective that you as a questions
researcher bring to the topic and the type of research
Ultimately, good research questions (Robson, 2002), i.e.
questions you are likely to consider. On the other hand, it
those that will drive valuable and sound research, are:
may be useful to think about how to draw on other
disciplinary perspectives that address the same topic. n
clear – unambiguous and easily understood;
n
specific – sufficiently specific to be clear about what
Work on human resources, for example, may draw on constitutes an answer;
economics and sociological perspectives to understand
n
answerable – clearly indicate what type of data are
motivation; alternatively political science or organiza- needed to answer the question and how the data
tional management perspectives may be applied to will be collected;
understand the decision-making of front line providers; n
interconnected – a set of questions are related in a
or the work may draw on clinical insights to understand meaningful way and form a coherent whole;
skills needs. All have policy relevance. n
substantively relevant – worthwhile, non-trivial
questions, worthy of the effort to be expanded in the
Therefore, Part 4 of this Reader includes papers that
research.
address particular health system functions, or building
blocks, from different disciplinary perspectives (see Part 4:
Table 8). Financing issues, for example, are examined
using:
n
policy analysis and sociological perspectives in order
to understand what influences why and how parti-
cular financing policies are prioritised, developed and
implemented;
n
the health economics lens in order to understand
what cost burdens households experience in acces-
sing care and how they cope with these costs, and
what is the impact on health of community-based
health insurance.

The papers addressing leadership and governance issues

draw, moreover, on:
n
policy analysis to understand the influences over
various experiences of policy change;
n
anthropology to generate in-depth insights about
decentralisation experience and explore global
discourses around maternal health care provision;
n
management sciences to understand the use of
information in district decision-making;

Part 2 - Doing Health Policy and Systems Research: Key steps in the process 51
 Step 2: Design the study
Once you have the research question/s, the next step is The research purpose and question/s shape the research
to develop the overarching design of the study: to turn strategy. Table 5 provides examples of the different over-
the questions into a project. The overarching study design arching designs that are relevant for different purposes
is not just a set of data collection methods. The design is across the dominant paradigms of knowledge.
comprised of the:
n
purpose of the study (see Step 1)
n
particular questions to be addressed (see Step 1)
n
strategy for data collection and analysis
n
sampling strategy
n
theory to be used within the study (Robson, 2002).

Table 5 A summary of broad study designs (Source: Adapted from Klopper, 2008;
Potter and Subrahmanian, 1998; Yin, 2009.)

Paradigm of Research strategy

Purpose Descriptive/explanatory questions
knowledge Collection of new data Analysis of existing data

Positivist Explanatory Experimental and quasi-experimental Simple and multiple-variable

design including, for example, before modelling
and after studies

Descriptive Survey designs: questionnaires, interviews Secondary data analysis (census

and indirect observation; data, record data)
Repeated surveys to allow trend analysis Quantitative content analysis
over time (newspaper reports, speeches, etc.)

Exploratory Survey designs (pilot studies)

Relativist Explanatory Case study (theory building, longitudinal) Qualitative content analysis
Grounded theory (theory building) Discourse analysis
Historical analysis

Descriptive Case study

Ethnographic designs with the focus
on unstructured direct and indirect
observations, for example narrative
inquiry, critical ethnography

Exploratory Field designs or ethnographic designs with

the emphasis on the use of informants,
for example autho-ethnography,
autobiography, life histories
Case study (such as generating
categorizations)
Qualitative interviews and panels

52 Gilson L, ed. (2012). Health Policy and Systems Research: A Methodology Reader
Alliance for Health Policy and Systems Research, World Health Organization
Research strategies can also be grouped into two main knowledge paradigms to which they are mostly linked.
sets: fixed designs that are established before data It also highlights examples of common data collection
collection and flexible designs that evolve during the study methods, key principles of sampling and the primary
(Robson, 2002). Table 6 summarizes the key chara- characteristics of analysis. Note that within either strategy
cteristics and forms of these two sets of strategies, set, multiple methods may be used in which qualitative
and links them both to the standard forms of research and quantitative data collection approaches are combined.
questions for which they are appropriate and the

Table 6 Key features of fixed and flexible research strategies (Source: adapted from Robson, 2002)

Fixed strategy Flexible strategy

Characteristics Calls for tight pre-specification before data Design evolves during data collection
collection Data often non-numerical
Data generally numbers Often called qualitative
Often called quantitative Quantitative data may also be collected
Rarely collect qualitative data (multi-method study)

Dominant Positivist Critical realist

knowledge Interpretivist/Social constructivist
paradigm

Overarching Primary data collection methods Primary data collection methods

study design Experimental Case study
types Quasi-experimental Grounded Theory
Non-experimental (for example cross- Ethnography
sectional, before and after studies, trend Life histories
analyses) Phenomenological research (qualitative interviewing)
Secondary data analysis Secondary data analysis
Modelling Historical analysis
Archive analysis
Discourse analysis

Forms of What is impact of x? How and why? (where investigator has little
research How and why? (where investigator has control over events, or limited knowledge
question control over events, and existing knowledge about mechanism involved)
about mechanisms involved) What (what is going on here)?
What (how many, how much, who, where?)

Examples of Structured and semi-structured interviews Qualitative individual interviews

dominant data (including open-ended questions) Focus group discussions
collection Routine record review Observation
methods Document review

Key sampling Representive of sample population Purposive sampling guided by theory, to ensure
principles maximum variability across relevant units

Characteristics Statistical analysis following predetermined Iterative

of data analysis rules Interpretative

Part 2 - Doing Health Policy and Systems Research: Key steps in the process 53
 There is also a third category of research strategy: mixed-
method studies, which deliberately combine elements
Using theory and
of fixed and flexible design “to expand the scope of, conceptual frameworks
and deepen the insights from, their studies” (Sandelowski, to inform the study
2000). This strategy is not linked to a particular know-
ledge paradigm or set of methods, nor does it reflect a Given the complexity of the phenomena addressed by
mix of paradigms. Instead it purposefully combines HPSR, theory should play an important role within every
different methods of inquiry in order to capture different study design and within both fixed and flexible research
strategies. In evaluation work, for example, there is
dimensions of the central phenomenon of focus. Mixed-
increasing acknowledgement of the importance of
method studies, thus, entail various combinations of
theory-driven inquiry in adequately addressing complex
sampling and/or data collection and/or data analysis
causality (de Savigny & Adam, 2009) – in both experi-
techniques in order to:
mental or quasi experimental designs and the case study
n
allow triangulation across data sets;
work linked to critical realist evaluation (see Part 4,
n
enable the elaboration of results, through complemen- ‘Advances in impact evaluation’). However, currently,
tary data and analyses;
theory is too rarely used in HPSR and as a result policy
n
guide the development of an inquiry by identifying
analysis work, for example, is often quite descriptive.
additional sampling, data collection and analysis
Opportunities for the theory-building and explanatory
needs.
work that would better inform policy-making and imple-
Within a study different methods may be used sequen- mentation are ignored (Gilson & Raphaely, 2008; Walt et
tially (at different times) or concurrently (at the same al., 2008).
time). Examples of what a mixed-method study could
entail in practice are given below. In broad terms, theory provides a language for describing
and explaining the social world being studied and
n
The research could entail an initial small-scale
represents a general explanation of what is going on in a
intensive study using qualitative methods to develop
detailed understanding of a phenomenon. This would situation. It offers the basis for generating hypotheses
be followed by a larger-scale structured survey under- (predicted answers that can be statistically tested in fixed
taken to generate more extensive understanding designs), as well as looser propositions of how different
of the same phenomenon, and that uses a tool deve- dimensions of a phenomena may be linked, which can
loped with the detailed understanding generated
be explored or considered in analysis (flexible designs).
from the initial study.
The ‘middle range theory’ represented by the latter can
n
An initial structured survey, using a random sampling
be captured in the form of a conceptual framework (a
approach to gather knowledge around a pheno-
menon within one population of respondents, could set of concepts and their inter-linkages) that may offer
provide the basis for purposeful sampling of respon- explanations or predictions of behaviour, or outcomes,
dents within the same population to allow more but may also simply identify relevant elements and
detailed inquiry and gain a deeper understanding of relationships.
the results of the initial survey.
A conceptual framework to guide study design can be
n
The mixing of methods might only occur in data
developed from a review of relevant empirical and
analysis, perhaps by interpreting different sets of
study results or through converting one type of data theoretical literature. The framework can help to identify
into the other in order to allow statistical analysis of relevant concepts and variables (fixed strategies) or
qualitative data. issues (flexible strategies) for investigation, and to guide
However, whichever approach is used, mixed-method the selection of samples or cases (flexible strategies).
studies involve a focus on a particular phenomenon and In addition, a conceptual framework may be revised
a purposeful combination of methods to achieve justified as the data collected are analysed. Alternatively, it may
goals in the context of the particular inquiry. be generated as a result of the data analysis process.

54 Gilson L, ed. (2012). Health Policy and Systems Research: A Methodology Reader
Alliance for Health Policy and Systems Research, World Health Organization
In either case, the conceptual framework can be put back
into the public domain to be questioned and perhaps
Step 3: Ensure research
used to support future research. Such theory building is quality and rigour
a process of knowledge generation.
The criteria used to make judgements of research quality
Therefore, HPSR is not solely concerned with generating
and rigour differ between paradigms of knowledge.
empirical evidence to inform policy decisions. Rather,
Whereas positivist research emphasizes validity and
HPSR can combine theoretical and empirical work or
reliability – ensured through careful study design, tool
be primarily theoretical and still maintain its policy
development, data collection and appropriate statistical
relevance.
analysis – relativist research considers the trustworth-
Combined theoretical and empirical work has, for iness of the analysis – whether it is widely recognized to
example, aided understanding of the norms and customs have value beyond the particular examples considered.
influencing the decision-making of health system actors The different criteria and questions used in assessing the
in particular contexts (such as Riewpaiboon et al., 2005; quality of research based on fixed and flexible designs
Sheikh and Porter, 2010). It has also traced the patterns are summarized in Table 7. Table 8 indicates how trust-
and influences over time of policy change across sub- worthiness can be established by providing information
national, national and global levels (for example Walt, on study design, data collection, and the processes of
Lush & Ogden, 2004). Theory-driven evaluation, mean- data analysis and interpretation.
while, supports research that seeks to explain how new
Ultimately, good quality HPSR always requires a critical
policies and interventions influence health system
and questioning approach founded on four key processes
operations (Marchal, Dedzo & Kegels, 2010). Combined
(Gilson et al., 2011):
theoretical and empirical work can also generate ideas
about how to influence policy agendas (for example n
An active process of questioning and checking during
Shiffman, 2007: advocacy in agenda setting) or manage the inquiry (Thomas, 1998): ask how and why things
happened – not only what happened; check answers
policy change (for example Walker & Gilson, 2004:
to questions to identify additional issues that need to
managing front line providers acting as street-level be followed up in order to deepen understanding of
bureaucrats). Such ideas have relevance beyond the the experience.
original settings in which the research was conducted. n
A constant process of conceptualizing and recon-
Purely theoretical research can also lead to new ways to ceptualizing (Thomas, 1998): Use ideas and theory
to develop an initial understanding of the problem,
describe the nature and organization of health systems,
or situation of focus, in order to guide data collection
or what influences their performance, and to understand but use the data collected to challenge those ideas
what drives particular policy actors in their decision- and assumptions and, when necessary, to revise your
making (for example Bloom, Standing & Lloyd, 2008 ideas in response to the evidence.
(plural health systems); de Savigny & Adam, 2009 n
Crafting interpretive judgements (Henning, 2004)
(systems thinking); Gilson, 2003 (trust and health based on enough evidence, particularly about context,
systems); Kutzin, 2001 (financing); Mackian, Bedri & to justify the conclusions drawn as well as deliberate
consideration of contradictory evidence (negative
Lovel, 2004 (health seeking behaviour)). Through such
case analysis) and review of initial interpretations by
work HPSR informs policy by expanding our under- respondents (member checking).
standing of what strengthening a health system involves, n
Researcher reflexivity: be explicit about how your
and identifies research questions for empirical investi- own assumptions may influence your interpretation
gation. and test the assumptions in analysis (Green &
Thorogood, 2009).
Part 3 of the Reader presents references to some con-
ceptual frameworks that are valuable in HPSR.

Part 2 - Doing Health Policy and Systems Research: Key steps in the process 55
 Table 7 Criteria and questions for assessing research quality (Source: adapted from Robson, 2002)
Fixed designs Flexible designs

Reliability: Is your variable measure reliable? Confirmability: Do the data confirm the general findings
and lead to their implications?
Construct validity: Are you measuring what you think
you are measuring? Dependability: Was the research process logical and well
documented?
Internal validity: Does the study plausibly demonstrate
a causal relationship? Credibility: Is there a match between participants’ views
and the researcher’s reconstruction of them?
External validity: Are the findings statistically
generalizable? Transferability: Do the findings generate insights that are
transferable to other settings?

Table 8 Processes for ensuring rigour in case study and qualitative data collection
and analysis (Source: Gilson et al., 2011)

Example:
Principle A study of the influence of trust in workplace relationships over
health worker motivation and performance, involving in-depth
inquiry in four case studies (Gilson et al., 2004)

Prolonged engagement with the subject of inquiry Case study:

Although ethnographers may spend years in the field, A period of three to four weeks spent in each case study
HPSR tends to draw on lengthy and perhaps repeated facility
interviews with respondents, and/or days and weeks of Respondents
engagement within a case study site Informal engagement & repeated formal interviews

Use of theory Conceptual framework derived from previous work

To guide sample selection, data collection and analysis, (Gilson et al., 2005)
and to draw into interpretive analysis Case study selection based on assumptions drawn from
framework (see below)
Theory used in triangulation and negative case analysis
(see below)

Case selection Four primary health care facilities: two pairs of facility
Purposive selection to allow prior theory and initial types, & in each pair one well and one poorly performing
assumptions to be tested or to examine ‘average’ or as judged by managers using data on utilization and
unusual experience tacit knowledge (to test assumptions that staff in ‘well
performing’ facilities have higher levels of motivation
and workplace trust)

Sampling In small case study facilities, sampled all available staff;

Of people, places, times etc, initially, to include as many as in larger facilities for interviews: sampled staff of all
possible of the factors that might influence the behaviour groupings and with a range of staff in each group
of those people central to the topic of focus (subsequently (considering e.g. age, sex, length of time in facility);
extend in the light of early findings) random sample of patients visiting each facility; all facility
Gather views from wide range of perspectives and supervisors & area manager
respondents rather than letting one viewpoint dominate

Multiple methods (case studies) For each case study site:

Use multiple methods for case studies Two sets of formal interviews with all sampled staff
Researcher observation & informal discussion
Interviews with patients
Interviews with facility supervisors and area managers

56 Gilson L, ed. (2012). Health Policy and Systems Research: A Methodology Reader
Alliance for Health Policy and Systems Research, World Health Organization
Table 8 (Continued) Processes for ensuring rigour in case study and qualitative data collection
and analysis (Source: Gilson et al., 2011)

Example:
Principle A study of the influence of trust in workplace relationships over
health worker motivation and performance, involving in-depth
inquiry in four case studies (Gilson et al., 2004)

Triangulation Within cases:

Looking for patterns of convergence by comparing results Initial case reports based on triangulation across all data
across multiple sources of evidence (e.g. across interviewees, sets for that case (and across analysts in terms of
and between interview and other data), between individual staff members’ experience), generating overall
researchers, across methodological approaches, with theory judgments about facility-wide experience as well as noting
variation in individual health worker experience
Cross-cases:
Initial case reports compared with each other to look for
common and different experiences across cases, and also
compared with theory to look for convergence or divergence

Negative case analysis Within cases:

Looking for evidence that contradicts your explanations Triangulation across data identified experiences that
and theory, and refining them in response to this evidence contradicted initial assumptions (e.g. about the influence
of community interactions over motivation, and about the
association between low motivation and poor caring
behaviour), and identified unexpected influences (e.g. a
general sense of powerlessness among health workers)
Cross-cases:
Cross-site analysis identified facility-level experience that
contradicted initial assumptions underpinning study (e.g.
about link between high levels of workplace trust, strong
health worker motivation and positive caring behaviour),
and identified unexpected conclusions (e.g. about the
critical importance of facility level management over trust
and motivation)
Report notes weak evidence to support links between
levels of workplace trust and client perceptions, but also
stronger evidence of links between levels of workplace
trust and motivation

Peer debriefing and support Preliminary case study reports initially reviewed by other
Review of findings and reports by other researchers members of the research team

Respondent validation (Member checking) Preliminary cross-case analysis fed back for review and
Review of findings and reports by respondents comment to study respondents; feedback incorporated into
final reports

Clear report of methods of data collection and Report provides clear outline of methods and analysis
analysis (Audit trail) steps as implemented in practice (although more could be
Keeping a full record of activities that can be opened to fuller and reflexive)
others and presenting a full account of how methods
evolved to the research audience

Part 2 - Doing Health Policy and Systems Research: Key steps in the process 57
 Step 4: Apply ethical Be concerned about safeguarding:
1. the scientific validity and trustworthiness of the data
principles – through careful and deliberate training for all
research staff, including fieldworkers, to equip them
As with all research, it is important to take account of
with the attitudes and communication skills necessary
ethical issues in conducting HPSR. Although the focus of
to conduct good quality interviews and get beyond
the research differs from other health research, there are
their differences in race, class, nationality, gender or
always issues of power at play between those doing the
education with respondents; and treatment of field-
research and those being researched, and so there is real
workers as true partners in the research inquiry,
potential for disrespectful and unfair treatment. Robson
recognizing their essential role in shaping the nature
(2002) suggests that all ‘real world researchers’ need to
and quality of data.
watch out for the following ten questionable ethical
2. social value and a favourable risk–benefit ratio of the
practices:
study – by careful consideration of the individual
n
involving people without consent
and community-level risks and benefits of participa-
n
coercing them to participate
tion in the study, through engagement with a range
n
withholding information about true nature of the
of stakeholders at the start of the study and constant
research
review and reflection during the study.
n
otherwise deceiving participants
3. informed consent and respect for participants and
n
inducing participants to commit acts diminishing of
their self-esteem communities – by ensuring that all team members
are familiar with the study’s key messages and can
n
violating rights of self-determination
call for assistance when unexpected ethical issues
n
exposing participants to physical or mental stress
arise; are able to, and do, demonstrate respect for
n
invading privacy
participants in all their engagements with commu-
n
withholding benefits from some participants
nities; and re-negotiate relationships as and when
n
not treating participants fairly or with respect.
necessary rather than concentrate efforts only on
These are similar to the concerns of all health research. formal consent procedures (which may be infeasible
The challenges may be particularly acute in cross-cultural in an HPSR study or impact negatively on the rela-
research, such as when HPSR is undertaken in lower- tionships with study participants that are essential to
income countries by researchers or others from higher- gathering honest information).
income settings (Molyneux et al., 2009). Thus, one of the 4. independent review – by supporting ethics commit-
eight ethical principles proposed by Emanuel et al. (2004) tees to pay particular attention to the proposed
for clinical research is collaborative partnership between process of research and interactions among different
investigators and research sponsors in higher-income
actors within HPSR work, rather than primarily exami-
countries and researchers, policy-makers and commu-
ning study design and tools.
nities in lower-income countries (see Box 7).
Ultimately, however, “the social relationships established
However, as HPSR differs in nature from medical research,
between researchers and field-teams and community
there are some particular ethical debates in, and peculiar
members, are critical to fulfilling the moral (as opposed
ethical challenges for, this area of work. From reflection
to legal) aspects of ethics guidelines” (Molyneux et al.,
on the experience of conducting household-level HPSR
2009:324). Such relationships will always be important
studies in different countries, for example, Molyneux et
in HPSR, whether the interviewees are community
al. (2009) make the following four sets of proposals on
members or policy elites.
how to implement the principles of Box 7 in this form of
research.

58 Gilson L, ed. (2012). Health Policy and Systems Research: A Methodology Reader
Alliance for Health Policy and Systems Research, World Health Organization
 Box 7: Eight ethical principles for clinical research
in low- and middle-income countries

• Collaborative partnership • Social value

• Scientific validity • Fair selection of study population
• Favourable risk-benefit ratio • Independent review
• Informed consent • Respect for recruited participants and study communities
Source: Emanuel et al., 2004

References
Alliance for Health Policy and Systems Research (AHPSR) Gilson L, Khumalo G, Erasmus E, Mbatsha S and McIntyre
(2009). Priority setting for Health Policy and Systems D (2004). Exploring the influence of workplace trust over
Research. AHPSR Briefing Note 3. Geneva, World Health health worker performance: Preliminary national overview
Organization (http://www.who.int/alliance-hpsr/en/ , report South Africa. Johannesburg: Centre for Health
accessed 22 August 2011). Policy. Unpublished report.

Bloom G, Standing H, Lloyd R (2008). Markets, informa- Gilson L et al. (2011). Building the field of health policy
tion asymmetry and health care: Towards new social and systems research: social science matters. PLoS
contracts. Social Science & Medicine, 66 (10):2076-2087. Medicine 8(8):e1001079.

de Savigny D, Adam T, eds (2009). Systems thinking for Green A, Bennett S, eds (2007). Sound choices:
health systems strengthening. Geneva, World Health enhancing capacity for evidence-informed health policy.
Organization (http://www.who.int/alliance-hpsr/resources Geneva, World Health Organization.
/9789241563895/en/index.html, accessed 10 July 2010).
Green J, Thorogood N (2009). Qualitative methods for
Emanuel E et al. (2004). What makes clinical research in health research. London, Sage Publications.
developing countries ethical? The benchmarks of clinical
Henning E (2004). Finding your way in qualitative
research. Journal of Infectious Diseases, 189:930–937.
research. Pretoria, Van Schaik Publishers.
Gilson L (2003). Trust and the development of health
Klopper H (2008). The qualitative research proposal.
care as a social institution. Social Science & Medicine,
Curationis. 31(4):62–72.
56(7):1453–1468.
Kutzin J (2001). A descriptive framework for country-level
Gilson L, Palmer N and Schneider H. (2005) Trust and
analysis of health care financing arrangements. Health
health worker performance: exploring a conceptual
Policy, 56:171–204.
framework using South African evidence. Social Science
and Medicine 61(7): 1418-1429 Mackian S, Bedri N, Lovel H (2004). Up the garden
path and over the edge: where might health-seeking
Gilson L, Raphaely N (2008). The terrain of health policy
behaviour take us? Health Policy and Planning,
analysis in low- and middle-income countries: a review
19(3):137–46.
of published literature 1994–2007. Health Policy and
Planning, 23(5):294–307. Marchal B, Dedzo M, Kegels G (2010). A realist evalu-
ation of the management of a well-performing regional
hospital in Ghana. BMC Health Services Research, 10:24.

Part 2 - Doing Health Policy and Systems Research: Key steps in the process 59
 Molyneux C et al. (2009). Conducting health-related Thomas A (1998). Introduction. In: Thomas A, Chataway J,
social science research in low-income settings: ethical Wuyts M, eds. Finding out fast: investigative skills for
dilemmas faced in Kenya and South Africa. Journal of policy and development. London, Sage Publications:1–18.
International Development, 21(2):309–325.
Varkevisser CM, Pathmanathan I, Brownlee A (2003).
Parsons W (1995). Public policy: an introduction to Designing and conducting health systems research
the theory and practice of policy analysis. Aldershot, projects: Volume 1: proposal development and fieldwork
Edward Elgar. [e-book]. Amsterdam, KIT Publishers, International
Development Research Centre (http://www.idrc.ca/en/ev-
Potter S, Subrahmanian R (1998). Information needs and
33011-201-1-DO_TOPIC.html, accessed 7 July 2010).
policy change. In: Thomas A, Chataway J, Wuyts M, eds.
Finding out fast: investigative skills for policy and Walker L, Gilson L (2004). We are bitter but we are
development. London, Sage Publications:19–40. satisfied: nurses as street level bureaucrats in South
Africa. Social Science & Medicine, 59(6):1251–1261.
Riewpaiboon W et al. (2005). Private obstetric practice
in a public hospital: mythical trust in obstetric care. Social Walt G, Lush L, Ogden J (2004). International organ-
Science & Medicine, 61(7):1408–1417. izations in transfer of infectious diseases: iterative loops
of adoption, adaptation and marketing. Governance.
Rist RC (1998). Influencing the policy process with
17(2):189–210.
qualitative research. In: Denzin NK, Lincoln YS, eds.
Collecting and interpreting qualitative materials. Walt G et al. (2008). ‘Doing’ health policy analysis:
Thousand Oaks, California, Sage Publications:400–424. methodological and conceptual reflections and challenges.
Health Policy and Planning, 23(5):308-317.
Robson C (2002). Real world research: a resource for
social scientists and practitioner-researchers, 2nd ed. Yin RK (2009). Case study research: design and methods,
Oxford, Blackwell Publishing. 4th ed. Thousand Oaks, California, Sage Publications.

Sandelowski M (2000) Combining qualitative and

quantitative sampling, data collection, and analysis
techniques in mixed-method studies. Research in Nursing
& Health, 23:246–255.

Schneider H (2011). Introduction to Health Systems

Research and Evaluation. Masters in Public Health course
power point slides. Cape Town, University of Cape Town.

Sheikh K, Porter J (2010). Discursive gaps in the imple-

mentation of public health policy guidelines in India: the
case of HIV testing. Social Science & Medicine, 71(11):
2005–2013.

Sheikh K et al. (2011). Building the field of health

policy and systems research: framing the questions.
PLoS Medicine 8(8):e1001073.

Shiffman J (2007). Generating political priority for

maternal mortality reduction in 5 developing countries.
American Journal of Public Health, 97(5):796–803.

60 Gilson L, ed. (2012). Health Policy and Systems Research: A Methodology Reader
Alliance for Health Policy and Systems Research, World Health Organization