ERES4123 : Research Methodology in

Education

Chapter 4
Data Collection Method and Sampling

Dr. Siti Farhah A Aziz

1
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

OUTLINE

• Research method overview

© School of Education & Humanities | 2

 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Research Methodology
The methodology is an outline of the overall data collection and
analysis strategy that will be used to implement the research cycle

• The methodology should:

✔ Be compatible with the preliminary data analysis plan
✔ Be designed in a way that ensures the intended scope of the research (i.e. objectives
and research questions) can be feasibly achieved to the required quality, given the time,
resources and access available

• Designing a methodology involves three key components:

✔ Selecting the overall research method
✔ Selecting the appropriate data collection approach(es)
✔ Designing the sampling strategy
© School of Education & Humanities | 3
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Categories of Research Methods

Research methods are broadly distinguished between the following categories:

Quantitative Qualitative Mixed Methods

✔ Measure prevalence of ✔ Explore and discover themes, ✔ Combines both qualitative and
issues, verify hypotheses and develop theories, rather than quantitative to
establish causal relations verify hypotheses and measure
✔ (1) collect and analyse both types
between variables occurrences
of data and
✔ Large samples, structured ✔ Smaller samples,
✔ (2) use both approaches in
data collection, semi-structured data collection,
tandem
© School of Education & Humanities | 4
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Selecting your Research Methods

What factors to consider when choosing one research method over

another?
✔ Overall applicability to meet research objectives
✔ Time i.e. key planning and decision-making milestones to inform
✔ Resources available
✔ Material resources
✔ Financial resources
✔ Human resources
✔ Access to population of interest

© School of Education & Humanities |

 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Differences between quantitative & qualitative research

Distinction between the two can be made based on the following three criteria:

© School of Education & Humanities |

 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Differences between quantitative & qualitative research

Distinction between the two can be made based on the following three criteria:

Quantitative Qualitative

1. Type of data collection Structured, close-ended Semi-structured (but not

data collection tools unstructured) data collection
tools
2. Type of analysis Measuring prevalence, Exploratory, and primarily
quantifying issues, and involves inductive analysis
primarily involves deductive
analysis

3. Type of sampling Can use both probability or Non-probability sampling 🡪

strategy non-probability sampling 🡪 generalisation to the wider
generalisation to the wider population not possible
population possible
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Unit of Measurement

The unit that will be used to record,

measure and analyze observations/
information collected

Examples?

Individual
Family
Household
Community/ group
Town/ village
Facility
Cow

© School of Education & Humanities |

 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Unit of Measurement
✔ Unit will impact the time, resources needed to collect and analyse information
✔ Unit will define the depth of information possible and scope of analysis

Household level
Individual level
information

o
s s ible t ower
po l
Community/Group level It is e from a ent
Depth of

t m
g g rega easure ehold
a of m ous ly
unit d (e.g. h ut rare
ar )b
upw munity rsa
m e
to co vice-v
Location level

Time / Cost / Access

© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Data Collection Strategies

No one best way: decision depends on:

▪ What you need to know: numbers or stories

▪ Where the data reside: environment, files, people
▪ Resources and time available
▪ Complexity of the data to be collected
▪ Frequency of data collection
▪ Intended forms of data analysis

© School of Education & Humanities |

 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Rules for Collecting Data

If must collect original data:

• be sensitive to burden on others
• pre-test, pre-test, pre-test
• establish procedures and follow them (protocol)
• maintain accurate records of definitions and coding
• verify accuracy of coding, data input

© School of Education & Humanities |

 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Types of Data

© School of Education & Humanities |

 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Structured vs Semi Structured Approach

Structured Semi-Structured

• All data collected in the same way • Systematic and follow general
• Especially important for multi-site procedures but data are not
and cluster evaluations so you collected in exactly the same way
can compare every time
• Important when you need to make • More open and fluid
comparisons with alternate • Does not follow a rigid script
interventions ✔ may ask for more detail
✔ people can tell what they want
in their own way

© School of Education & Humanities |

 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

When to use?
Structured Semi-Structured

• need to address extent questions • conducting exploratory work

• have a large sample or population • seeking understanding, themes,
• know what needs to be measured and/or issues
• need to show results numerically • need narratives or stories
• need to make comparisons across • want in-depth, rich, “backstage”
different sites or interventions information
• seek to understand results of
data that are unexpected

© School of Education & Humanities |

 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Data collection approaches: Quantitative

• Data in numerical form

• Data that can be precisely measured
• age, cost, length, height, area, volume,
weight, speed, time, and temperature
• Harder to develop
• Easier to analyze

© School of Education & Humanities |

 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Data collection approaches: Qualitative

• Data that deal with description

• Data that can be observed or self-reported,
but not always precisely measured
• Less structured, easier to develop
• Can provide “rich data” — detailed and widely
applicable
• Is challenging to analyze
• Is labor intensive to collect
• Usually generates longer reports

© School of Education & Humanities |

 ECOM4113 – Introduction to Communication
Chapter 1 – Communication and Its Process

Types of Quantitative Research design

© School of Education & Humanities | 17

 ECOM4113 – Introduction to Communication
Chapter 1 – Communication and Its Process

Types of Qualitative Research design

© School of Education & Humanities | 18

 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Data Collection Tools

• Participatory Methods
• Records and Secondary Data
• Observation
• Surveys and Interviews
• Focus Groups
• Diaries, Journals, Self-reported Checklists
• Expert Judgment
• Delphi Technique
• Other Tools

IPDET © 2009 19
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Tool 1: Participatory Methods

• Involve groups or communities heavily in data collection
• Examples:
• community meetings
• mapping
• transect walks

IPDET © 2009 20
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Community Meetings

• One of the most common participatory methods

• Must be well organized
• agree on purpose
• establish ground rules
• who will speak
• time allotted for speakers
• format for questions and answers

IPDET © 2009 21
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Mapping

• Drawing or using existing maps

• Useful tool to involve stakeholders
• increases understanding of the community
• generates discussions, verifies secondary sources of information,
perceived changes
• Types of mapping:
• natural resources, social, health, individual or civic assets, wealth, land
use, demographics

IPDET © 2009 22
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Transect Walks
• Evaluator walks around community observing people, surroundings, and
resources
• Need good observation skills
• Walk a transect line through a map of a community — line should go
through all zones of the community

IPDET © 2009 23
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Tool 2: Records and Secondary Data

• Examples of sources:
• files/records
• computer data bases
• industry or government reports
• other reports or prior evaluations
• census data and household survey data
• electronic mailing lists and discussion groups
• documents (budgets, organizational charts, policies
and procedures, maps, monitoring reports)
• newspapers and television reports

IPDET © 2009 24
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Using Existing Data Sets

Key issues: validity, reliability, accuracy, response rates, data dictionaries,

and missing data rates

IPDET © 2009 25
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Advantage/Challenge: Available Data

Advantages Often less expensive and faster than
collecting the original data again

Challenges There may be coding errors or other

problems. Data may not be exactly
what is needed. You may have
difficulty getting access. You have to
verify validity and reliability of data

IPDET © 2009 26
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Tool 3: Observation
• See what is happening
• traffic patterns
• land use patterns
• layout of city and rural areas
• quality of housing
• condition of roads
• conditions of buildings
• who goes to a health clinic

IPDET © 2009 27
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
Chapter 1 – Communication and Its Process

Observation is Helpful when:

• need direct information
• trying to understand ongoing behavior
• there is physical evidence, products, or outputs than can be observed
• need to provide alternative when other data collection is infeasible or
inappropriate

IPDET © 2009 28
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Degree of Structure of Observations

• Structured: determine, before the observation, precisely what

will be observed before the observation
• Unstructured: select the method depending upon the situation
with no pre-conceived ideas or a plan on what to observe
• Semi-structured: a general idea of what to observe but no
specific plan

IPDET © 2009 29
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Ways to Record Information from Observations

• Observation guide
• printed form with space to record
• Recording sheet or checklist
• Yes/no options; tallies, rating scales
• Field notes
• least structured, recorded in narrative, descriptive style

IPDET © 2009 30
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Guidelines for Planning Observations

• Have more than one observer, if feasible

• Train observers so they observe the same things
• Pilot test the observation data collection instrument
• For less structured approach, have a few key questions in mind

IPDET © 2009 31
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Advantages and Challenges: Observation

Advantages Collects data on actual vs. self- reported

behavior or perceptions. It is real-time vs.
retrospective

Challenges Observer bias, potentially unreliable;

interpretation and coding challenges;
sampling can be a problem; can be labor
intensive; low response rates

IPDET © 2009 32
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Tool 4: Surveys and Interviews

• Excellent for asking people about:
• perceptions, opinions, ideas
• Less accurate for measuring behavior
• Sample should be representative of the whole
• Big problem with response rates

IPDET © 2009 33
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Structures for Surveys

• Structured:
• Precisely worded with a range of pre-determined
responses that the respondent can select
• Everyone asked exactly the same questions in
exactly the same way, given exactly the same
choices
• Semi-structured
• Asks same general set of questions but answers to
the questions are predominantly open-ended

IPDET © 2009 34
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Structured vs. Semi-structured Surveys

Structured harder to develop

easier to complete
easier to analyze
more efficient when working with large numbers
Semi-struct easier to develop: open ended questions
ured more difficult to complete: burdensome for people to
complete as a self-administrated questionnaire
harder to analyze but provide a richer source of data,
interpretation of open-ended responses subject to bias

IPDET © 2009 35
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Modes of Survey Administration

• Telephone surveys
• Self-administered questionnaires distributed by mail, e-mail, or websites
• Administered questionnaires, common in the development context
• In development context, often issues of language and translation

IPDET © 2009 36
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Mail / Phone / Internet Surveys

• Literacy issues
• Consider accessibility
• reliability of postal service
• turn-around time
• Consider bias
• What population segment has telephone access? Internet access?

IPDET © 2009 37
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Advantages and Challenges of Surveys

Advantages Best when you want to know what

people think, believe, or perceive, only
they can tell you that
Challenges People may not accurately recall their
behavior or may be reluctant to reveal
their behavior if it is illegal or stigmatized.
What people think they do or say they
do is not always the same as what they
actually do.

IPDET © 2009 38
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Interviews
• Often semi-structured
• Used to explore complex issues in depth
• Forgiving of mistakes: unclear questions can be
clarified during the interview and changed for
subsequent interviews
• Can provide evaluators with an intuitive sense of the
situation

IPDET © 2009 39
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Challenges of Interviews
• Can be expensive, labor intensive, and time consuming
• Selective hearing on the part of the interviewer may miss information that
does not conform to pre-existing beliefs
• Cultural sensitivity: e.g., gender issues

IPDET © 2009 40
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Tool 5: Focus Groups

• Type of qualitative research where small homogenous groups of people
are brought together to informally discuss specific topics under the
guidance of a moderator
• Purpose: to identify issues and themes, not just interesting information,
and not “counts”

IPDET © 2009 41
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Focus Groups Are Inappropriate when:

• language barriers are insurmountable
• evaluator has little control over the situation
• trust cannot be established
• free expression cannot be ensured
• confidentiality cannot be assured

IPDET © 2009 42
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Focus Group Process

Phase Action

1 Opening Ice-breaker; explain purpose; ground rules;

introductions
2 Warm-up Relate experience; stimulate group interaction;
start with least threatening and simplest questions
3 Main body Move to more threatening or sensitive and
complex questions; elicit deep responses; connect
emergent data to complex, broad participation
4 Closure End with closure-type questions; summarize and
refine; present theories, etc; invite final comments
or insights; thank participants

IPDET © 2009 43
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Advantages and Challenges of Focus Groups

Advantages Can be conducted relatively quickly and easily;

may take less staff time than in-depth, in-person
interviews; allow flexibility to make changes in
process and questions; can explore different
perspectives; can be fun

Challenges Analysis is time consuming; participants not be

representative of population, possibly biasing the
data; group may be influenced by moderator or
dominant group members

IPDET © 2009 44
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Tool 6: Diaries and Self-Reported Checklists

• Use when you want to capture information about events in people’s daily
lives
• Participants capture experiences in real-time not later in a questionnaire
• Used to supplement other data collection

IPDET © 2009 45
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Guidelines for Diaries or Journals

Step Process

1 Recruit people face-to-face

• encourage participation, appeal to altruism, assure
confidentiality, provide incentive
2 Provide a booklet to each participant
• cover page with clear instructions, definitions, example
• short memory-joggers, explain terms, comments on last
page , calendar
3 Consider the time-period for collecting data
• if too long, may become burdensome or tedious
• if too short may miss the behavior or event
IPDET © 2009 46
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Self-reported Checklists
• Cross between a questionnaire and a diary
• The evaluator specifies a list of behaviors or events and asks the
respondents to complete the checklist
• Done over a period of time to capture the event or behavior
• More quantitative approach than diary

IPDET © 2009 47
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Advantages and Challenges of Diaries and

Self-reported Checklists
Advantages Can capture in-depth, detailed data that might be
otherwise forgotten
Can collect data on how people use their time
Can collect sensitive information
Supplements interviews provide richer data

Challenges Requires some literacy

May change behavior
Require commitment and self-discipline
Data may be incomplete or inaccurate
Poor handwriting, difficult to understand phrases

IPDET © 2009 48
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Tool 7: Expert Judgment

• Use of experts, one-on-one or as a panel
E.g., Government task forces, Advisory Groups

• Can be structured or unstructured

• Issues in selecting experts

IPDET © 2009 49
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Selecting Experts
• Establish criteria for selecting experts not only on recognition as expert but
also based on:
• areas of expertise
• diverse perspectives
• diverse political views
• diverse technical expertise

IPDET © 2009 50
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Advantages and Challenges of Expert

Judgment
Advantages Fast, relatively inexpensive

Challenges Weak for impact evaluation

May be based mostly on
perceptions
Value of data depends on how
credible the experts are perceived to
be

IPDET © 2009 51
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Tool 8: Delphi Technique

• Enables experts to engage remotely in a dialogue and reach consensus,
often about priorities
• Experts asked specific questions; often rank choices
• Responses go to a central source, are summarized and fed back to the
experts without attribution
• Experts can agree or argue with others’ comments
• Process may be iterative

IPDET © 2009 52
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Advantages and Challenges of Delphi

Technique
Advantages Allows participants to remain anonymous
Is inexpensive
Is free of social pressure, personality influence,
and individual dominance
Is conducive to independent thinking
Allows sharing of information
Challenges May not be representative
Has tendency to eliminate extreme positions
Requires skill in written communication
Requires time and participant commitment

IPDET © 2009 53
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

Data Collection Summary

Choose more than one data collection technique
No “best” tool
Do not let the tool drive your work but rather choose the right tool to
address the evaluation question

IPDET © 2009 54
© School of Education & Humanities |
 ECOM4113 – Introduction to Communication
ERES4123 Research Methodology in Education
Chapter 1 – Communication and Its Process

A Final Note….

“I never guess. It is a capital mistake

to theorize before one has data.
Insensibly one begins to twist facts and theories,
instead of theories to suit facts.”
--Sir Arthur Conan Doyle

Questions?

IPDET © 2009 55
© School of Education & Humanities |
 ERES4123 : Research Methodology in
Education

Thank you

56