Quantitative, Qualitative,

and Mixed Methods

Analysis
 Research: The Scientific Approach
CRITERIA DESCRIPTION WHY NECESSARY
Systematic empiricism Making observations in a Aids in refuting or
systematic manner developing a theory in order
to test hypotheses
Publicly verifiable Presenting research to the Aids in determining the
knowledge public so that it that it can be veracity of a theory
observed, replicated, criticized,
and tested
Empirically solvable Stating questions in such a Aids in determining
problems way that they are answerable whether a theory can
by means of currently potentially be tested
available research technique using empirical
techniques and whether it
is falsifiable.
 Scientific Research: Its Basic Goals

1. To describe behavior
2. To predict behavior
3. To explain behavior
 Scientific Research: Its Basic Goals

1. To describe behavior
Description begins with careful observation; allows us to learn
about behavior and when it occurs; allows us to observe that 2
events are systematically related to one another; without
description as a first step, predictions cannot be made.
 Scientific Research: Its Basic Goals

2. To predict behavior
Prediction allows us to identify the factors that indicate
when an event or events will occur; For example, if we
observed that men change TV channels with greater
frequency than women, we could then make predictions
about how often men and women might change
channels when given the chance.
 Scientific Research: Its Basic Goals
3. To explain behavior
Explanation allows us to identify the causes that determine when and why a
behavior occurs; to explain a behavior, we need to demonstrate that we can
manipulate the factors needed to produce or eliminate the behavior.
For example, if gender predicts channel changing, what might cause it? It could
be genetic or environmental. Maybe men have less tolerance for commercials
and thus change channels at a greater rate. Maybe women are more interested in
the content of commercials and are thus less likely to change TV channels.
Obviously there are wide variety of possible explanations. As scientists, we test
these possibilities to identify the best explanation of why a behavior occurs.
 Descriptive Methods
Observational Method- simply observing human or animal behavior.
Naturalistic observation –observing the behavior of human or
animals in their natural habitat.
Laboratory observation – observing the behavior of humans or
animals in a more contrived and controlled situation, usually
the laboratory.
 Descriptive Methods
Case-study Method- is an in-depth study of one or more individuals.

Survey method – questioning individuals on a topic or topics and then

describing their responses (through mail, over the phone, on the Internet, or in
a personal interview).
 Predictive (Relational) Methods
Correlational Method – assesses the degree of relationship between two
measured variables. If 2 variables are correlated with each other, then we can
predict from one variable to the other with a certain degree of accuracy.

Correlation does not imply causation.

 Predictive (Relational) Methods
Positive relationship – a relationship between two variables in which an
increase in one variable is accompanied by an increase in the other variable.

Negative relationship – a relationship between two variables in which an

increase in one variable is accompanied by a decrease in the other variable.
 Predictive (Relational) Methods
Quasi- experimental method– allows us to compare naturally occurring groups of
individuals.
Example: We could examine whether alcohol consumption by students in a
fraternity or sorority differs from that of students not in such organizations.

Subject variable – a characteristic inherent in the subjects that cannot be changed.

Alternative explanation – is the idea that it is possible that some other, uncontrolled,
extraneous variable may be responsible for the observed relationship.
 Explanatory Method
Experimental method – a research method that allows a researcher to
establish a cause and effect relationship through manipulation of a variable and
control of the situation.
The basic premise of experimentation is that the researcher controls as
much as possible to determine whether a cause and effect relationship exists
between the variables being studied.
control group – is the group that serves as the baseline or “standard”
condition.
experimental group – is the group that receives the treatment.
 A Review of Research Methods
Goal Met Research Methods Advantages/
Disadvantages
Description Observational method Allows description of
behavior/s
Case study method Does not support
reliable predictions
Survey method Does not support
cause and effect
explanations
 A Review on Research Methods
Goal Met Research Methods Advantages/
Disadvantages
Prediction Correlational method Allows description of
behavior/s
Quasi-experimental Supports reliable
method predictions from one
variable to another
Does not support
cause and effect
explanations
 A Review on Research Methods
Goal Met Research Methods Advantages/
Disadvantages
Explanation Experimental method Allows description of
behavior/s

Supports reliable
predictions from one
variable to another

Supports cause and

effect explanation
 Critical Thinking Check

1. In a recent study, researchers found a negative correlation

between income level and incidence of psychological disorders.
Graduate Student A thinks this means that being poor leads to
psychological disorders. Is he correct in his conclusion? Why or why
not?
 Critical Thinking Check
2. In a study designed to assess the effects of smoking on life
satisfaction, subjects were assigned to groups based on whether or
not they reported smoking. All subjects then completed a life
satisfaction inventory.
1. What is the independent variable? Explain why.
2. What is the dependent variable? Explain why.
3. Is the independent variable a subject variable or a true
manipulated variable? Explain why.
 The interrelationship between the building blocks of research

Ontology Epistemology Methodology Methods Sources

What’s out What and How can Which precise Which data
there to how can we go about procedures can Can we
know? we know acquiring that we use to collect?
about it? knowledge? acquire it?

(Daniel & Harland, 2017, p.36 – after Grix, 2002)

ASK A SERIES OF QUESTIONS

 Qualitative Research

• Focus on making sense of or interpreting social or

human phenomena: experiences, behaviors, organizational
functioning, social movements, interactional relationships,
descriptions of culture
• Phenomena are studied in their natural settings
• Investigators interpret phenomena in terms of the meanings
people bring to them
Denzin and Lincoln, 2000.
 THE THREE MAIN RESEARCH TRADITIONS
‘QUANTITATIVE’ The collection, analysis and interpretation of primarily
numerical data in order to predict, explain or describe
phenomena.

‘QUALITATIVE’ The collection, analysis and interpretation of primarily

non-numerical data which exists in the form of words,
images, stories, etc.

‘MIXED METHODS’ Both ‘quantitative’ and ‘qualitative’ methods are

combined within the same study.
 THE THREE MAIN RESEARCH TRADITIONS
‘QUANTITATIVE’, ‘QUALITATIVE’ & ‘MIXED METHODS’

Despite the tendency for research to be characterised as ONE of

these three, such a division can be artificial, limiting and confusing.

➢ Ultimately, ‘quantitative data’ is simply data which has been defined or

‘coded’ according to qualitative concepts.

➢ The obverse of this, is that ‘qualitative data’ is often converted

to a quantitative form in order to analyse, describe, explain and
interpret. (See O’leary, 2010, p.105.)
 THE QUANTITATIVE & QUALITATIVE TRADITIONS
QUANTITATIVE QUALITATIVE
PARADIGM positivism, empiricism subjectivism, interpretivism
constructivism

METHODOLOGY scientific method, hypothesis driven, ethnomethodology, phenomenology,

deductive, reliable, valid, reproducible, ethnography, action research, inductive,
objective, generalizable subjective, idiographic, intuitive

METHODS large scale, surveys, random control trials small scale, interviewing, observation,
document analysis

DATA TYPE generally quantitative generally qualitative

ANALYSIS statistical thematic exploration

(O’Leary, 2010. p.105)

Ontological and epistemological features of methodology

(Daniel & Harland, 2017. p.39)

 Relationship between the three main research traditions

Epistemological stance

QUANTITATIVE QUALITATIVE
TRADITION POSITIVISM INTERPRETIVISM TRADITION

PRAGMATISM
MIXED METHODS

(Daniel & Harland, 2017. p.39)

 Research Methods

In the social sciences, there are 3 generic types of

research methods
• Quantitative methods
• Qualitative methods
• ‘Mixed methods’ blending quantitative and
qualitative
 Qualitative work: Common features
• The researcher is the primary instrument for data collection and
analysis
• Focuses on phenomena that occur in natural settings, and the
data are analyzed without the use of statistics.
• Involves fieldwork
• Researchers are more interested in interpreting and making sense of what
they have observed
• Not necessarily believe that there is a single “truth” to be discovered but
rather that there are multiple positions or opinions and that each have some
degree of merit.
 Qualitative Methods - Traditions
Tradition Purpose Methods Allied Discipline

Ethnography Cultural description Intensive fieldwork Anthropology

Phenomenology Explore lived Analysis of cases to Philosophy

experience formulate meaning Psychology

Biography Meaning of Interviews and Literature

experience as documents exploring the Literary Criticism
recorded through life of an individual
story
Grounded Theory Constant comparative Sociology
development, method
Theory provide an
explanation
 Qualitative Research

• Attends to description of how vs. how many

• Is an inductive and open process: the investigator builds concepts,
hypotheses, and theories from analysis of the data.
• The data collection and analysis process is iterative.
• The investigator returns to the data with new questions and ideas
to explore until there is a very deep understanding of the
phenomenon / issue.
 Features of Types of Qualitative Studies
Type of Study Description
Case Study An in-depth study of one or more individuals, groups, social settings, or
events in the hope of revealing things that are true of all of us
Archival Study A method that involves describing data that existed before the time of the
study.
Interview A method that involves asking questions in a face to face manner; it may
be conducted anywhere
Focus group A method that involves interviewing six to ten individuals at the same
Interview time
Field studies A method that involves observing everyday activities as they happen in a
natural setting
Action research Research conducted by a group of people to identify a problem, attempt
to resolve it, and then assess how successful their efforts were.
 Qualitative Quantitative
Assumptions Assumptions
• Reality is socially • Social facts have an
constructed objective reality
• Primacy of subject • Primacy of method
• *Variables are complex, • *Variables are measured
interwoven and difficult to with existing tools
measure
• Relationships between • Relationships between
variables are generally variables can be assessed
described as observed using standard statistics
patterns or cases
 Qualitative Quantitative
Purpose Purpose
• Contextualization • Generalizable findings
• Interpretation • Prediction
• Understanding peoples’ • Causal explanations
perspectives
Researcher’s Role
Researcher’s Role
• Detachment and
• Personal involvement and impartiality
partiality
• Objective portrayal
• Emphatic understanding
Design
Design
• a priori and inflexible
• Flexible and emergent
 Qualitative Quantitative
Purpose Purpose
• Contextualization • Generalizable findings
• Interpretation • Prediction
• Understanding peoples’ • Causal explanations
perspectives
Researcher’s Role
Researcher’s Role
• Detachment and
• Personal involvement and impartiality
partiality
• Objective portrayal
• Emphatic understanding
Design
Design
• a priori and inflexible
• Flexible and emergent
 Qualitative Quantitative
Approach Approach
• End with hypotheses and grounded • Begins with hypotheses and theories
theory
• Manipulation and control
• Emergence and portrayal • Experimentation
• Naturalistic • Deductive
• Inductive • Component analysis
• Searches for patterns • Seeks consensus, the norm
• Seeks pluralism, complexity • Reduces all data to numerical indices
• Minor use of numerical indices • Precise technical language, numerical
• Thick description through writing presentation
 Survey Methods
Types of Mail survey A written survey that is self-administered
Survey Telephone survey A survey conducted by telephone in which the
Methods questions are read to the respondents
Personal interview A face to face interview of the respondent

Question Open-ended questions Questions for which respondents formulate their own
Types responses
Close-ended questions Questions on which respondents must choose from a
limited number of alternatives
Partially open-ended Closed-ended questions with an open-ended “Other”
questions option
Rating scales (Likert scale) Questions on which respondents must provide a
rating on a numerical scale.
 Survey Methods
Sampling Random sampling A sampling technique in which each member of the
Techniques population is equally likely to be chosen as part of the
sample
Stratified random sampling A sampling technique intended to guarantee that the
sample represents specific subgroups or strata
Cluster sampling A sampling technique in which clusters of
participants that represent the population are
Probability sampling identified and included in the sample
Convenience sampling A sampling technique that involves getting
participants wherever is convenient
Quota sampling A sampling technique that involves getting
participants wherever you can find them and typically
wherever is convenient; however, you ensure that the
Non-probability sampling sample is like the population on certain characteristics
 Survey Methods

Concerns Sampling bias

Interviewer bias
Socially desirable responses
Return rate
Expense
 Critical Thinking Check

1. Imagine that you want to study cell phone use by drivers. You
decide to conduct an observational study of drivers by making
observations at three locations- a busy intersection, an entrance/exit
to a shopping mall parking lot, and a residential intersection. You
are interested in the number of people who use cell phones while
driving. How would you recommend conducting this study? How
would you recommend collecting the data? What concerns do you
need to take into considerations?
 Standards for Evaluation
Quantitative Qualitative
• Internal Validity • Credibility / Trustworthiness

• External Validity • (Transferability)

• Reliability • Confirmability, dependability

• Objectivity • Engagement, reflexivity

 Trustworthiness
• Ways to achieve trustworthiness:
• Member checks: recycling interpretation back to the key
informants
• Searching for disconfirming evidence
• Triangulation: multiple data sources and multiple methods
• Thick description: a thorough description of the context of
the study
 Confirmability
• Ways to address confirmability:
• Collection of data in ways that allow for audits
• Audio recordings
• Full transcripts of interviews
• Collection of low inference observational data
• Engaging a team approach
• Independent auditors
 Reflexivity
• Ways to address reflexivity:
• Document beliefs, framework, theories underlying
approach to the problem before beginning the data
collection.
• Actively journal reflections, possible biases/ limitations to
‘lens’ or ‘gaze’. Document how to overcome limitations to
more fully examine phenomena.
• Engage other perspectives in team analysis.
• Report reflexivity in final product
 Summary: What’s Involved in Qualitative Inquiry
• Spend time in the field gathering data
• Engage in data analysis to reduce raw data to themes or categories;
complex & time-consuming
• Write extensively to show multiple perspectives in order to substantiate
findings
• Participate in a form of research that does not have firm guidelines or
specific procedures, and is evolving and constantly changing

Source: Creswell JW. 1998. Qualitative inquiry and research design: choosing among
five traditions. Thousand Oaks, CA: Sage.
 Qualitative/Quantitative
• Both approaches are highly respected
• When done well, both contribute equally to the knowledge base
• When possible, researchers should blend the methods
• enriches the data to enable deeper understanding
• validates the conclusions from more than one perspective
 Summary of
Quantitative and Qualitative features
Feature Quantitative Qualitative
Role of researcher Detached, objective Engaged
Purpose Test hypothesis Describe;
Develop theory
Data collection methods A priori, structured Flexible, iterative
Data analysis Deductive, statistical Inductive, iterative leading to more
data collection and revision of
question
Type of data Numbers Predominately Words
Product Status of hypothesis Rich description; Theory
development;
A book
 Mixed Methods

General consensus that no longer a quantitative vs.

qualitative methods – both are necessary.
 Mixed Methods
Attack the research problem with an arsenal of methods
that have non-overlapping weaknesses in addition to
complementary strengths.

J Brewer & A Hunter Foundations of multimethod research: Synthesizing styles. Sage 2006. page 4.
 Mixed Methods
Scope of mixing methods
– within study
– within line of inquiry
– within broad topic area
 Approaches to mixing methods
Sequential studies
Quantitative Qualitative
Qualitative Quantitative
 Approaches to mixing methods
Mixing methods in the same study
• Dominant/ Less Dominant
Secondary method plays a small role

• Triangulation
Equal weight, both Quantitative and
Qualitative data collected & analyzed in a
complementary manner.
 Examples of mixing
Data transformation – convert data of one type to
the other can be analyzed together.

Typology development – one type of data used

to develop a typology that is used to drive analysis
with other type of data.

Extreme case analysis – pursue data collection

or analysis of data of the other type with the intent of
refining the initial explanation for the extreme case.
 The Power of Qualitative Data
• Stories are powerful and memorable.
• have a concrete, vivid, meaningful flavor that often proves far more convincing and
memorable to a reader – another researcher, a policymaker, a practitioner
• Reports include explanations of processes in context and derive
explanations
• Enable investigators to address complex social issues where little is
understood. Methods push beyond initial conceptions to generate or
revise conceptual frameworks

Source: Miles MB, Huberman AM. 1994. Qualitative data analysis: an expanded sourcebook. 2nd ed.
Thousand Oaks, CA: Sage.
Making Decisions about which Methods to Use
• What is the research question?
• What is the goal / purpose of the study?
• What kinds of information are needed to answer the Q?
• When is the information needed?
• What resources are available to conduct the study?

Given answers to the preceding questions, what methods are appropriate?

Source: Patton MQ. 1990. Qualitative evaluation and research methods, 2nd ed.
Newbury Park, CA: Sage. p. 12
 Qualitative Analysis
• Affixing codes to interview data and field notes from observations
• Noting reflections in the margins
• Sorting and sifting through these materials to identify similar phrases,
relationships, patterns, themes, commonalities, & differences
• Isolating patterns, processes, commonalities, & differences and
incorporating methods to further explore them into the next wave of data
collection
• Gradually developing a small set of generalizations about what consistently
appears in the data
• Confronting those generalizations with a formalized body of knowledge in
the form of constructs or theories
 Themes of Qualitative Inquiry
Theme Definition

Studying real-world situations as they unfold

Naturalistic naturally; non-manipulative; unobtrusive; lack of pre-
Inquiry determined constraints on outcomes
The whole phenomenon is understood as a complex
Holistic system that is more than the sum of its parts; not
perspective reduced to a few discrete variables and cause and
effect relationships
Personal Direct, close contact with people, situation, or
contact and phenomenon under study; researcher’s insights are
insight important to inquiry and understanding

Source: Patton MQ. 1990. Qualitative evaluation and research methods, 2nd ed. Newbury Park, CA: Sage.
 Themes of Qualitative Inquiry
Theme Definition
Attention to process; assumes change is constant and
Dynamic systems
ongoing
Assumes each case is special and unique; captures details
Unique case orientation of individual cases; cross-case analysis follows from
individual case studies
Places findings in a social, historical, and temporal context;
Context sensitivity
dubious of generalizations from other times and settings
Overriding objective is to understand the subject of
Empathic neutrality investigation in all its complexity rather than proving,
advocating, or advancing personal agendas
Open to adapting inquiry as understanding deepens and
Design flexibility situations change; avoids getting locked into rigid designs;
pursues new paths of discovery as they emerge
Source: Patton MQ. 1990. Qualitative evaluation and research methods, 2nd ed. Newbury Park, CA: Sage.
Defining, Measuring, and Manipulating Variables
Outline:
Defining Variables, Properties of Measurement, Scales
(Levels) of Measurement
Discrete and Continuous Variables, Types of Measures
Reliability
Validity
The Relationship Between Reliability and Validity
 Defining Variables

• Operational Definition- A definition of a variable in terms of the

operations (activities) a researcher uses to measure or manipulate it
(Kerlinger, 1996)
Example: Defining hunger as the number of hours without food is
an operational definition, whereas defining hunger as that “gnawing
feeling” is not an operational definition.
Why?
 Properties of Measurement
• Identity – a property of measurement in which objects that are different receive
different scores.
• Magnitude (also called ordinality) – a property of measurement in which the
ordering of numbers reflects the ordering of the variable.
• Equal unit size – a property of measurement in which a difference of 1 is the
same amount throughout the entire scale.
• Absolute zero – a property of measurement in which assigning a score of zero
indicates an absence of the variable being measured.
 Scales (Levels)of Measurement
• Nominal scale – a scale in which objects or individuals are assigned to categories
that have no numerical properties.

• Ordinal scale – A scale in which objects or individuals are categorized and the
categories form a rank order along a continuum.

• Interval scale – A scale in which the units of measurement (intervals) between the
numbers on the scale are all equal in size

• Ratio scale – a scale in which, in addition to order and equal units of measurement,
an absolute zero indicates an absence of the variable being measured.
 Features of Scales of Measurement

NOMINAL ORDINAL INTERVAL RATIO

Examples Ethnicity Class rank Temperature Weight
Religion Letter grade (Fahrenheit and Height
Sex Celsius) Time
Many
Psychological
Test
 Features of Scales of Measurement
NOMINAL ORDINAL INTERVAL RATIO
Properties Identity Identity Identity Identity
Magnitude Magnitude Magnitude
Equal unit size Equal unit size
Absolute zero

Mathematical Determine Determine Determine Determine

operations whether = or ≠ whether = or ≠ whether = or ≠ whether = or ≠
Determine Determine Determine
whether < or > whether < or > whether < or >
Add Add
Subtract Subtract
Multiply
Divide
 Critical Thinking Check
Identify the scale of measurement for each of the following variables.
1. ZIP code
2. Performance of teachers ( Very good, good, satisfactory, below satisfactory)
3. Reaction time
4. Score on the Graduate School Entrance Exam
5. Class rank
6. Number on a football jersey
7. Miles per gallon
 Features of Types of Measures
Self-Report Tests Behavioral Physical

Description Questionnaires or A measurement Careful Measures of

interviews that instrument used observation and bodily activity
measure how to assess recordings of
people report that individual behavior
they act, think or differences
feel
 Features of Types of Measures
Self-Report Tests Behavioral Physical
Examples Behavioral self- Ability tests Counting Weight
report behaviors EEG
GSR
Cognitive self- Personality Tests Classifying Blood pressure
report behaviors

Affective self-
report
 Features of Types of Measures
Self-Report Tests Behavioral Physical
Considerations Are subjects being Are subjects Is there Is the individual
truthful? being truthful? reactivity? taking the
measure skilled
at using the
equipment?

How reliable How reliable and

How accurate are and valid are How objective valid is the
subjects’ memories the tests? are the measuring
observers? instrument?
 Reliability
Reliability refers to the consistency or stability of a measuring instrument.

A correlation coefficient measures the degree of relationship between two

sets of scores and can vary between -1.00 and +1.00.

Positive correlation – a direct relationship between two variables in which an

increase in one is related to an increase in the other, and a decrease in one is
related to a decrease in the other.

Negative correlation – an inverse relationship between two variables in which

an increase in one variable is related to a decrease in the other and vice-versa.
Values for Weak, Moderate and Strong Correlation
Coefficients

Correlation Coefficient Strength of Relationship

± .70 – 1.00 Strong

± .30 - .69 Moderate

± .00 - .29 None (.00) to weak

Statistical Analysis: Correlation Coefficients
Types of Coefficients
Pearson Spearman Point-Biserial Phi
Type of Data Both variables Both variables One variable Both variables
must be are ordinal is interval or are nominal
interval or (ranked) ratio, and one and
ratio variable is dichotomous
nominal and
dichotomous
Correlation ±.0 -1.0 ±.0 -1.0 ± 0 -1.0 ±.0 -1.0
Reported
r2 Applicable
Yes Yes Yes Yes
Features of Types of Reliability
Test/Retest Alternate-Forms Split-Half Interrater

What is Measured Stability over time Stability over time Equivalency of Agreement
and equivalency of items between raters
items
How It is Correlate Have at least two
Accomplished Administer the Administer performance for people count or
same test to the alternate but a group of rate behaviors, and
Same people at equivalent forms of people on two determine the
two different times the test to the same equivalent halves percentage of
people at two of the same test agreement
different times between them
 Validity
• Validity – a measure of the truthfulness of a
measuring instrument. It indicates whether
the instrument measures what it claims to
measure.
 Features of Types of Validity
Content Criterion/ Criterion/ Construct
Concurrent Predictive
What it Whether the test covers The ability of the The ability of the The extent to which
measures? a representative sample test to estimate test to predict future the test measures a
of the domain of present performance theoretical construct
behaviors to be performance or trait
measured

Asks experts to assess

the test to establish that
the items are Correlate Correlate Correlate
representative of the performance on the performance on the performance on the
How It is trait being measured test with a test with a behavior test with
Accomplished concurrent behavior in the future performance on an
established test or
with people who
have different levels
of the trait the test
 Hypothesis Testing
Concept Description

Null Hypothesis The hypothesis stating that the independent

variable has no effect and that there will be no
difference between the two groups

Alternative The hypothesis stating that the independent

hypothesis or variable has an effect and that there will be a
Research hypothesis difference between the two groups.
 Hypothesis Testing
Concept Description
Two-tailed or nondirectional An alternative hypothesis stating that a difference is
test expected between the groups, but there is no
prediction as to which group will perform will
perform better or worse.

One-tailed or directional test An alternative hypothesis stating that a difference is

expected between the groups, and it is expected
between the groups, and it is expected to occur in a
specific direction.
 Hypothesis Testing
Concept Description
Type 1 error The error of rejecting the null hypothesis when we
should have failed to reject it.

Type II error The error of failing to reject the null hypothesis when
we should have rejected it.

Statistical significance When the probability of a Type I error is low (.05 or

less)
 Critical Thinking Check
1. A researcher hypothesizes that children in City A weigh less (because they spend
more time outside) than the national average. Identify the null and alternative
hypothesis. Is this a one- or two-tailed test?

Answer: The mean weight of children in City A is greater or equal to (≥) the mean weight of children
in general.
The mean weight of children in City A is less than (<) the mean weight of children in general.
This is a one-tailed test.
 Critical Thinking Check
2. A researcher collects data on children’s weights from a random sample of children in
City A weigh less than the national average. The researcher, however , does not realize
that the sample includes many children who are small for their age and that in reality
there is no difference in weight between children in City A and the national average.
What type of error is the researcher making?

Answer: The researcher concluded that there was a difference when, in reality, there was
no difference between the sample and the population. This is a Type I error.
 Analysis of Research Data: Descriptive
Statistics
• Descriptive statistics –numerical measures that describe a distribution by
providing information on the central tendency of the distribution, the width
of the distribution and the shape of the distribution.

• Measure of central tendency – a number that characterizes the

“middleness’ of an entire distribution.

• Measure of variation- a number that indicates the degree to which scores

are either clustered or spread out in a distribution.
 Measures of Central Tendency
Mean Median Mode
definition The arithmetic The middle score The score
average in a distribution of occurring with
scores organized greatest frequency
from highest to
lowest or lowest to
highest
Use with Interval and ratio Ordinal, interval, Nominal, ordinal
data and ratio data or ratio data
Cautions Not for use with Not a reliable
distributions with a measure of central
few extreme scores tendency
 Critical Thinking
1. If one teacher gave a 100 item test and one student got 25 points
higher than anyone else, which measure of central tendency
would you recommend against using?
 Measures of Variation
Range Average Deviation Standard
Deviation
definition The difference The average distance of The square root of
between the the scores from the mean the average
lowest and highest of the distribution squared deviation
scores in the from the mean of
distribution a distribution
Use with Primarily interval Primarily interval and ratio Primarily interval
and ratio data data and ratio data
Cautions A simple measure A more sophisticated The most
that does not use measure in which all sophisticated and
all scores in the scores are used but which most frequently
distribution in its may not weight extreme used measure of
calculation scores adequately variation.
 Inferential Statistics
• Inferential statistics –procedures for drawing conclusions about a
population based on data collected from a sample.

• Parametric tests – a statistical test that involves making assumptions about

estimates of population characteristics or parameters.

• Nonparametric tests- a statistical test that does not involve the use of any
population parameters, mean of the population and standard deviation of
the population are not needed, and the underlying distribution does not have
to be normal.
 Analysis of Data
• To determine whether the SAT scores of teachers for 10 Language majors in
Class 1 at University C have higher average SAT scores than the population
of students at University C. Paired t-Test was conducted and the SPSS
output is presented below set at .05 significance level.
One –sample Statistics
N Mean Std. Deviation Std. Error
Mean
SAT score 10 1176.0000 131.80457 41. 68026

Test Values
95% Confidence Interval of
Mean the Difference
t df Sig. (2-tailed) Difference Lower Upper
SAT score 2.063 9 .035 86.0000 -8.2873 180.2873
 Analysis of Data
• To determine whether the SAT scores of teachers for 10 Language majors in
Class 1 at University C have significantly different average SAT scores than
the population of students at University C. Paired t-Test was conducted and
the SPSS output is presented below set at .05 significance level.
What could be the null hypothesis and alternative hypothesis in this situation?
Is there enough evidence to support to allow us to reject the null hypothesis?
Justify your answer.
What kind of a test is this?
Make a conclusion based on the SPSS output given.
Type I and Type II errors
 Analysing quantitative data
• Always good to group and/or visualise the data initially
→ outliers/cleaning data
• What average are you looking for?
Mean, median or mode?
• Spread of data:
• skewness/distribution
• range, variance and standard deviation
 What are you looking for?
• Trying to find the signal from the noise
• Generally, either a difference (between/within groups) or a
correlation
• Choosing the right test to use:
parametric vs non-parametric (depends what sort of data you have
– interval/ratio vs nominal/ordinal and how it is distributed)
• Correlation does not imply causation!
 Example correlations

From ‘Spurious correlations’

website
http://www.tylervigen.com
/spurious-correlations
 Interpreting test statistics

• Significance level – a fixed probability of wrongly rejecting the null

hypothesis H0, if it is in fact true. Usually set to 0.05 (5%).
• p value - probability of getting a value of the test statistic as extreme as or
more extreme than that observed by chance alone, if the null hypothesis H0,
is true.
• Power – ability to detect a difference if there is one
• Effect size – numerical way of expressing the strength or magnitude of a
reported relationship, be it causal or not
 Example of quant data/analysis*

• Matched users were those who learning styles were matched with the lesson plan e.g.
sequential users with a sequential lesson plan. Mismatched participants used a lesson
plan that was not matched to their learning style, e.g. sequential users with a global
lesson plan.

• H0 – there will be no statistically significant difference in knowledge gained between

users from different experimental groups
• H1 – students who learn in a matched environment will learn significantly better
than those who are in mismatched environment
• H2 – students who learn in a mismatched environment will learn significantly worse
than those who learn in a matched environment
* Case study taken from: Brown, Elizabeth (2007) The use of learning styles in adaptive hypermedia. PhD
thesis, University of Nottingham. http://eprints.nottingham.ac.uk/10577/
 Interpreting test statistics

• Statistical testing was carried out using a univariate ANOVA in SPSS, to determine if there was any
significant difference in knowledge gained.
• Initial conjecture suggests that the mismatched group actually performed better than the matched
group.
• However, the difference between the two groups was not significant (F(1,80)=0.939, p= 0.34, partial
eta squared = 0.012) and hence hypotheses 1 and 2 can be rejected.
 What quant researchers worry about
• Is my sample size big enough?
• Have I used the correct statistical test?
• have I reduced the likelihood of making Type I and/or Type II errors?
• Are my results generalisable?
• Are my results/methods/results reproducible?
• Am I measuring things the right way?
 What’s wrong with quant research?
• Some things can’t be measured – or measured accurately
• Doesn’t tell you why
• Can be impersonal – no engagement with human behaviours or individuals
• Data can be static – snapshots of a point in time
• Can tell a version of the truth (or a lie?)
“Lies, damned lies and statistics” – persuasive power of numbers
 Qualitative approaches
• Any research that doesn’t involve numerical data
• Instead uses words, pictures, photos, videos, audio recordings. Field notes,
generalities. Peoples’ own words.
• Tends to start with a broad question rather than a specific hypothesis
• Develop theory rather than start with one
→ inductive rather than deductive
 Gathering qual data
• Tends to yield rich data to explore how and why things happened
• Don’t need large sample sizes (in comparison to quantitative research)
• Some issues may arise, such as
• Respondents providing inaccurate or false information – or saying what they think the
researcher wants to hear
• Ethical issues may be more problematic as the researcher is usually closer to
participants
• Researcher objectivity may be more difficult to achieve
 Sources of qual data
• Interviews (structured, semi-structured or unstructured)
• Focus groups
• Questionnaires or surveys
• Secondary data, including diaries, self-reporting, written accounts of past
events/archive data and company reports;
• Direct observations – may also be recorded (video/audio)
• Ethnography
 Analysing qual data
• Content analysis
• Grounded analysis
• Social network analysis (can also be quant)
• Discourse analysis
• Narrative analysis
• Conversation analysis
 Example of qual data research*

• Describing and comparing two types of

audio guides: person-led and technology-led
• Geolocated audio to enable public, informal
learning of historical events
• Data sources: questionnaires, researcher
observations, and small focus groups

* Taken from: FitzGerald, Elizabeth; Taylor, Claire and Craven, Michael (2013). To the
Castle! A comparison of two audio guides to enable public discovery of historical events.
Personal and Ubiquitous Computing, 17(4) pp. 749–760. http://oro.open.ac.uk/35077/
 Data analysis and findings
• Comparison of the two different walks
• Differences/similarities of the walks
• Issues surrounding participant engagement
• Thematic analysis
• Mode of delivery
• Number of participants and social interactions
• Geographical affordances of places and locations
• User experience
• Opportunities for learning
• Other factors
• Findings, lessons learned, recommendations
 What qual researchers worry about
• Have I coded my data correctly?
• Have I managed to capture the situation in a realistic manner?
• Have I described the context in sufficient detail?
• Have I managed to see the world through the eyes of my participants?
• Is my approach flexible and able to change?
 What’s wrong with qual research?
• It can be very subjective
• It can’t always be repeated
• It can’t always be generalisable
• It can’t always give you definite answers in the way that quantitative research
can
• It can be easier to carry out (or hide) ‘bad’ (poor quality) qual research than
‘bad’ quant research
 Other aspects of research design
• Validity
• Reliability
• Trustworthiness*
• Dependability: showing that the findings are consistent and could be repeated
• Confirmability: a degree of neutrality or the extent to which the findings of a study are
shaped by the respondents and not researcher bias, motivation, or interest
• Credibility: confidence in the 'truth' of the findings
• Transferability: showing that the findings have applicability in other contexts
* See Lincoln, YS. & Guba, EG. (1985). Naturalistic Inquiry.
Newbury Park, CA: Sage Publications.
 Summary
• The type of approach you choose will be determined by your
research question, your epistemological and ontological stances
and your skills or ability to utilise a certain appoach
• For most people in ed tech, a mixed methods approach will be
used
• So long as you make an informed choice and can justify it, it
should be fine ☺
• Just be aware of the limitations of your approach(es) and try to
compensate where necessary