CREATING THE

APPROPRIATE
RESEARCH DESIGN
OUTLINE
• Weak Experimental Research Designs
• Strong Experimental Research Designs
• How To Choose or Construct the Appropriate Experimental Design
• After a research topic has been selected and decisions have been made about the
independent and dependent variables, it is necessary to develop a plan for collecting data
and testing the effect of one or more independent variable (IV) on the dependent variable
(DV).

• This plan is the research design of the experiment.

• The term research design refers to the outline, plan, or strategy that specifies the
procedure to be used in seeking an answer to your research question(s).

• It specifies such things as how to collect and analyze the data.

• The goal in research is to use the strongest design that is possible, ethical, and feasible for
your research question.
• But what makes a design strong or weak?
• Strong designs often include:
- pretests,
- control groups,
- random assignment.
 Weak Experimental Research Designs
• Designs that do not control for many extraneous variables and provide weak
evidence of cause and effect.
A- One-Group Posttest-Only Design
• In the one-group posttest-only design, a single group of research participants is measured
on a dependent variable after having undergone an experimental treatment.
• For example, perhaps an institution starts a training program (the experimental treatment
condition). The institution wants to evaluate the effectiveness of the program; so upon
completion of the program, it assesses knowledge, attitudes, and behavioral outcomes of
the program participants. If the outcome (i.e., dependent variable) measures are positive,
the administrator might want to conclude that the program worked.
• The structure of the one-group posttest-only design includes an experimental manipulation
(X) followed by measurement (O) of the dependent variable(s).
• The design does not have:
- a no-treatment control group (which would allow a comparison of participants’
posttest performance with the performance of a ­similar group that did not receive
the treatment),
- a pretest (which would allow a comparison of participants’ posttest performance
with their performance prior to treatment).

• Because this design does not include either of these comparisons, it should be
viewed as a faulty design.

• It is ­difficult to know if any effect is due to the treatment or to some confounding

extraneous variable.
B- One-Group Pretest–Posttest Design
• Design in which a treatment condition is interjected between a pretest and posttest of the
dependent variable.

• A group of research participants is measured on the dependent variable, O, prior to

administration of the treatment condition. The independent variable, X, is then
administered, and O is again measured. The difference between the pretest and posttest
scores is taken as an indication of the effectiveness of the treatment condition.

• For example, imagine that your school district adopted an expensive new curriculum for
reading in first grade. Students’ reading was measured at the beginning of the school year
(pretest, O), the reading curriculum was employed daily throughout the academic year
(treatment, X), and reading was measured again at the end of the year (posttest, O). Results
indicate that students’ reading improved by a full grade level.
• Researchers should have included an equated group of first graders who did not receive the
new curriculum.
• If a significant difference had been found between the scores of these two groups, it could
have been attributed to the influence of the experimental curriculum, because both groups
would have experienced any history, testing, regression artifact, instrumentation, and
maturation effects that had occurred, and, therefore, these variables would have been
controlled.
• The design of onegroup pretest–posttest study is weak, not so much because the sources of
rival hypotheses can affect the results, but because in most cases we do not know if they did.
C- Posttest-Only Design with Nonequivalent Groups
• The primary disadvantage of the previous two designs is that we cannot know if the
independent variable influenced the dependent variable.
• It’s a design in which the performance of an experimental group is compared with that of
a nonequivalent control group at the posttest.
• In this design, one group of research participants receives the treatment condition (X)
and is then ­compared on the dependent variable (O) with a group that did not receive
this treatment condition.
• The problem is that in this design, the comparison group is a nonequivalent group.
• That is, the participants in the comparison group might differ in important ways from the
participants in the experimental group.
• This threat to internal validity was called ­selection (groups that differ on more variables than
just the independent variable).
• In our example of the new reading curriculum, let’s imagine that some schools adopt the
new curriculum and others do not. At the end of the year, we compare reading scores of the
schools that adopted the curriculum to those that did not, and we observe that the schools
with the new curriculum scored better on the reading test. The problem is that these groups
might differ in many important ways including initial reading levels of the students, parental
involvement, and parental education levels. The researcher will not know whether the
posttest difference in reading level is due to the treatment or due to one of these initial
differences between the groups.
• The only way to ensure that the groups are equated is to assign participants
randomly to the two groups.
 Strong Experimental Research Designs
• Strong experimental research designs have greater internal validity.
• That is, they provide more assurance that the effect of the independent variable
(IV) on the dependent variable (DV) has been isolated and tested.
• In order to achieve internal validity, we must eliminate potential rival hypotheses.
• This can be accomplished by two primary means:
- control techniques,
- control group.

• The most important control technique, is random assignment to groups (also called
randomization) because this is the only means by which known and unknown variables can
be controlled.
• A popular synonym for designs with random assignment to the experimental and control
groups is RCT (which stands for randomized controlled trial).
• A control group is a group of research participants that do not receive the active level
of the independent variable; they might either receive zero amount of the
independent variable or receive an amount that is in some sense a standard value,
such as what they would typically receive if they were not participants in research.

• An experimental group (also called a treatment group) is a group of research

participants that receive some level of the independent variable that is intended to
produce an effect.

• A control group serves two functions.

- source of comparison. The control group is used to estimate the counterfactual =
What the experimental group participants’ responses would have been if they had not
received the treatment.
- control for rival hypotheses. If a control group is included and random assignment is
used, extraneous variables will equally impact the performance of both the control and
the experimental participants, effectively holding the influence of the extraneous
variables constant.
If an extraneous variable affects both groups equally, then the groups will not differ on
that variable, enabling the researcher to conclude that the reason the groups differed
at the posttest was because of the treatment.
• Strong experimental designs = Designs that effectively control extraneous
variables and provide strong evidence of cause and effect.

A- Between-Participants Designs
• Groups are produced by random assignment, and the different groups are exposed
to the different levels of the independent variable (uses a between subjects IV).
• This kind of independent variable is called a between participants variable (or a
between-subjects variable).
• The use of random assignment of the ­participants to different groups eliminates
most of the threats to internal ­validity. Because these designs rely on random
assignment, they are also called ­randomized designs.
• The basic between-participants research design is the posttest-only control-group
design.
Posttest-Only Control-Group Design
• Administration of a posttest to two or more randomly assigned groups of
participants that receive the different levels of the independent variable.
• For example, if a researcher was investigating the effects of one independent
variable (e.g., social skills training) and the presence-versus-absence form of
variation (one group receives the training, one group does not) was being used
with this independent variable, participants would be randomly assigned to two
groups.
• This design is similar in appearance to the nonequivalent posttest-only design,
except for one important difference: The nonequivalent posttest-only design lacks
random assignment.
• The between-participants posttest-only control-group design provides the
necessary equivalence on extraneous variables by randomly assigning participants
to two or more groups.
• All possible known and unknown extraneous variables are controlled (excluding
those such as experimenter expectancies and any differential treatment the
participants might receive during the experiment on any variable other than the
independent variable).
• By including a randomized control group, all of the threats to internal validity are
controlled.

• These threats are controlled because the effects of these extraneous variables
affect both the experimental and the control groups.

• As long as the extraneous variables affect both groups equally, any observed
difference between the groups will not be due to these variables, and we can
assume that the difference is due to the independent variable.
Strengths and Weaknesses of the Posttest-Only Control-Group Design
 Difficulties

1- Although randomization is the best control technique available for achieving

equivalence, it does not provide complete assurance that the necessary equivalence
has been attained.
This is particularly true when the group of participants being randomized is small
(e.g., less than 30 participants).
If there is any doubt that random assignment will work, it is advisable to combine
matching and statistical control along with the randomization technique.

2- Lack of a pretest.
 Strengths

• Inclusion of a control group and random assignment of participants to the groups.

• These features produce a very strong design for eliminating threats to internal
validity.
• For example, let’s say that a recent study found that college students who were
assigned to monthly meetings with their advisor were more likely to complete
college. We might want to know if meeting each month was necessary for the
program to be successful. To obtain information about how many advisor ­sessions
are needed, the researcher could randomly assign students to three different
groups that varied on the number of sessions received. In one experimental group,
the participants would meet with their advisor once each month for a year. In a
second experimental group, the participants would meet with their advisor once
every 2 months, and a control group would be included that did not have any
meetings with their advisor.
• This design includes more than one experimental group and it allows to pose and
address more specific research questions.
Posttest-only control group design with three levels of
variation of the independent variable.
B- Within-Participants Designs
• In a within-participants design, all research participants are members of all experimental
conditions in the experiment.

• Within-participants designs are also called repeated measures designs because all
participants are ­measured “repeatedly”.

• The basic within-participants design is the within-participants posttest-only design in

which the participants are given a posttest measuring their performance on the dependent
variable after they have been exposed to each experimental condition.

• This kind of independent variable is called a within-participants variable or a within-

subjects variable.
• Mahoney, Taylor, Kanarek, and Samuel (2005) to test the effect of breakfast type on
cognitive performance. Elementary school students participated in three within-
participant sessions (on 3 different days). Each session consisted of one of three types of
breakfast (cereal, oatmeal, no breakfast) followed by a series of cognitive tasks. Order of
breakfast type was counterbalanced across the three sessions. Mahoney et al. found that
the students performed worst in the no breakfast sessions and best in the oatmeal
sessions.

• Because of the variability among different people, within-participants designs (where

participants serve as their own control) are often the designs that are employed in
research using cognitive or physiological measures.
• Because of this, participants serve as their own control, and variables such as age,
gender, and prior experience remain constant over the entire experiment.

• In other words, if all participants are in all conditions, the conditions can’t differ
because some kinds of people are in one condition but not in another.
Strengths and Weaknesses of Within-Participants Designs

 Strengths

• Because the participants serve as their own control, they are perfectly matched in the
various treatment conditions; this increases the sensitivity of the experiment.
• the within-participants design does not require as many participants as does the between-
participants design. If 25 participants are needed in each treatment condition and there are
three treatment conditions, then only 25 participants are needed in a within-participants
design, whereas 75 participants (25 times 3) are needed in the between-participants design.
When participants are difficult to obtain, this within-participants design advantage is
important.
 Disadvantages

1- Within-participants designs can be taxing on participants because they have to be present for
multiple treatment conditions.
2- Confounding influence of a sequencing effect. Because the primary characteristic of a within-
participants design is that all participants participate in all experimental treatment conditions, a
sequencing rival hypothesis is a real possibility. Fortunately, you can use the control technique
of counterbalancing to help rule out the sequencing threat to internal validity.
Noting that the counterbalancing controls only linear sequencing effects; if the sequencing
effects are nonlinear (called differential carryover effects), then a confounding carryover
sequencing effect will remain
 As a result, the within-participants design is not the most commonly used design.
C- Mixed Designs (i.e., Combination of Between and Within)

Pretest–Posttest Control-Group Design

• Administration of a posttest to two or more randomly assigned groups of participants after
the groups have been pretested and administered the different levels of the independent
variable.
• This is a mixed design because it has a between-subjects IV (the treatment groups composed
of different participants) and a within-subjects factor of “time” (where all participants receive
the pretest at time 1 and the posttest at time 2).
• Because of random assignment, this design is strong on internal validity, ruling out all of the
basic threats to internal validity.
• For example, you decide to conduct an experiment using the pretest– posttest
control group design to test a treatment for social anxiety. You randomly assign
your 100 participants to the two levels of the treatment variable where the control
group does not receive anxiety reduction therapy and the experimental group
does receive anxiety reduction therapy. You have 50 participants in both groups
making up your between-subjects IV. Your within-subjects independent variable is
“time,” because you measured participants’ anxiety level before the intervention
(pretest) and after the intervention (posttest). Your dependent variable is level of
social anxiety.

• Both groups had high social anxiety at the beginning of your experiment (pretest
means), but after the intervention, on the posttest, the experimental group was
lower in anxiety than was the control group. This is exactly what was hoped for.
The treatment helped the experimental group members lower their anxiety.
Mean results of pretest–posttest control-group design
studying effectiveness of anxiety reduction therapy.
Advantages and Disadvantages of Including a Pretest
 Advantages

1- When a pretest is included, the researcher does not have to assume that
randomization worked properly; the researcher can check to see if the groups are
similar on the dependent variable after random assignment but before the
experimental conditions are introduced. + If the researcher measures the
participants on additional variables relevant to the study, initial comparability can
also be checked on those variables (e.g., motivation, intelligence, attitudes).

2- If a pretest is included, the researcher can determine if a ceiling effect (situation

where participants’ pretest scores on the dependent variable are too high to allow
for additional increases) or floor effect (situation where participants’ pretest scores
on the dependent variable are too low to allow for additional decreases) is likely to
occur.
• For example, if a dependent variable had a maximum value of 100 and the average
score for some participants was 98 or 99 on the pretest, then there would be little
room left for improvement. If no treatment effect is found, pretest scores should be
examined to determine if a ­ceiling or floor effect might have occurred.

3- If the experimental and control groups are slightly different on the dependent
variable measure at the pretest, the researcher can use a statistical technique called
analysis of covariance to statistically control for these pretest differences. Not only
does this statistical technique adjust for pretest differences, it also provides a more
accurate and powerful test of the differences between the experimental and control
group posttest scores.

4- To gain an empirical demonstration of whether an overall change in response

occurred from pretesting to post-testing. The most direct way of gaining such evidence
of change is to determine if there is a statistically significant difference between the
pretest and posttest change scores of the experimental and control groups
 Disadvantages

• The participants might change in some way because they were given a pretest.
Note, however, that because both groups were given the pretest, they both should
be affected equally by taking the pretest; therefore, the internal validity of the
design is not weakened.
• However, external validity can sometimes be weakened when a pretest is included
in the design.
D- Factorial Designs
• When there is more than one independent variable of interest, a factorial design is the
experimental design of choice.
• In a factorial design, two or more independent variables are simultaneously studied to ­
determine their separate and interactive effects on the dependent variable.
• The independent variables in factorial designs can be:
- between-subjects variables (i.e., participants experience only one level of the
independent variable),
- within-subjects variables (i.e., participants experience all levels of the independent
variable),
- or a combination of between and within-subjects variables (producing a mixed design).
Factorial design with two between-subjects independent variables
• In a design layout (i.e., a picture showing logical structure) of a factorial design in which
one of the independent variables has three levels (variable A) and the other has two levels
(variable B).
• The levels of variable A are A1, A2, and A3, and the levels of variable B are B1 and B2.
There are six combinations of these two independent variables—A1B1, A1B2, A2B1, A2B2,
A3B1, and A3B2.
• Each of these combinations is referred to as a cell in the design layout and represents an
experimental condition.
• The number of cells in a design layout is obtained by multiplying the number of levels of
the independent variables—in this case there are six cells (3 × 2 = 6).
• The participants would be randomly assigned to the six cells and would receive the
appropriate treatment combination when the experiment is conducted. The
participants randomly assigned to A1B1 receive the A1 level of the first independent
variable and the B1 level of the second independent variable.

• Once the experiment is conducted, the researcher obtains the two types of means :
- A cell mean is the mean score of the participants in a cell.
- A marginal mean is the mean score of all participants receiving one level of an
independent variable (ignoring or averaging across the levels of the other independent
variable)
Factorial design with two independent variables
• The factorial design allows the investigation of two types of effects:
- A main effect refers to the separate influence of each independent variable on the
dependent variable.
- An interaction effect is the joint or “interactive” effect of the independent
variables. A two-way interaction effect occurs when the effect of one independent
variable on the dependent variable varies at the different levels of the other
independent variable.

For example, perhaps the effect of caffeine consumption on the dependent variable
test anxiety varies according to the amount of sleep someone has had. When you
have two independent variables in your factorial design, you analyze the data for
two main effects (one for each independent variable—such as caffeine consumption
and amount of sleep) and one interaction effect (for the “interaction” of the two
independent variables).
• For example, let’s say that we are interested in factors that affect driving
performance. Our first independent variable (variable A) is caffeine consumption with
the levels of low (A1), medium (A2), and high (A3). The second independent variable
(variable B) is sleep deprivation with the levels of not deprived (B1) and deprived (B2).
Students are randomly assigned to each combination of these two independent
variables (i.e., to each cell). The dependent variable is driving performance
(operationalized as the number of correct maneuvers on the training course).

• To determine if there are any main effects, you compare the marginal means for each
independent variable. The marginal means for caffeine consumption are 2.2, 7.3, and
5.3, which suggests that (ignoring the sleep deprivation variable) driving performance
is best with a medium level of caffeine consumption. The marginal means for sleep
deprivation are 5.4 and 4.4, which suggest that (ignoring caffeine consumption) sleep
deprivation leads to slightly lower driving performance. According to these two sets
of marginal means, it looks like there is a main effect for caffeine consumption and a
main effect for sleep deprivation.
• To determine if an interaction effect is present, construct a line plot of the cell means and
visually inspect the results.
• Specifically, to determine if an interaction is present in the line graph, use these two rules:
- No interaction rule: If the lines are parallel, there is no interaction; interpret any main effects
that are present.
- Interaction rule: If the lines are not parallel, there is an interaction; interpret the interaction
effect, and do not interpret main effects.
Line graph of cell means.
• The interaction effect suggests that the relationship between caffeine
consumption and driving performance changes at the different levels of sleep
deprivation.
• For participants who are sleep deprived, the best driving performance is obtained
under high caffeine consumption, and low consumption has the lowest level of
driving performance.
• For participants who are not sleep deprived, the best driving performance is
obtained under medium caffeine consumption, and it appears that low and high
consumption have about the same low level of driving performance.
Factorial Designs Based on within-subjects independent variables
• Type of independent variable where all participants receive all levels of the
independent variable.
• For example, if one of your IVs has two levels and the other has three levels, then
you have a 2 × 3 design with six cells (i.e., 2 × 3 = 6). If you have two independent
variables each with two levels then you would have four cells (i.e., 2 × 2 = 4).
• An advantage of within-subjects independent variables is that you need fewer
participants because the same people are in all of the cells.
• When you have within-subjects IVs, experimenters usually counterbalance the
order in which the participants receive the treatment combinations because the
order of presentation of treatment conditions can have an effect on the outcome.
• For example, in a 2 × 3 design with randomized counterbalancing, participants
would receive 6 treatment conditions in random order. Using a table of random
numbers or using a random number generator, you might find that person one is
to receive order 6, 1, 3, 5, 4, 2; person two is to receive order 2, 1, 3, 4, 6, 5, person
three is to receive order 1, 2, 3, 5, 4, 6, and so forth.

• As an example of a 2 × 3 design, you might want to know if different reward

amounts (large versus small) are equally effective for easy, moderate, and difficult
tasks. Reward is a within-subjects IV because all participants receive both levels of
rewards (at different times). Likewise, task difficulty is a within-subjects IV
because all participants perform easy, moderate, and difficult tasks (at different
times). Because one of these independent variables has two levels and the other
IV has three levels, the number of treatment combinations is 6 (i.e., 2 × 3 = 6). The
order in which the participants receive these 6 conditions will be different (i.e.,
counterbalanced) to eliminate order and carryover effects.
Factorial Designs Based on a Mixed Model
• A factorial design that uses a combination of within-participants and between
participants independent variable.
• In this design, participants are randomly assigned to the different levels of
variation of the between-subjects IV, but all participants take each level of
variation of the within-subjects IV.

• For example, you might want to know if different types of motivational

instructions are equally effective for easy, moderate, and difficult tasks.
Motivational instruction is the between-subjects independent variable because
you assign participants to the three motivational instructions conditions, forming
three independent groups. The within-subjects independent variable is task
difficulty, and each group will perform the easy, moderate, and difficult tasks.
Because both of these independent variables (type of instructions and difficulty of
task) have three levels, the number of treatment combinations is 9 (i.e., 3 × 3 = 9).
• Regardless of the type of IV (between, within, or a combination), if you have two
IVs you can test for the effects produced by each of the two independent variables,
as well as for the interaction between the two independent variables.
• In comparison to the design where there were two between-subject IVs, the mixed
design has the advantage of needing fewer participants because all participants
take all levels of variation of one of the independent variables.
• Therefore, the number of participants required is only some multiple of the
number of levels of the between-subjects independent variable.
Strengths and Weaknesses of Factorial Designs

 Strengths

• Factorial designs enable us to include as many independent variables as we

consider important.
• Researcher can control a potentially confounding variable by building it into the
design as an independent variable. For example, if you are worried that an effect
might be different for men and women, you can add gender to your design.
• Enables the researcher to study the interactive effects of the independent
variables on the dependent variable.
 Weakness

• However, there are several difficulties associated with increasing the number of
independent variables.
- First, there is an associated increase in the number of research participants
required.
- E.g. In a three-variable design, with two levels of variation per independent
variable, a 2 × 2 × 2 arrangement exists, yielding eight cells, and 120 participants
are required in order to have 15 participants per cell.
- Second, increased difficulty of simultaneously manipulating the combinations of
independent variables.
- In an attitude study, it is harder to simultaneously manipulate the credibility of the
communicator, type of ­message, gender of the communicator, prior attitudes of
the audience, and intelligence of the audience (a five independent variable
problem) than it is just to manipulate the credibility of the communicator and prior
attitudes of the audience.

- A third complication arises when higher-order interaction effects are significant.

- In a design with three independent variables, it is possible to have a three-way
interaction: A two-way interaction that changes at the different levels of the third
independent variable.
 How To Choose or Construct the
Appropriate Experimental Design
• There are several factors to consider in making the design decision, including:
- the nature of the research problem,
- the specific research question,
- the extraneous variables that must be controlled,
- the relative advantages and disadvantages inherent in alternative designs.

• Experimental research is appropriate for research questions concerning cause and

effect, and the randomized or strong designs are the best experimental designs
available.
• Here are some considerations that are under your control when constructing an
experimental research design:
(1) Should I use a control group
(2) Should I use multiple treatment comparison groups (comparing more than one active
treatment)?
(3) Should I use a pretest?
(4) Should I use just one or multiple pretests (to get a stable baseline)?
(5) Should I use just one or multiple posttests (to get a stable treatment effect or identify
delayed outcomes)?
(6) Should I use a within-subjects or a between-subjects independent variable, or should I
use both?
(7) Should I include multiple theoretically interesting independent variables in the design
(as in factorial designs)?
(8) Should I include more than one dependent variable (to see how the treatment affects
several different outcomes)?