Control Problems in

Experimental Research
Types of Design

• Between-Subjects Designs (Independent

Measures)
Any experimental design in which different
groups of subjects serve in the different
conditions of the study.

• Within-Subjects Designs (Repeated Measures)

Any experimental design in which the same
subjects serve in each of the different conditions
of the study.
Between-Subjects Design

• Almost all subject variable based IVs must be

studied using between-subjects design.
• Examples: Comparing two groups of extroverts
and introverts. Comparing role of gender.
• The only exemption is when you are studying a
subject variable over time in a longitudinal
study, then you may study the same subject.
• Secondly, certain manipulated based IV studies
also utilize between-subject design.
• This is done when exposure to one condition of
the experiment may affect the DV.
Between-Subjects Design

• Advantages

– Each subject enters the study fresh, and

naive with respect to the procedures to be
tested.
Between-Subjects Design

• Disadvantages

– Large numbers of subjects may need to be

recruited, tested and debriefed.

– Differences between the conditions could

be due to the independent variables, but
they might also be due to differences
between the two groups of people.
• To deal with this confound equivalent
groups are created
The Problem of Creating Equivalent
Groups in Between-Subjects Design

• Two Common Techniques used are:

– Random Assignment
– Matching
RANDOM ASSIGNMENT

• It is not similar to random selection.

• An example of a simple random
sample would be the names of 25 employees
being chosen out of a hat from a company of
250 employees
• It’s a method of randomly placing subjects,
once selected, into different groups.
• In it every subject for the study has equal
chance of being placed in each of the groups
being formed.
RANDOM ASSIGNMENT

• Random assignment has the affect of spreading

potentially confounding factors equally throughout
different groups.
• It works well when there are large number of subjects
assigned to each group.
• The greater the number of subjects; greater the
chances of making equivalent groups.
• However, through random assignment chances are
your groups will have different number of people in it.
• Flipping a coin with 20 subjects; what if the coin comes
tail all the 20 times.
RANDOM ASSIGNMENT

• To tackle the issue of assigning equal number

of participants in each of the conditions, we
use
– Block randomization or Blocked random
assignment
Block Randomization

• A procedure ensuring that each condition

of the study has a subject randomly
assigned to it before any condition is
repeated a second time. Each block
contains all of the conditions of the study
in randomized order.
• Example: A study comparing the
effectiveness of 4 different presentation
rates on memory for a word list.
Block Randomization

• STEP 1:
• Decide how many subjects you want to
test.
• Equal number of subjects per condition
• The number of subjects must be a
multiple of total number of conditions.
• E.g., 4 conditions 80 subjects and 20
blocks
Block Randomization

• STEP 2:
• Designate the conditions by the numbers
1, 2, 3, and 4.
• Each block will be designated some
random sequence of those four numbers.
Block Randomization

• STEP 3:
• Go to table of random numbers and work your way
down column across rows, for numbers 1-4.
• Select each number once before selecting another
number.
• Create blocks 1-3-4-2, 1,3,2,4

2 2 1 7 6 8 6 5 8 4 6 8 9 5

1 9 3 6 1 7 5 9 4 6 1 3 7 9

1 6 7 7 2 3 0 2 7 7 0 9 6 1

7 8 0 3 7 6 7 1 6 1 2 0 4 4

0 3 2 8 1 2 2 6 0 8 7 3 3 7
Block Randomization
• STEP 4
• Create a master sheet with 20 blocks and
assign different participants to each
condition within each block.

Block 1 1-3-4-2
Block 2 1-3-2-4

For further study :

https://opentextbc.ca/researchmethods/ch
apter/experimental-design/
However, if random assignment fails to
create equivalent groups.
i.e., small number of participants are
available, we must use another method:
– Matching.

Because Non-equivalent groups can

lead to type II error.
Matching

• Participants are paired together on some

subject variable (trait or characteristic like
anxiety, GPA, age, etc.,)
• One member of each pair is randomly
assigned to one of the conditions.
Matching

• Matching is required to be used when the

following conditions are met:

– The number of (N) subjects is small so

random assignment is risky and might yield
nonequivalent groups.
– Matching variable is correlated with DV (it is
likely to confound and must be controlled)
– There is some reasonable way to create
matching pairs.
Matching

Matching can be difficult when multiple subject

variable underlying factors can affect the
outcomes of experiments, (e.g., education level,
intelligence level, age, mood, etc.) so the
researchers prefer to make groups large enough
to allow for random assignment and expect that
RA will distribute the possible confounds evenly
in the different conditions of the study.

18
Matching

Experiment example:

•Comparing the problem-solving skills of

two groups through their performance in a
between group design.
•Here, academic skills of the subjects
might confound the results.
Matching
Step 1
•For all participants, get the scores of the matching
variable (academic skill level measured by current GPA)
•Create matching groups with same academic skill level,
i.e., GPA
•Example GPAs for ten participants:
S1: 3.24 S6: 2.45

S2: 3.91 S7: 3.85

S3: 2.71 S8: 3.12

S4: 2.05 S9: 2.91

S5: 2.62 S10: 2.21

Matching

Step 2
•Arrange matching variable scores (academic skill
level as per GPA) in ascending order
S4: 2.05 S9: 2.91

S10: 2.21 S8: 3.12

S6: 2.45 S1: 3.24

S5: 2.62 S7: 3.85

S3: 2.71 S2: 3.91

Matching

Step 3
•Create pairs of the subjects based on their scores
for the matching variable, with each pair
consisting of adjacent GPAs

PAIR 1 2.05 AND 2.21

PAIR 2 2.45 AND 2.62
PAIR 3 2.71 AND 2.91
PAIR 4 3.12 AND 3.24
PAIR 5 3.85 AND 3.91
Matching

Step 4
•For each pair randomly assign one subject to group 1
and other to group 2
•Now the study will proceed with the assurance that the
mean of GPAs for both groups will be equal hence both
the groups are equal.

GROUP 1 GROUP 2
2.05 2.21
2.62 2.45
2.91 2.71
3.12 3.24
Why we use Within-Subjects Designs

• Conduct an experiment when few participants

are available.
• Conduct the experiment more efficiently.
• Increase the sensitivity of the experiment
• Study changes in participants behaviour over
time.
• For example: change in mood due to
environment or tasks assigned, change in
preferences, etc.
Within-Subjects Design

• Each participants is exposed to each level of

independent variable.
• Every subject in this design is measured several
times.
• Also labeled as “repeated measure design”
• Fewer subjects are needed to be included in study
• Only reasonable choice in field of physiological
psychology i.e., sensation, perception.
• It helps to make comparisons between conditions
that are required for just a brief amount of time but
an extensive preparation (e.g., surgery)
Within-Subjects Design
Example
•A perceptual study using Muller-Lyer Illusion
•We want to check that the illusion is strongest at the vertical
orientation.
•4 conditions are given (a) Horizontal (b) 45 degree left (c) 45
degree right (d) Vertical
•Subjects are shown the 4 conditions and asked to adjust the
length for one of the lines until they think both the lines are of
the equal length.
•The test takes hardly 5 seconds for all the 4 conditions, so it
is preferable that we repeat the 4 different orientations with
the same participants and develop a comparison.
Within-Subject Design

• Advantages

– Fewer subjects need to be recruited.

– Equivalent group problem from between

subjects does not occur here.
Within-Subject Design
• Imagine a between-subjects design example:
• Comparing two golf balls for distance they cover when hit, you
recruit 8 pro golfers and assign them randomly to 2 groups.
– Variability within group is visible as difference in S.D. values
– Variability between the groups is visible as difference in mean
– Pros in 2nd group are hitting their ball farther than 1st group Pros.
– S.D. Group 1 (6.44) Group 2 (6.52)
– Mean Group 1 (253) Group 2 (265)
Within-Subject Design
• Reasons for the result:
1. Chance: perhaps this is not a statistically significant difference and
maybe it’s a type 1 error
2. May be the brand of ball is the reason that ball in group 1 goes farther
than the ball given to group 2 (research hypothesis)
3. Some confound with the second group (wind changed for 2 nd group)
4. Individual difference (2nd group golfers are stronger and more skilled).

• The 4th issue can be resolved by creating equivalent groups

• But it would be easier to place the participants in a within-subject
design and avoid all of the above issues.
Only one group Golf ball 1 Golf ball 2

Subject Pro1 255 269

Subject Pro2 261 266
Subject Pro3 248 260
Subject Pro4 250 270
By making the experiment into a within-subject design, have
we eliminated all the chances of error by removing individual
differences?
Can we now say for sure that the superiority of golf
ball 2 is really due to the brand of the ball and not
some other factor?

No.
Within-Subject Design
Other possible reasons behind the change in results:
•Maybe there was a practice or warm up affect?
•Maybe the pro golfer detected a defect in the first trial and
improved his skill in second trial?

This brings us to the problem of Sequence/Order effects: When a

subject has completed part 1 of study, subject’s experience may
influence his performance on part 2, 3, 4… of the same study.
•Progressive Effect: Fatigue/boredom from trial to trial steadily
reduces performance in later trials. Practice from trial to trial
steadily improves performance in later trials. The effect is
progressive (steady) in nature, either positive or negative.
•Carry-over Effect: Sometimes a specific sequence of trials has
an effect on the subject’s performance, and the effect is different
for different sequences.
Within-Subjects Designs

Disadvantage of within-subject design

– Sequence/Order Effect
Can occur when the experience of participating in
one of the conditions/trials of the experiment
influences performance in subsequent following
conditions.
• Progressive Effect
– Fatigue effect
– Practice effect
• Carry-over Effect
Within-Subjects Designs

• Progressive Effect
Form of sequence effect in which systematic
changes in performance occur as a result of
completing the sequence of conditions (e.g.,
warm-up/practice effects, fatigue/boredom effect).

• Carry-over Effect
Form of sequence effect in which one particular
order of conditions could produce one type of
effect, while a different order could produce a
different type of effect.
Controlling Sequence/Order Effect

The typical way to control sequence effect is to use

more than one sequence. This strategy is known as
COUNTERBALANCING

•The kind of counterbalancing used depends on the

nature of study, especially whether the study includes:
– Testing once per condition or
– Testing more than once per condition
Controlling Sequence/Order Effect

Testing Once Per Condition

In some experiments subjects will be tested in each condition,
but only tested once per condition.

EXAMPLE study by Reynolds in 1992

Research Question: ability of chess players to recognize the
level of expertise in other players.
15 chess players (participants) have to look at different chess
games btw two other players and rate them on their skill level
Each participant will look at all the 6 chess games

Reynolds was faced with a question “how to avoid the

sequence affect that might be present”
Complete Counterbalancing
– Complete Counterbalancing
Every possible sequence will be used exactly once.
Total number of sequence can be determined by
calculating X!
X = number of conditions
! = mathematical equation of a factorial
EXAMPLE: if a study has 3 conditions, then
3! = 3 x 2 x 1 = 6
6 sequences in a study with 3 conditions A,B,C will be
ABC, ACB, BCA, BAC, CAB, CBA

ISSUE: as the number of conditions increase the total

number of possible sequences increases
Problem with Complete Counterbalancing

• Reynold had 6 conditions in his study

• Using formula 6! = 6*5*4*3*2*1 = 720 sequences
• Meaning 720 participants are required so that
one participant can be assigned to each
possible sequence.
• Clearly a different strategy is needed
– Partial counterbalancing
– Balanced Latin Square
Partial Counterbalancing

• Whenever a subset of the total number of

possible sequences is used, the result is
called partial counterbalancing.

• Reynold chose a random sample, let’s

say 15 sequences out of the 720 possible
sequences.
Balanced Latin Square

• Its name come from an ancient roman puzzle about

arranging letters in a matrix, so that each letter appears
only once in each row and column.

• In a balanced Latin square each condition of the study

occurs equally often in each sequential position and
each condition precedes and follows each other
condition only once.

• When using Latin square, it is necessary for the

numbers of subjects to be equal to or be a multiple of
the number of rows in the square.
Balanced Latin Square STEP 1

• Build the first row with this formula

A B “X” C “X-1” D, “X-2” E, “X-3” F and so on
• A refers to the first condition of the study
• X refers to the letter symbolizing the final condition of
the experiment
• To build a 6x6 square, i.e., for an experiment with 6
conditions, this first row would substitute
• X = 6th letter of the alphabet = letter F
• X-1 = E
• So, the first row would be
A B F “X” C E “X-1” D
Balanced Latin Square STEP 2

• Build the second row – Directly below each letter of

row 1, place in row 2 the letter that is next in the
alphabet. Only exception is under F (the last letter),
where you will return to the first of the 6 letters (i.e., A)
and write that instead.

Row 1: A B F C E D
Row 2: B C A D F E
Balanced Latin Square STEP 3

• Build the remaining four rows.

A B F C E D
B C A D F E
C D B E A F
D E C F B A
E F D A C B
F A E B D C
Balanced Latin Square STEP 4

• Take each of the six conditions of the study

and randomly assign them to each of the six
letters.
• Then randomly assign each participant to one
row (i.e., one sequence of conditions).
• It is necessary for the numbers of subjects to
be equal to or be a multiple of the number of
rows in the square. (i.e., total sample should
either be 6, 12, 18, 24, or so on).
Controlling Sequence/Order Effects

Testing More Than Once Per Condition

–In Reynolds study it made no sense to ask all the 15

subjects to go through the 6 conditions again once they
have done that already.
–Similarly in a memory test if subjects are asked to recall
word lists made from a Latin square sequence of 4x4, it
is no use to ask them to repeat the sequence.

–However, in certain scenarios you have to make the

subjects experience the conditions more than once.
Example
• In the Muller-Lyer Illusion study you want to see at
which degree of presentation will the subjects perceive
the illusion to be the strongest.
• 4 conditions are given
• A = HORIZONTAL
• B = 45o TO THE LEFT
• C = 45o TO THE RIGHT
• D = VERTICAL
• To avoid sequence affect we will give the same
situations again to the subjects in one of the two
manners:
• Reverse counterbalancing
• Block randomization
Reverse Counterbalancing

• Occurs when subjects are tested more than once per

condition
• Subjects experience one sequence, then a second with
the order reversed from the first.

• In the given example the reverse sequence will be

ABCD then DCBA
Block Randomization

• Also used in a counterbalancing procedure to ensure

that when subjects are tested in each condition more
than once, they experience each condition once before
experiencing it again.
• Example: rather than giving sequence in a reverse
manner i.e., ABCD DCBA
• We will present the conditions within a sequence
through a random order made via block randomization
i.e., BCAD CADB
Problems With Counterbalancing

• Counterbalancing procedures help reduce

sequence effect, but they are based on the
assumption that sequential effects are linear
• This doesn’t always happen (carryover effect)
• Example – Maze learning experiment
Example: (Maze Learning)

Maze B – difficult Maze A – easy

Example: (Maze Learning)
• In a within-subject study, half of the subjects learn maze A (easy)
followed by maze B (difficult) and the other half learn maze B and
then maze A.
• To remove sequence effect, counterbalancing will be used.
• The participants will first get the sequence AB, then BA
• After half an hour boredom will occur, and fatigue effect can be
assumed to be linear (steady)

Errors due to
Difficulty Boredom Total score
Maze A (Then) 10 0 10
Maze B 15 +3 18
Maze B (Then) 15 0 15
Maze A 10 +3 13
Example: (Maze Learning)
• In this condition the overall effect can wash out the
singular affect discrepancy, but in some cases this
doesn’t happen.
• Example: Solving Maze A gave people insight about
mazes in general and helps in solving B
• So, the sequence A:B will transfer much more learning
than sequence B:A.
• In this case we can say that Asymmetric Transfer
occurred.
• Asymmetric Transfer
Occurs when one sequence produces a transfer effect
that is different from that produced by another
counterbalanced sequence (a kind of carryover effect)
Control Problems in Developmental
Research
Developmental studies involve age as the prime factor IV.
Both within groups design or between groups design fall into
these two study designs: Cross-sectional & Longitudinal

•Cross-sectional Study – analysis/prevalence study is a

type of observational study that analyzes data from a
population, or a representative subset, at a specific point in
time—i.e., cross-sectional data
•Used with between group designs
•In developmental psychology a design in which age is the
independent variable and different groups of people are
tested – each group is of a different age.
•E.g., a study comparing language performance of 3, 4, and,
5 year old children simultaneously.
Cross-Sectional Study
Advantage
– Takes lesser time (as compared to longitudinal)

Disadvantage
– Problem of non-equivalents groups – individual
differences might occur between different subjects,
which might affect the DV.
– Special kind of nonequivalent group problem in
cross sectional studies is: Cohort Effects
– A cohort is the group of people born at the same
time. Cohort effects can reduce the internal
validity of cross-sectional studies because
differences between groups could result from the
effects of growing up in different historical eras.
Longitudinal Study
Longitudinal Study – In developmental psychology, a
design in which age is the independent variable and the
same group of people is tested repeatedly at different
ages.
•E.g., Lewis Terman, the longest repeated measure study
“comparing gifted children with normal population” over a
period of more than 20 years. It had the lowest attrition
rate even after 35 years = 93% still participated
•Advantage: no problem of nonequivalent groups
•Disadvantage
– Attrition: Occurs when subjects fail to complete a
study, usually in longitudinal studies and they drop
out. So, subjects finishing the study may be
nonequivalent to those who started it.
Problem With Biasing

• Bias
A preconceived expectation about what
is to happen in an experiment.
• Two types:
– Experimenter bias
– Subject bias
Experimenter Bias
• Occurs when an experimenter’s expectations about a study
affect its outcome. Experimenter testing a hypothesis might
inadvertently do something that leads subject to behave in a
way to confirm the hypothesis.
• A biased experimenter might treat the research participants
in various conditions differently
• Rosenthal study: subjects were shown pictures and were
asked to rate them (-10 to +10). IV was experimenter’s
expectancy. Experimenter A was told that subjects would rate
the pictures positively, Experimenter B was told they would
rate negatively. Results showed clear difference between 2
groups.
• This affect is also called Experimenter expectancy effect.
• Apart from this, experimenter’s attitude, race, gender, etc.,
can also affect the participants’ performance.
Controlling for Experimenter Bias

• Automation
Removing the human factor in perception studies and
keeping the procedure totally machinery operated
reduces the experimenter expectancy affect.

• Double Blind Procedure

A control procedure designed to reduce bias;
neither the subject nor the person conducting the
experiment session knows which condition of the
study is being tested.
Subject Bias

• When subjects are expecting their specific role in the

study.

• When subjects’ behavior is affected by the knowledge

that they are in an experiment, and they are important
to the study’s success. This phenomenon is called the
Hawthorne Effect.

• It’s the tendency for performance to be affected

because people know they are being studies.
Subject Bias: Demand Characteristics

• Subjects, usually in order to help experimenter, take the role

of good subjects.

• Orne (1962) concluded that usually subjects tend to be

cooperative through out repetitive and boring experiments for
the sake of study, but when the subject finds out the
hypothesis and acts in a certain way to prove or disapprove
it, it is called as Demand characteristic.

• Demand Characteristics
Any feature of the experimental design or procedure that
increases the chances that subjects will detect the true
purpose of the study.
E.g., Presence of panic button in a study that measures level
of stress if participant is put in a sensory deprivation room.
Subject Bias: Evaluation apprehension

• Rosenthal used the term first.

• Evaluation apprehension refers to the

subjects wish to be evaluated positively by the
experimenter, i.e., to behave in a way that is
more socially desirable, to appear intelligent,
competent, emotionally stable, etc.
Controlling for Subject Bias
• To reduce subject bias due to demand characteristics,
process of deception is used. Deception is when the subject
is given either false or incomplete information regarding the
nature of the study.
• Placebo Control Group. It provides comparison between
groups getting treatment and those who think they are getting
treatment but are placebos. If both groups behave the same,
the results can be attributed to subject bias instead of the IV.
• Manipulation Check. Achieved during debriefing by asking
subjects in deception study what they think was the purpose
of the study. Also, a subject can be checked between the
testing procedure and asked for clarity of instructions and
what they think is going on in the experiment, etc.
• Field Research. Unaware subjects are perfect subjects!
No demand characteristics and no evaluation apprehension.