Journal of Abnormal Psychology Copyright 2001 by the American Psychological Association, Inc.

2001, Vol. 110, No. 1,40-48 0021-843X/01/S5.00 DOI: 10.1037//0021-843X.110.1.40

Misunderstanding Analysis of Covariance

Gregory A. Miller Jean P. Chapman
University of Illinois, Champaign University of Wisconsin—Madison

Despite numerous technical treatments in many venues, analysis of covariance (ANCOVA) remains a
widely misused approach to dealing with substantive group differences on potential covariates, partic-
ularly in psychopathology research. Published articles reach unfounded conclusions, and some statistics
texts neglect the issue. The problem with ANCOVA in such cases is reviewed. In many cases, there is
no means of achieving the superficially appealing goal of "correcting" or "controlling for" real group
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

differences on a potential covariate. In hopes of curtailing misuse of ANCOVA and promoting appro-
This document is copyrighted by the American Psychological Association or one of its allied publishers.

priate use, a nontechnical discussion is provided, emphasizing a substantive confound rarely articulated
in textbooks and other general presentations, to complement the mathematical critiques already available.
Some alternatives are discussed for contexts in which ANCOVA is inappropriate or questionable.

In research comparing groups of participants, classical experi- inherently, not 4th graders, and ANCOVA cannot "control for"
mental design (Campbell & Stanley, 1963) relies, whenever pos- that fact. Age is so intimately associated with grade in school that
sible, on random assignment of participants to groups. Observed removal of variance in basketball ability associated with age would
differences between such groups, prior to experimental treatments, remove considerable (perhaps nearly all) variance in basketball
are due to chance rather than being meaningfully related to the ability associated with grade. The results of the ANCOVA would
group variable. In contrast, when preexisting groups are studied, be meaningless. As a complement to this problem of the covariate
observed pretreatment differences may reflect some meaningful, removing too much of the independent variable of interest, a
substantive differences that are attributable to group membership. problem can arise when preexisting groups differ systematically on
It has been noted that, "Even in the absence of a true treatment more than the covariate. The covariate will leave those differences
effect, the outcome scores in the treatment groups are likely to intact, thus biasing the estimate of the treatment effect (Reichardt
differ substantially because of initial selection differences. As a & Bormann, 1994), which has been called specification error.
result, selection differences are a threat to validity. . . . " (Reichardt
& Bormann, 1994, p. 442).
In this article, we discuss why attempts to control statistically Nonrandom Group Assignment
for such differences are, in general, inappropriate. For example,
When group membership is determined nonrandomly, there is
consider a data set consisting of age as a potential covariate, grade
typically no thorough basis for determining whether a given pre-
in school as the grouping variable, and basketball performance as
treatment difference reflects random error or a true group differ-
the dependent variable. An analysis of covariance (ANCOVA)
ence. This uncertainty complicates interpretation of apparent treat-
might be run in hopes of asking whether 3rd and 4th graders would
ment effects because it is impossible to distinguish a main effect of
differ in performance were they not different in age. This might
treatment from an interaction between the effects of the treatment
seem to be a reasonable question, in that one could ask whether
and the pretreatment difference, and from meaningful overlap
some maturational change at that age makes a nonlinear contribu-
(variance shared) between the treatment and the pretreatment char-
tion to basketball ability. However, in fact it makes no sense to ask
acteristic. The confound due to an undetected interaction is widely
how 3rd graders would do if they were 4th graders. They are,
understood—the pooled regression is inappropriate as a basis for
"correcting" for the covariate. However, the other confound, due to
undetected or misinterpreted overlap of pretreatment difference
Gregory A. Miller, Departments of Psychology and Psychiatry and and grouping factor, is commonly neglected, although it was noted
Beckman Institute, University of Illinois, Champaign; Jean P. Chapman, in the first modern treatment of ANCOVA (Cochran, 1957) and
Department of Psychology, University of Wisconsin—Madison. occasionally more recently (e.g., Porter & Raudenbush, 1987).
The writing of this article was supported by National Institute of Mental The problem of preexisting group differences arises very com-
Health Grant MH39628. We thank Loren J. Chapman, Lawrence J. Hubert, monly in psychopathology research because random assignment to
Brandy G. Isaacks, Jack B. Nitschke, and E. Keolani Taitano for comments group (diagnostic category in the typical experimental design in
on an earlier draft.
psychopathology but called "treatment" in much of the statistical
Correspondence concerning this article should be addressed to Gregory
literature) is routinely infeasible and/or unethical. It is thus ex-
A. Miller, Department of Psychology, University of Illinois, 603 East
Daniel Street, Champaign, Illinois 61820, or to Jean P. Chapman, Depart- tremely tempting for psychopathologists to seek to use analytic
ment of Psychology, University of Wisconsin, 1202 West Johnson Street, methods in an attempt to avoid the interpretative problems that
Madison, Wisconsin 53706. Electronic mail may be sent to gamiller arise when groups differ pretreatment. It is unfortunate that, in the
@uiuc.edu or [email protected]. general case, no such analytic method is available, nor can one be

40
 SPECIAL SECTION: ANALYSIS OF COVARIANCE 41

developed (a point that will be explained below). This logical fact

has proven difficult for psychopathologists to accept, perhaps
because of the burden that preexisting group differences place on
interpretation of experimental results. Despite numerous technical
treatments in the literature (e.g., Chapman & Chapman, 1973;
Cochran, 1957; Elashoff, 1969; Fleiss & Tanur, 1973; Huitema,
1980; Jin, 1992; Lord, 1967, 1969; Maxwell & Delaney, 1990;
Maxwell, Delaney, & Manheimer, 1985; Porter & Raudenbush,
1987; Wainer, 1991; Wildt & Ahtola, 1978) and more accessible QUASI-EXPERIMENT
TRUE EXPERIMENT
statements (e.g., Neale & Oltmanns, 1980; Siddle & Turpin, 1980),
together making an overwhelming case against inappropriate at- Figure 1. Two (of several) possible relationships between the group
tempts to "control for" such group differences, they remain com- (Grp), covariate (Cov), and dependent (DV) variables. In both cases,
mon in the research literature and, if anything, even more common removing variance in DV that is associated with Cov will reduce the
in research grant applications.1 Given the continuing popularity of variance in DV that is not associated with Grp and thus will enhance the
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

inappropriate uses or interpretations of ANCOVA, the present relationship detected between Grp and DV. In the left panel, Grp and Cov
This document is copyrighted by the American Psychological Association or one of its allied publishers.

share no variance. This is the classic situation for a true experiment, with
article offers a relatively nontechnical critique, in hopes of helping
random assignment to groups. Removing the variance associated with Cov
to popularize the correct use of ANCOVA and helping researchers
will not alter Grp. Given random assignment, individual characteristics
to avoid its more common abuses. such as height or presence of hallucinations would generally be randomly
distributed across the groups, and group means should not differ except by
chance. In the right panel, Grp and Cov do share variance. This is often the
Understanding Analysis of Covariance case when preexisting groups are studied, such as comparisons of two
diagnostic groups—a quasi-experiment rather than a true experiment. In
ANCOVA is part of the ANOVA (analysis of variance) tradi-
such a case, removing the variance associated with Cov will also alter Grp
tion. ANCOVA was developed to improve the power of the test of in potentially problematic ways.
the independent variable, not to "control" for anything. It is helpful
here to place ANOVA and ANCOVA in the more general frame-
work of multiple regression and correlation (MRC), understood
within the general linear model. In fact, at least one popular However, the distinction between the classical and MRC-based
ANOVA package, BMDP2V (Dixon, 1992), actually computes its perspectives on ANCOVA is not important for the issues ad-
analyses using multiple regression rather than using the equations dressed in the present article, and the MRC approach has some
typically presented in textbooks on ANOVA. Some sources place expository advantages. For illustrative purposes, we assume a
ANCOVA in the context of MRC, with the covariate in ANCOVA simple design having one covariate, Cov, one grouping variable,
understood as a regression predictor entered before what are, in Grp, and one dependent variable, DV, discussed in the MRC
ANOVA, the main effects and interactions (e.g., Harris, Bisbee, & framework advocated by Cohen and Cohen (1983). Figure 1 shows
Evans, 1 97 1 ; for extended treatments of this viewpoint, see Cohen two of several possible relationships among the three variables,
& Cohen, 1983, and Judd, McClelland, & Smith, 1996). More
with overlap indicating shared variance and, equivalently, a non-
commonly, other sources have argued that ANCOVA is not strictly
zero correlation. In the left panel, Cov and Grp share no variance,
equivalent to regressing the dependent variable on the covariate,
reflecting random assignment to group. On the right, they are
then doing an ANOVA on the residuals (e.g., Elashoff, 1969;
correlated.
Maxwell & Delaney, 1990; Maxwell et al., 1985; Porter & Rau-
denbush, 1987). In this view, ANCOVA provides a test of the main Following the approach of Cohen and Cohen (1983), Cov is
effect of group by comparing the error sums of squares resulting entered first in the regression. This removes variance from Grp
from two models rather than a regression. The two models are: that Cov shares with Grp, if any (area 2 + area 5 in Figure 1),
leaving a residual portion of Grp with which it is not correlated,
GrpKS (area 3 and area 6). Entry of Cov also removes variance it
shares with DV from DV (area 4 + area 5), leaving a residual
portion of DV with which it is not correlated, DVres (area 6 + area
7). For either panel in Figure 1, removal of Cov leaves areas 3, 6,
Roughly, Yy is the dependent variable for the fth subject in the jth and 7. The regression then enters GrpKS into the model and
group, ;u is the grand mean, Oj is the treatment effect for the ^'th
computes its correlation with £>Vres. What the investigator may
group, /JYjj is the product of a population regression coefficient
view as the conventional F test of Grp is actually, instead, an
and the score on the covariate for the ith subject in the y'th group,
evaluation of the variance shared by GrpKS and DVKS: How large
and €y is an error term for the fth subject in the jth group. The Oj
area 6 is, compared with the sum of area 6 + area 7.
term obviously differentiates the equations, but the two |BXy terms
would not generally be identical, either. Further discussion of these
equations and of the F test of the main effect of group based on
them can be found in Maxwell et al. (1985) and elsewhere. The 1
Gregory Miller was Chair of the NIMH Clinical Psychopathology
point in this second view is that the actual main-effect test is not Review Committee when this article was written. Approximately one third
simply the direct outcome of a regression, as the MRC approach of the grant applications he reviewed proposed a questionable or clearly
implies. invalid use of ANCOVA.
 42 MILLER AND CHAPMAN

ANCOVA When Groups Do Not Differ on the Covariate Violation of this assumption is not as disabling as nonequiva-
lence of groups on the covariate. When faced with heterogeneity of
In the classical treatment of random assignment to groups, Cov regression slopes, the investigator is encouraged (e.g., Cohen &
should, in principle, share no variance with Grp, as a direct result Cohen, 1983) simply to frame the analysis as a hierarchical or
of the random assignment (Figure 1, left panel). That is, the simultaneous regression and in that context to include an interac-
expected value of Cov will be the same for every group, and, tion term consisting of the product of Cov and Grp (see Cohen &
except for random error, the group means for Cov will be identical. Cohen, 1983, for methods of representing categorical variables in
As a consequence, entry into the regression of Cov before Grp will such an approach). In effect, Cov is no longer viewed as a quali-
remove no variance from Grp. Thus, Grp = Grpres. (In practice, tatively distinct covariate with a purely methodological role in the
some nonzero correlation between Cov and Grp may be observed, analysis but as a meaningful, substantive part of the analysis. Such
but generally the correlation will be negligible. Viewpoints on interactions may be theoretically interesting. Given that the inves-
cases in which it is not negligible are discussed below.) When tigator's goal is to identify sources of variance in DV, the test of
Grp = Grpres, the only effect of ANCOVA is to remove variance such an interaction may be fruitful. Rogosa (1980) noted that the
(area 4) from DV which, from the standpoint of Grp, is simply typical all-or-none framing of the assumption of equality of re-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

noise (error). Thus, Grp will correlate more highly with DVKS than gression slopes is simplistic and often inappropriate. For example,
This document is copyrighted by the American Psychological Association or one of its allied publishers.

with DV, resulting (all things being equal) in a larger effect size with very large sample sizes, differences in slope might be "sig-
and a more powerful significance test. Specifically in the left panel nificant" but trivially small; with small sample sizes, functionally
of Figure 1, Grp overlaps with a higher proportion of (the smaller) important differences in slopes might not be "significant." He
DVres (area 6 + area 7) than with DV (area 4 + area 6 + area 7), discussed other ways to frame the issue and some alternative
so the correlation of Grp and DVKS is higher than that of Grp analytic strategies in the face of nonparallel regression slopes.
and DV.
Used this way, ANCOVA serves as a legitimate and appealing Invalid ANCOVA When Groups Differ on the Covariate:
noise-reduction technique for evaluating the relationship of Grp Consensus of the Technical Literature
and DV. Other than the loss of a degree of freedom associated with
The assumption of homogeneity of regression slopes is fairly
inclusion of Cov in the analytic model, ANCOVA would appear to
well known. In contrast, the importance of groups not differing on
be valuable in improving the power of the test of Grp (see Bock,
Cov is not widely recognized in the psychopathology literature.
1975, and Porter & Raudenbush, 1987, for technical treatments of
Mistakenly, investigators frequently turn to ANCOVA in hopes of
the gain in efficiency from an appropriate use of ANCOVA). For
"controlling for" group differences on the covariate. There is no
example, a study comparing simple phobic and social phobic
statistical means of accomplishing this "control" (Chapman &
patients' response to phobia-relevant material might use individ-
Chapman, 1973; Fleiss & Tanur, 1973; Lord, 1967).
ually developed videos that differ in duration, and duration itself In fact, "control" is altogether the wrong metaphor for under-
might affect peak anxiety rating. If the groups do not differ on standing what ANCOVA accomplishes. We have found that in-
mean video duration, duration might serve as a covariate, reducing vestigators are frequently surprised when this is pointed out. Some
variance in anxiety rating that is driven by video duration and assert that "controlling" or "removing" nontrivial group differ-
unrelated to diagnostic group. In general, this will improve the ences on the covariate is the primary use of ANCOVA. It is
observed relationship between Grp and DV. Given this benefit, important to establish that relevant literature roundly condemns
ANCOVA is much underutilized in the psychopathology literature. this view, before attempting to provide an accessible explanation
It is important to measure Cov as reliably as possible to maxi- for why this view is mistaken and before considering some alter-
mize its ability to capture noise variance in DV and to ensure that natives. Modern literature on ANCOVA began with Cochran
the adjusted DV, £>Vres, is not contaminated by noise associated (1957), who stated, "[I]t is important to verify that the treatments
with the measurement of Cov. Measurement error could distort the have had no effect on" the covariate and "a covariance adjustment
resulting mean effects and significance test because the adjustment . . . may remove most of the real treatment effect" (p. 264). Camp-
is necessarily for observed scores rather than true scores on the bell and Stanley (1963) spoke favorably of ANCOVA in general
covariate (Reichardt, 1979; Elashoff, 1969; Fleiss & Tanur, 1973; but cautioned, "The usual statistics [including ANCOVA, cited
Maxwell & Delaney, 1990; Richards, 1980; although see Huitema, earlier on the same page] are appropriate only where individual
1980; Overall & Woodward, 1977; Porter & Raudenbush, 1987; students have been assigned at random to treatments" (p. 23). In
Reichardt, 1979; and Reichardt & Bormann, 1994, for discussions contrast, ANCOVA "to compare naturally occurring groups
of how this problem might be dealt with). This issue of measure- which is contrary to the admonitions of many experts in experi-
ment error is particularly problematic in studies with a small mental design, can yield statistically significant results which are
sample size. entirely spurious" (Evans & Anastasio, 1968, p. 225). Elashoff
With this MRC perspective on ANCOVA in mind, the assump- (1969) explained:
tions in ANCOVA can be readily summarized (for more extended,
A basic postulate underlying the use of analysis of covariance to
technical treatments of the assumptions of ANCOVA, see
adjust treatment means for the effects of the covariate x is that the x
Elashoff, 1969; Fleiss & Tanur, 1973; Huitema, 1980; Maxwell et variable is statistically independent of the treatment effect. In other
al., 1985; Porter & Raudenbush, 1987; Wildt & Ahtola, 1978). words, this means that the distribution of covariate values is not
Widely noted is that ANCOVA assumes that groups do not differ affected by the treatments either through direct causation or through
in the regression of DV on Cov. This is often referred to as the correlation with another affected character (and the x variable does not
assumption of homogeneity of regression slopes. affect the treatment). . . . Therefore, i f . . . treatments are not manip-
 SPECIAL SECTION: ANALYSIS OF COVARIANCE 43

ulated as independent variables but are classifications of naturally that if the treatments differentially affect the covariate scores, then
occurring groups this assumption will not be valid... . Analysis of an ANCOVA .. . would in fact remove from the treatment sum of
covariance is inappropriate if the covariate is not independent of the squares part of the treatment effect you really want included"
treatment, (pp. 388-389) (Maxwell & Delaney, 1990, pp. 380, 382-383).
Chapman and Chapman (1973) stated that there is no statistical
method that can address the question of whether two groups that Invalid ANCOVA When Groups Differ on the Covariate:
differ on variable A would differ on variable B if they did not differ The Substantive Problem
on variable A and added, "The only legitimate use of analysis of
covariance is for reducing variability of scores in groups that vary Granting this highly consistent sentiment in the technical liter-
randomly. Its use is invalid for preexisting disparate groups that ature, what is it that makes ANCOVA unacceptable in the face of
differ on the variable to be covaried out" (p. 82). Cohen and Cohen group differences on Cov? Technical discussions of this issue,
(1975) stated the principle forcefully: known as Lord's Paradox, have long been available (e.g., Bock,
1975; Fleiss & Tanur, 1973; Holland & Rubin, 1983; Lord, 1967,
[O]ne does not answer such questions with [ANCOVA]. .. . What are 1969; Maris, 1998; Maxwell & Delaney, 1990). It may be useful,
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

clearly not warranted by the results of [ANCOVA] are subjunctive

This document is copyrighted by the American Psychological Association or one of its allied publishers.

in persuading researchers on this issue, to convey the issue in terms

formulations like: 'If black and white varieties [of corn that differ in
of theoretical substance instead of mathematical proof. The central
height] were of equal height, then they would have equal yields.'
problem is that often one does not know what Grplss represents
. . . [S]uch subjunctive questions can not be answered by [ANCOVA],
or indeed, by any method of analyzing data. (pp. 396-397, 398, 399; when Cov and Grp are related. "When the covariate . . . is affected
emphasis in original).2 by the treatment, the regression adjustment may remove part of the
treatment effect or produce a spurious treatment effect" (Elashoff,
Jin (1992) provided this portrayal of the issue: "[Investigators ask], 1969, p. 388). The grouping variable, its essence, has been altered
'What would the results be if the subjects were at the same basal in some substantive way that is frequently not specifiable in a
level?' However [given nonrandom group assignment], this is an conceptually meaningful way (see also Evans & Anastasio, 1968).
unrealistic question" (p. 182). Porter and Raudenbush (1987) pro- Thus, Grpres is not a good measure of the construct that Grp is
vided an extended technical treatment and concluded, "It is crucial intended to measure: "the adjustment made on the dependent
that the covariable be unaffected by the treatment" (p. 385); "In variable is biased because some effects attributable to the treat-
short, ANCOVA cannot be counted on to estimate the right effects ment are eliminated from the dependent variable" (Wildt & Ah-
or test the correct hypothesis when it is used to analyze nonran- tola, 1978, p. 15). In the right panel of Figure 1, this problem is
domized experiments" (p. 391); and "ANCOVA will. .. remove manifested in the fact that area 2 and area 5 are no longer part of
some or all of the treatment effect for the dependent variable" (p. the (residualized) grouping variable.
391). In their monograph on ANCOVA, Wildt and Ahtola (1978) This problem is often not acknowledged in statistical textbooks
stated unequivocally: on ANCOVA and appears to be largely unknown in the psycho-
pathology literature, but it is quite important. For example, the
[T]his assumption, that the covariate is independent of the treatment,
considerable diagnostic comorbidity of depression and anxiety and
is a basic tenet of the analysis of covariance model. When the
the assumption that they share symptoms, psychological processes,
covariate and the treatment are not independent, the regression ad-
justment may obscure part of the treatment effect or may produce and even some neural processes (e.g., Keller et al., 2000) renders
spurious treatment effects.. . . [Violation of this assumption may complex any attempt to separate their components. If we compare
seriously affect the interpretation of results, (p. 90) a sample of depressed patients with nonpatient controls and covary
out anxiety, which happens to be higher in the patients, it is not
Fleiss and Tanur (1973) took a similarly stark position: necessarily the case that the residual group difference is a clear,
clean representation of depression as it would exist without the
[N]o amount of statistical manipulation can tell one what might have
comorbid anxiety. What we should believe about that depends on
been had certain differences been non-existent.. .. The overwhelming
weight of logic is on the side of those who warn that neither the our model of the relationship between depression and anxiety. If
analysis of covariance nor any other statistical technique can undo they happen to co-occur because of nonspecific severity factors
systematic differences which were out of the investigator's control, (p. that themselves are not specifically related to depression, our
513, 517) ANCOVA might be effective in removing such variance, leaving
"pure" depression. If, however, we believe that the negative affect
Huitema (1980) agreed: that depression and anxiety share is central to the concept of
depression, then removing negative affect (by removing anxiety)
If a nonrandomized design other than the biased assignment design is
employed and the covariate is measured after treatments are admin-
will mean that the group variance that remains has very poor
istered [i.e., if the groups' covariate means could have been affected construct validity for depression.
by—correlated with—the grouping variable], the ANOVA on the Turning to a common example outside the psychopathology
covariate as well as the ANOVA and the ANCOVA on [the dependent literature, one on which many technical treatments rely, we offer a
variable] will be essentially uninterpretable because treatment effects modest, nonmathematical elaboration of Lord's Paradox. Lord
and pretreatment differences among the populations will be con- (1967) contrasted the approach used by two hypothetical statist!-
founded. (Huitema, 1980, p. 109)

Finally, "The basic desideratum is that the covariate and the 2

Most of this is repeated in Cohen and Cohen (1983, p. 425), with the
treatment be statistically independent," and "The basic concern is verbatim conclusion drawn.
 44 MILLER AND CHAPMAN

cians analyzing a hypothetical data set. The data are for boys and older and younger children on height, because growth is an inher-
girls at the beginning and end of an academic year. The boys, as a ent (not chance or noise) differentiation of the two groups.
group, start and end the year weighing more than the girls, and As noted above, it would be entirely reasonable, given such a
neither groups' average weight changes over time. An issue is data set, to explore the relationships among age, height, and weight
whether diet affected boys and girls differentially during the (Cohen & Cohen, 1983). The point is that statistical control, in the
school year. The statistician who favored an ANCOVA (incor- sense of cleanly removing the effect of Cov, is not what one would
rectly so, in the opinion of Lord and numerous subsequent authors be able to accomplish with ANCOVA. Cohen and Cohen (1983)
quoted above) used initial weight as a covariate and concluded: provided the following extreme example: "Consider the fact that
the difference in mean height between the mountains of the
If one selects on the basis of initial weight a subgroup of boys and a
Himalayan and Catskill ranges, adjusting for differences in atmo-
subgroup of girls having identical frequency distributions of initial
weight,... the subgroup of boys is going to gain substantially more
spheric pressure, is zero!" (p. 425), the point being that one has not
during the year than the subgroup of girls. (Lord, 1967, p. 305) in any sense "equated" the two mountain ranges by using atmo-
spheric pressure as a covariate.
Lord's pro-ANCOVA statistician concluded from this that there is
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

a meaningful differential effect of diet on boys and girls. However,

Invalid ANCOVA in Psychopathology Research
This document is copyrighted by the American Psychological Association or one of its allied publishers.

the two selected subgroups are not representative of the larger

groups of boys and girls. In effect, the two levels of the gender One can readily imagine additional examples in psyehopathol-
effect no longer represent the samples of boys and girls in the ogy. If likelihood of diagnosis of schizophrenia rises with age, it
original analysis. What this statistician observed is merely an would not be appropriate to try to "control for" age differences
example of the principle of regression toward the mean. By se- between two samples to evaluate the relationship between propor-
lecting subsets of boys and girls matched on weight, the statistician tion of each sample receiving a diagnosis and some dependent
will have selected boys weighing less than the boys' mean and variable such as current employment status. Age would be sys-
girls weighing more than the girls' mean. Regression toward the tematically related to the defining characteristic of the groups, so
mean, when the participants are reassessed, would be expected removing variance associated with age would, in effect, corrupt the
solely as a result of the lack of a perfect correlation between initial grouping variable itself.
and final weight. Thus, comparison of the subset of boys gaining In a real-world example, Deldin (1996) wanted to investigate
weight and the subset of girls losing weight is not evidence for whether a brain-wave measure known as P300 is reduced in
systematic differential effects of diet on boys and girls. depression. This question is complicated by the fact of consider-
Lord's (1967) example is compelling, in part because he con- able comorbidity between anxiety and depression, noted above.
structed a case in which it is known that the groups do not change Two groups of participants, diagnosed as depressed and nonde-
over time, so they cannot change differentially over time, as a pressed, completed a self-report anxiety measure. Anxiety score
function of diet or any other factor. We suggest that the more might be used as a covariate in an ANCOVA with diagnosis as the
general point in this example is that, because gender and weight independent variable and P300 as the dependent variable. The
are confounded (correlated) to begin with, there is no statistical hope in such an analysis would be "to control" anxiety and thus be
means to unconfound them in these examples—here, to study the able to observe the relationship between pure depression (not
potential differential effect of diet on the two groups. confounded with anxiety) and P300. However, this would make no
As another illustration, consider a data set in which two groups sense on clinical, substantive grounds (fortunately, Deldin did not
are older men and younger women, and gender is of interest as an conduct an ANCOVA). The literature does not generally view
independent variable, Grp. Using age as a covariate does indeed anxiety as an independent, confounding variable in depression but
remove age variance. The problem is that, because age and gender as intimately related to depression and perhaps, in part, indistin-
are correlated in this data set, removing variance associated with guishable from it (e.g., Heller, Etienne, & Miller, 1995; Watson et
Cov will also remove some (shared) variance due to Grp. Within al., 1995). Statistical methods cannot remove the "effect" of anx-
this data set, there is no way to determine what values of DV men iety from depression if conceptually they are overlapping
younger than those tested or women older than those tested would constructs.
have provided. Far from "controlling for" age, the ANCOVA will As a final example, consider the effects of using gender as a
systematically distort the gender variable. As in our presentation of covariate in a comparison of schizophrenic and depressed groups,
Lord's Paradox above, Grpres will not be a valid measure of the diagnoses in which it is well established that men and women,
construct of gender. Again, this is seen in the right panel of respectively, are overrepresented (Kessler et al., 1994; Walker &
Figure 1, when area 2 and area 5 are removed from Grp. Lewine, 1993). Removing gender variance could systematically
Consider a data set consisting of childrens' age, height, and alter the apparent nature of and relationships between the diagnos-
weight. If we conduct an ANCOVA in which height is the covari- tic groups. If, further, the dependent variable were performance on
ate, age is the grouping variable, and weight is the dependent a task on which women are believed to be superior (i.e., if Cov
variable, we are attempting to ask whether younger and older correlates with both Grp and DV), such an ANCOVA would
children would differ in weight if they did not happen to differ in corrupt both independent and dependent variables, providing a
height. If the groups indeed do not differ on the covariate, this meaningless F test.
question can be asked. But if there is something about the construct Misuse or misinterpretation of ANCOVA is so widespread in
of age in childhood that inherently involves differences in height, the psychopathology literature, in our experience, that it is difficult
the question makes no sense, because then age with height par- to cite examples without stepping on toes. The report by Rosvold,
tialed out would no longer be age. There is no way to "equate" Mirsky, Sarason, Bransome, and Beck (1956) may be used without
 SPECIAL SECTION: ANALYSIS OF COVARIANCE 45

causing offense, because it was published in a respected journal discussed narrow purposes or conditions under which ANCOVA
before the problem was recognized (Cochran, 1957) and because it might be interpretable despite nonrandorri assignment.
is a deservedly famous article for other reasons, having recently There can thus be a range of views on how to interpret certain
been republished as a landmark (Journal ofNIH Research, 1997). cases in which groups differ on Cov. Even in the case of random
Among other important contributions, this paper introduced the assignment to groups, nontrivial differences on Cov will occasion-
still widely used Continuous Performance Test (CPT) to the neu- ally arise by chance. In the right panel of Figure 1, the issue is
ropsychology and psychopathology literatures. The design in- whether area 2 and area 5 are substantively important parts of Grp.
cluded several groups of child and adult brain-damaged and con- If not, then after their removal GrpKS would still be a valid
trol samples, with X and AX denoting two types of trials. measure of the Group construct.
How is an investigator to know when a Type I error has
Since there was a significant age difference between the Child sub- occurred in a test of group differences on Cov? A moderate
groups and a significant IQ difference between the Adult sub- position might be that, if the investigator has good reason to
groups . . . , the differences between the paired subgroups means on X believe that group differences on Cov truly arose by chance,
and AX in these two groups were evaluated by means of an analysis ANCOVA is appropriate (Maxwell & Delaney, 1990). The ratio-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

of covariance. (Rosvold et al., 1956, p. 346)

nale for this view is that ANCOVA would only be removing noise
This document is copyrighted by the American Psychological Association or one of its allied publishers.

variance from Grp, not anything substantive about Grp. Of course,

Unfortunately, IQ would be very likely to be meaningfully related
this rationale turns on the strength of the assumption that the
to brain damage, so using IQ as a covariate would disrupt any
differences on Cov did, in fact, arise by chance. In other words, the
comparison of brain-damaged and control groups' performance:
investigator must be convinced (and must convince readers) that,
IQ differences would almost certainly be part of group differences
among the populations from which the groups are sampled, there
in brain-damage status. As a consequence, removing variance
is no relationship with Cov and that the available samples are
associated with IQ would alter the diagnostic group variable sub-
sometimes nevertheless a good representation of those popula-
stantively, and Grpres would not merely be a de-noised surrogate
tions. In the case of random assignment, the basis for this judgment
for Grp. Age as a covariate presents the same potential problem,
is quite strong, assuming good execution of the group assignment
but it is not as likely that age is substantively related to brain
process. In the case of nonrandom assignment—almost invariably
damage, so ANCOVA might be viable. (The legitimacy of
the case in studies of psychopathology—this is often a difficult
ANCOVA in the face of group differences on the covariate if the
judgment to defend. However, investigators should be free to
differences arose by chance is discussed below.)
consider it and to make their case.
This range of examples serves to convey the substantive (not
Overall and Woodward (1977) went a step beyond the arising-
merely mathematical) problem that arises when groups differ
randomly criterion and argued that what matters is whether Grp
meaningfully on the covariate. ANCOVA does indeed "remove"
could have caused the group differences on Cov. If not, in their
the variance due to the Cov, but it does not successfully "control"
view, then ANCOVA may be legitimate, even if the group differ-
for Cov if Cov is a meaningful part of Grp. On the contrary, and
ences on Cov are substantive. Similarly, Wildt and Ahtola (1978)
unfortunately for well-intentioned psychopathology researchers,
suggested that ANCOVA might be acceptable if the investigator is
ANCOVA removes meaningful variance from Grp, leaving an
certain that the Grp could not have affected Cov. We (see also
undercharacterized, vestigial GrpTl.s with an uncertain relationship
passages quoted above from Elashoff, 1969; Maxwell & Delaney,
to the construct that Grp represented. The relationship of such a
1990; and Porter & Raudenbush, 1987) do not find this argument
Grpres to DV or DVres is often not interpretable. In general,
compelling, in general. It will still happen that, because Cov will
ANCOVA is appropriate when the groups do not differ on the
be entered into the regression before Grp, Cov will in effect get
covariate—when inclusion of the covariate serves merely to re-
credit for any relationship of their shared variance that is also
move noise variance unrelated to the grouping variable (left panel
shared with DV. Thus, not only will GrpKS be a much-diminished
of Figure 1).
representation of the construct that Grp measures (a conceptual
problem) but also the relationship of Grpres and DVTe.s may be
Possibly Valid ANCOVA When Groups underestimated (a statistical problem). Cohen and Cohen (1983)
Differ on the Covariate offered another objection, in that it is often impossible to deter-
mine causal relationships between Grp and Cov. Overall and
Beyond this basic point, there are some gray areas regarding the Woodward offered a more specific analysis, however, which is
use and misuse of ANCOVA. Bock (1975) and Wainer (1991) sound: In the absence of random assignment, one might be able to
noted that the appropriateness of an ANCOVA depends not only render ANCOVA legitimate given an experimentally appropriate
on meeting statistical assumptions but also on the nature of the assignment to group on the basis of scores on the covariate. Their
question posed. Heckman (1989, p. 166) stated, "A decision about argument turns on particular patterns of relationships among Cov,
the appropriate statistical procedure requires information outside Grp, and DV. In effect, the point is that the legitimacy of
of statistics." Fleiss and Tanur (1973) concluded that it is not the ANCOVA depends on a careful examination of these relation-
analysis but, in part, "the phrasing of the hypotheses and of the ships. Unfortunately, this suggestion obviously would not apply in
inferences which are usually invalid" (p. 518) in the misuse of studies of preexisting groups typical in psychopathology.
ANCOVA, suggesting that if properly framed the analysis can be One other gray area to consider is the context of exploration of
appropriate (see Fleiss & Tanur, 1973, pp. 518-521, for further a data set, in contrast to the more commonly explicit goal of
discussion). Cochran (1957), Evans and Anastasio (1968), Max- hypothesis testing. As Huitema (1980, p. 108) outlined, it may be
well and Delaney (1990), and Porter and Raudenbush (1987) also useful to conduct a series of ANCOVAs, trying different variables
 46 MILLER AND CHAPMAN

as the covariate, as a means of understanding the patterns of shared strategies relative to ANCOVA (see also Reichardt & Bormann,
variance in a data set. He hastened to add that, when groups differ 1994). Cohen and Cohen (1983) and Harris et al. (1971) recom-
on the covariate, "it would be reasonable to speculate that the mended incorporating the covariate into the analysis—no longer
treatment effect on the dependent variable is mediated by the conceived as a covariate but as another substantive variable. Both
covariate. It couldn't be concluded. . . . " In effect, Huitema was of these are sound, relatively familiar strategies. Rosenbaum and
describing the strategy of Cohen and Cohen (1983) in using a Rubin (1984; Rosenbaum, 1995; Rubin, in press) discussed the
variety of hierarchical regression analyses to evaluate the relation- less widely known concept of propensity score, the conditional
ships among variables. Procedures such as those discussed by probability of assignment to a particular group or treatment given
Baron and Kenny (1986) might then be used to confirm media- a set of observed covariates. They noted that there are circum-
tional relationships. stances under which propensity score analysis can balance the
group on the covariates. It cannot address unobserved differences
Beyond Statistical Concerns in covariates, although there are methods of determining the extent
of bias arising from unobserved covariates (Rosenbaum, 1984).
We have emphasized that the problem with ANCOVA in the Less specific but potentially fruitful is the aggregation (perhaps via
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

face of group differences on the covariate is as much a substantive formal meta-analysis) of a large number of individually compro-
This document is copyrighted by the American Psychological Association or one of its allied publishers.

and interpretative issue as it is a mathematical issue. It can be mised studies, none of which may have random assignment but
noted that the problem of misuse of ANCOVA putatively to collectively present a strong inferential case (Rosenbaum, 1987).
"control for" differences on the covariate is not confined to aca- For example, no study of lung cancer has undertaken random
deme. An article in a prominent weekly political magazine stated: assignment to chronic smoking and nonsmoking groups, but a
strong case for causal factors in the face of potentially confounded
On the average, students in predominantly White districts did much
better on reading tests than those in predominantly Black districts. The covariates (such as hypertension, on which smokers and non-
reason? Not poverty, parent education, or class size. The White smokers might vary and on which smoking can have causal ef-
districts were, of course, generally better off, the parents of pupils fects) has been made on the basis of a variety of different studies.
were more highly educated, and so on. But when these differences Importantly for psychopathology research, Rosenbaum's sugges-
were held constant statistically, the difference that made a difference tion can be generalized beyond traditionally defined control
was the quality of the teaching staff, as measured by a language skills groups. For example, in a study comparing symptoms of psychosis
exam. (Thernstrom, 1991, p. 22; emphasis added) observed in schizophrenia and bipolar patients, there are advan-
tages to recruiting multiple samples, known to be different, within
It is apparent from this example that the public-policy stakes
each diagnosis.
associated with the misuse or misinterpretation of ANCOVA can
Placement of group means on the regression line. Fleiss and
be quite high. As we have argued on substantive grounds, com-
Tanur (1973) suggested another alternative to ANCOVA. They
plementing previous technical discussions, it is simply not possible
reasoned from the principle that a problem in using analysis of
for such differences to be "held constant statistically" as this article
covariance for the comparison of intact groups is that the regres-
claims was done, unless we were to suppose that teachers were
sion line within groups cannot legitimately be used for between-
assigned to White and Black school districts randomly or that
groups comparisons. They inferred from this that the investigator
differences in poverty, parental education, and class size existed
should consider the performance of many differing groups of
only by chance. Surely, teacher recruitment (and subsequent per-
participants, examining how different groups place on the regres-
formance) is in part driven by differences in district poverty, parent
sion line.
education, class size, etc. All of these variables are correlated, and
As a simple example to illustrate their method, an investigator
they must be understood as inherently confounded if one adopts
might wish to test the hypothesis that schizophrenic patients show
the superficially appealing but inappropriate goal of "controlling
a greater concreteness of proverb interpretation than is accounted
for" some of them. Ultimately, "quality of teaching staff is not a
for by their lower Verbal IQ score on the Wechsler Adult Intelli-
variable substantively left in the analysis by the time all of those gence Scale (WAIS). (Schizophrenic individuals' thinking has
covariates that are surely meaningfully correlated with it have been often been characterized as more concrete and less abstract than
partialed out. Only a highly residualized variable given the same nonpatients'. Judging that a hammer and a screwdriver are both
name is being tested. One could not conclude anything about the tools is more abstract than that they are both made of metal.) The
original variable, "quality of teaching staff." At best, one could investigator recognizes that she or he cannot simply match sub-
speculate, as Huitema (1980) suggested. Such nuances are often groups of schizophrenic and control participants on WAIS IQ and
lost in public debate and, in our experience, in the psychopathol- then compare the matched subgroups on concreteness because
ogy literature. such a comparison would be contaminated by regression toward
the mean (see discussion of Lord's Paradox above). Fleiss and
Some Alternatives to ANCOVA Tanur's (1973, p. 522) solution would be to extend the study of IQ
and concreteness to "many different kinds of subjects (normals,
Methods
neurotics, depressives, etc.), all having in common the fact that
Once again, analysis of covariance cannot tell us how groups they are not schizophrenic." Then, after obtaining mean scores on
would differ if they did not differ on the covariate. What then IQ and on WAIS Verbal IQ for each group, the investigator finds
should the investigator do instead of analysis of covariance? Max- a transformation of the scores such that a nearly linear relation can
well and Delaney (1990) discussed the analysis of gain scores and be fitted to all pairs of group means. In our example, a straight line
the blocking of subjects on the covariate, noting limitations in both would be fitted to the relation of mean group concreteness score to
 SPECIAL SECTION: ANALYSIS OF COVARIANCE 47

mean group IQ score. Then the means of the schizophrenic group A third problem is that the answer yielded by the study does not
are examined in relation to the fitted line. If the schizophrenic precisely match the original question. The investigators' hypoth-
patients score more deviantly on concreteness than predicted by esis was probably not concerned with a comparison of the con-
the multigroup regression line, one infers that their concreteness is creteness of schizophrenia with that of borderline retardation. In
unusual. our view, however, this kind of answer is often better than the
This solution is problematic. It is, of course, rather impractical available alternatives.
to test many groups in most such studies. More importantly, the
method may be flawed in some cases. If, as Fleiss and Tanur Substantive Questions
suggested, only transformation data rather than original data fit a
Future work may produce more general or more satisfactory
straight line, the meaning of that linearity is unclear. Some trans-
means to address the question psychopathologists usually attempt
formations might alter the data considerably, and the transforma-
to answer with ANCOVA, a question for which the technical
tion is selected to fit all of the pairs of group means to a straight
literature shows ANCOVA to be inappropriate: How would the
line except those of the schizophrenics. This procedure appears
groups differ on DV if they did not differ on Cov? However, we
vulnerable to random error that could foster the predicted result
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

have argued that fundamental logical problems with such a ques-

that the means of the schizophrenic group, but not the other
This document is copyrighted by the American Psychological Association or one of its allied publishers.

tion preclude its being meaningful.

groups, do not fit that line. It is quite possible that, if one included Rather than pursue such a question, we believe that psychopa-
the schizophrenic group among those for which a straight line is thologists would do well to frame questions that a rich experimen-
fitted, but excluded some other group, such as depressed patients, tal design can address. For example, the high comorbidity between
the means for that excluded group would fail to fall on the line, as depression and anxiety need not be seen as a barrier to research.
a procedural artifact. The comorbidity suggests that depression and anxiety are not
Regressing the dependent variable on the independent variable entirely distinct concepts and not phenomena that should be sep-
for control participants of a wide range of performance. M. B. arated or can be represented in a fully factorial design. Instead, one
Miller, Chapman, and Chapman (1993) suggested supplementing can articulate a specific relationship between them and then design
the "normal" control group with additional persons from a kind of an appropriate study. An appealing research design might include
group that is not grossly pathological but is known to be impaired depressed individuals varying in level of anxiety and/or anxious
on a variable that might be a confound in the results. They studied individuals varying in level of depression. A nonadditive model of
the prior preparatory interval (PPI) effect in schizophrenia (where- the characteristics and dynamics of comorbid depression and anx-
in reaction time depends on the length and predictability of the iety may best fit the resulting data.
intertrial interval) and wished to determine whether schizophrenic Similarly, variables that have often been viewed as confounds in
patients' greater overall slowness might be a confound. Accord- research on schizophrenia may be incorporated into models as
ingly, the investigators added elderly individuals to their normal significant conceptual players. For example, given the large liter-
control group because the elderly tend to demonstrate overall ature on cognitive deficits in schizophrenia, reduced IQ need not
slowness on reaction time. They found a very similar regression be viewed as something to "control for" but rather as a feature of
slope of the PPI effect score as a function of overall slowness for the disorder in many cases, to be studied as one studies more
elderly as for younger normal participants. Accordingly, they traditional clinical symptoms. The broader lesson here is that, if
computed a combined'regression line. Using the slope and inter- one's questions cannot be served by one's methods, one can
cept of that line, they computed residualized scores for the schizo- reconceptualize the questions to suit the best available methods.
phrenic patients' PPI effect. It is our hope that the present discussion provides an accessible
To apply this method to the example of concreteness of proverb portrayal of the proper use of ANCOVA. Relevant technical lit-
interpretation in schizophrenia, one might add borderline retarded erature overwhelmingly condemns its uncritical use in the face of
individuals to the control group, compute the regression of con- group differences on the covariate despite its continuing popularity
creteness score on IQ score for this expanded group of nonpatients, in such situations. Nojnrandom assignment to groups presents
and then use the regression line to compute residualized scores of psychopathologists with design, statistical, and interpretative chal-
concreteness for the schizophrenic patients. A finding of deviantly lenges, and we have provided some comments on attempts to
high residualized concreteness scores for the schizophrenic pa- address those challenges. Studies of psychopathology often in-
tients would indicate that their concreteness is greater than ex- volve group comparisons without random assignment. Investiga-
pected by the standards of a group that includes borderline retarded tors should consider ANCOVA and the alternatives briefly re-
persons. viewed here to improve their designs. A more fundamental point is
This method has several limitations. First, it is based on the that investigators should reexamine the issues they may have
hypothesis that the regression line for the original normal group wished to address with ANCOVA and consider whether other
and that for the additional impaired nonschizophrenic group have issues are more fruitful theoretically and more achievable
the same slope and intercept. If either the slopes or intercepts methodologically.
should differ much, the combining of the two groups to compute
a joint regression line would appear inappropriate. In addition, the References
range of scores on the predictor variable (IQ in the example) must Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable
be as great for the combined control group as for the schizophrenic distinction in social psychological research: Conceptual, strategic, and
patients. This is crucially important for the residualized scores to statistical considerations. Journal of Personality and Social Psychol-
be meaningful. ogy, 51, 1173-1182.
 48 MILLER AND CHAPMAN

Bock, D. (1975). Multivariate statistical methods in behavioral research. analyzing data: A model comparison perspective. Belmont, CA: Wads-
New York: McGraw-Hill. worth.
Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi- Maxwell, S. E., Delaney, H. D., & Manheimer, J. M. (1985). ANOVA of
experimental designs for research. Chicago: Rand McNally. residuals and ANCOVA: Correcting an illusion by using model com-
Chapman, L. J., & Chapman, J. P. (1973). Disordered thought in schizo- parisons and graphs. Journal of Educational Statistics, 10, 197-209.
phrenia. New York: Appleton-Century-Crofts. Miller, M. B., Chapman, L. J., & Chapman, J. P. (1993). Slowness and the
Cochran, W. G. (1957). Analysis of covariance: Its nature and uses. preceding preparatory interval effect in schizophrenia. Journal of Ab-
Biometrics, 44, 261-281. normal Psychology, 102, 145—151.
Cohen, J., & Cohen, P. (1975). Applied multiple regression/correlation Neale, J. M., & Oltmanns, T. F. (1980). Schizophrenia. New York: Wiley.
analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum. Overall, J. E., & Woodward, J. A. (1977). Nonrandom assignment and the
Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis of covariance. Psychological Bulletin, 84, 588-594.
analysis for the behavioral sciences (2nd edition). Hillsdale, NJ: Erl- Porter, A. C., & Raudenbush, S. W. (1987). Analysis of covariance: Its
baum. model and use in psychological research. Journal of Counseling Psy-
Deldin, P. J. (1996). Information processing in major depression: The ERP chology, 34, 383-392.
connection. Unpublished doctoral dissertation, University of Illinois, Reichardt, C. S. (1979). The statistical analysis of data from nonequivalent
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Champaign. group designs. In T. D. Cook & D. T. Campbell (Ms.), Quasi-experi-

This document is copyrighted by the American Psychological Association or one of its allied publishers.

Dixon, W. J. (1992). BMDP statistical software manual. Los Angeles: mentation: Design and analysis issues for field settings (pp. 147-205).
University of California Press. Boston: Houghton Mifflin.
Elashoff, J. D. (1969). Analysis of covariance: A delicate instrument. Reichardt, C. S., & Bormann, C. A. (1994). Using regression models to
American Educational Research Journal, 6, 383-401. estimate program effects. In J. S. Wholey, H. P. Harry, & K. E.
Evans, S. H., & Anastasio, E. J. (1968). Misuse of analysis of covariance Newcomer (Eds.), Handbook of practical program evaluation (pp. 417—
when treatment effect and covariate are confounded. Psychological 455). San Francisco: Jossey-Bass.
Bulletin, 69, 225-234. Richards, J. E. (1980). The statistical analysis of heart rate: A review
emphasizing infancy data. Psychophysiology, 17, 153-166.
Fleiss, J. L., & Tanur, J. M. (1973). The analysis of covariance in psycho-
Rogosa, D. (1980). Comparing nonparallel regression lines. Psychological
pathology. In M. Hammer, K. Salzinger, & S. Sutton (Eds.), Psychopa-
Bulletin, 88, 307-321.
thology: Contributions from the social, behavioral, and biological sci-
Rosenbaum, P. R. (1984). From association to causation in observational
ences (pp. 509-527). New York: Wiley.
studies: The role of strongly ignorable treatment assignment. Journal of
Harris, D. R., Bisbee, C. T., & Evans, S. H. (1971). Further comments:
the American Statistical Association, 79, 41-48.
Misuse of analysis of covariance. Psychological Bulletin, 75, 220-222.
Rosenbaum, P. R. (1987). The role of a second control group in an
Heckman, J. J. (1989). Causal inference and nonrandom samples. Journal
observational study (with discussion). Statistical Science, 2, 292-316.
of Educational Statistics, 14, 159-168.
Rosenbaum, P. R. (1995). Observational studies. New York: Springer-
Heller, W., Etienne, M. A., & Miller, G. A. (1995). Patterns of perceptual
Verlag.
asymmetry in depression and anxiety: Implications for neuropsycholog-
Rosenbaum, P. R., & Rubin, D. (1984). Reducing bias in observational
ical models of emotion and psychopathology. Journal of Abnormal
studies using subclassifications on the propensity score. Journal of the
Psychology, 104, 327-333.
American Statistical Association, 79, 516-524.
Holland, P. W., & Rubin, D. B. (1983). On Lord's paradox. In H. Wainer Rosvold, H. E., Mirsky, A. F., Sarason, I., Bransome, E. D., Jr., & Beck,
& S. Messick (Eds.), Principles of modem psychological measurement L. H. (1956). A continuous performance test of brain damage. Journal of
(pp. 3-35). Hillsdale, NJ: Erlbaum. Consulting Psychology, 20, 346-350.
Huitema, B. (1980). Analysis of covariance and alternatives. New York: Rubin, D. B. (in press). Propensity score methods. In L. Bickman (Ed.),
Wiley. Contributions to research design: Donald Campbell's legacy (Vol. 2).
Jin, P. (1992). Toward a reconceptualization of the Law of Initial Value. Thousand Oaks, CA: Sage.
Psychological Bulletin, 111, 176-184. Siddle, D. A. T., & Turpin, G. (1980). Measurement, quantification, and
Judd, C. M., McClelland, G. H., & Smith, E. R. (1996). Testing treatment analysis of cardiac activity. In I. Martin & P. H. Venables (Eds.),
by covariate interactions when treatment varies within subjects. Psycho- Techniques in psychophysiology (pp. 139-246). London: Wiley.
logical Methods, 1, 366-378. Thernstrom, A. (1991, December 16). Beyond the pale. New Republic.
Keller, J., Nitschke, J. B., Bhargava, T., Deldin, P. J., Gergen, J. A., Miller, Wainer, H. (1991). Adjusting for differential base rates: Lord's Paradox
G. A., & Heller, W. (2000). Neuropsychological differentiation of de- again. Psychological Bulletin, 109, 147-151.
pression and anxiety. Journal of Abnormal Psychology, 109, 3-10. Walker, E., & Lewine, R. (1993). Sampling biases in studies of gender and
Kessler, R. C., McGonagle, K. A., Zhao, S., Nelson, C. B., Hughes, M., schizophrenia. Schizophrenia Bulletin, 19, 1-7.
Eshleman, S., Wittchen, H., & Kendler, K. S. (1994). Lifetime and Watson, D., Weber, K., Assenheimer, J. S., Clark, L. A., Strauss, M. E., &
12-month prevalence of DSM-III-R psychiatric disorders in the United McCormick, R. A. (1995). Testing a tripartite model: I. Evaluating the
States. Archives of General Psychiatry, 51, 8-19. convergent and discriminant validity of anxiety and depression symptom
Lord, F. M. (1967). A paradox in the interpretation of group comparisons. scales. Journal of Abnormal Psychology, 104, 3-14.
Psychological Bulletin, 68, 304-305. Wildt, A. R., & Ahtola, O. T. (1978). Analysis of covariance. Beverly
Lord, F. M. (1969). Statistical adjustments when comparing preexisting Hills, CA: Sage.
groups. Psychological Bulletin, 72, 336-337.
Maris, E. (1998). Covariance adjustment versus gain scores—revisited. Received November 3, 1998
Psychological Methods, 3, 309-327. Revision received September 7, 2000
Maxwell, S. E., & Delaney, H. D. (1990). Designing experiments and Accepted September 7, 2000 •