the second linear combination of variables must not correlate with the first one.

The
UNIT 26 CANONICAL CORRELATION process is repeated until a successive linear combination is no longer significant.
ANALYSIS
Canonical correlation is a member of the multiple general linear hypothesis (MLGH)
family and shares many of the assumptions of mutliple regression such as linearity of
Structure Page No. relationships, homoscedasticity (same level of relationship for the full range of the
data), interval or near-interval data, untruncated variables, proper specification of the
26.1 Introduction model, lack of high multicollinearity, and multivariate normality for purposes of
26.2 Mathematical Formulation and Computations of Canonical correlation hypothesis testing.
26.3 Nonlinear Canonical Correlation
26.4 Summary Often in applied research, scientists encounter variables of large dimensions and are
26.5 Solutions/Answers faced with the problem of understanding dependency structures, reduction of
26.6 Practical Assignment dimensionalities, construction of a subset of good predictors from the explanatory
variables, etc. Canonical correlation Analysis (CCA) provides us with a tool to attack
26.1 INTRODUCTION these problems. However, its appeal and hence its motivation seem to differ from the
theoretical statisticians to the social scientists. We deal here with the various
Canonical correlation is a technique to identify and quantify the association between motivations of CCA as mentioned above and related statistical inference procedures.
two sets of variables. Each set can contain several variables. Simple and multiple We shall begin the unit by discussing the canonical correlation in Section 26.2. In
correlations are special cases of canonical correlation in which one or both sets contain Section 26.3, we shall focus on non linear canonical correlation
a single variable. This technique was given by H. Hotelling in 1935-36, for relating .
the arithmetic speed and arithmetic power to reading speed and reading power based Objectives
on a sample data received from 140 seventh grade students. Other examples where After reading this unit, you should be able to
canonical correlations can be helpful are: relating governmental policy variables with
economic goal variables; relating college performance variables (grades in courses in • understand the meaning and concept of canonical correlation
different subjects) with pre-college achievement variables (percentage of marks in • interpret the results of a canonical correlation analysis
high school, number of extracurricular activities in height school, etc.); relating yield
attributing parameters (test weight, plant height, number of grains per panicle, etc.) • make use of canonical correlation
and quality parameters (protein content, carbohydrate content, etc.) in case of a certain
crop; relating job satisfaction variables (supervisor satisfaction, workload satisfaction, 26.2 MATHEMATICAL FORMULATION AND
general satisfaction, etc.) and job characteristic variables (feedback, task identity, task
variety, etc.); relating physiological variables (weight in kg, waist in inches, pulse rate, COMPUTATIONS
etc.) with exercise variables (number of sit ups, jumps, etc.) and many others such Consider two groups of variables. The first set of p variables is represented by a
pairs. ( p ×1 ) random vector X and the second set of q variables is represented by the ( q ×1 )
random vector Y. Without loss of generality, we assume that p ≤ q . For two random
Canonical correlation analysis actually focuses on the correlation between a linear
vectors X and Y, let E(X) = X ; D(X) = ∑11; E(Y) = Y ; D(Y) = ∑22 ; Cov (X, Y) =
combination of the variables in one set and a linear combination of the variables in the
second set. The idea is first to determine the pair of linear combinations having the ∑12. Here E (.) denotes expectation and D(.) denotes the variance-covariance matrix.
largest correlation. Next we determine the pair of linear combinations having the For the sake of convenience, let us consider X and Y jointly as the random vector
largest correlation among all pairs uncorrelated with the initially selected pair. This X  X
process continues until the number of pairs of canonical variables equals the number Z = ⋯ with mean vector = E(Z) =  ⋯  and variance covariance matrix as
of variables in the smaller group. The pairs of linear combinations are called the  Y   Y 
canonical variables and their correlations are called canonical correlations. The
canonical correlations measure the strength of association between the two sets of Σ Σ12 
variables. The maximization aspect of the technique represents an attempt to D(Z) =  11  . The pq elements in Σ12 gives the measures of association
 Σ 21 Σ 22 
concentrate a high-dimensional relationship between two sets of variables into a few
between p variables of first set and q variables of second set. The canonical correlation
pair of canonical variables.
analysis summarizes the associations between X and Y in terms of few correlations
The purpose of canonical correlation is to explain the relation of the two sets of rather than the pq elements in Σ12 .
variables, not to model the individual variables. Consider U = a ′X and V = b′Y as the linear combinations of variables within the two
sets for some coefficient vectors a and b . Now, using simple statistical computations,
Analogous with ordinary correlation, canonical correlation squared is the percent of one can see that
variance in the dependent set explained by the independent set of variables along a Var(U) = a ′Σ11a ; Var(V) = b′Σ 22 b and Cov ( U, V ) = a′Σ12 b .
given dimension (there may be more than one). In addition to asking how strong the As a consequence
relationship is between two latent variables, canonical correlation is useful in a ′Σ12 b
determining how many dimensions are needed to account for that relationship. Corr(U,V) = . Through canonical correlation analysis we want to
Canonical correlation finds the linear combination of variables that produces the a ′Σ11a b′Σ 22 b
largest correlation with the second set of variables. This linear combination, or "root," find a and b such that Corr(U, V) is as large as possible.
is extracted and the process is repeated for the residual data, with the constraint that
The first linear combination of two sets of variables known as first pair of canonical
variables is Remark 3: The canonical correlations measure the linear association between two sets
U1 = a11x1 + a21x2 + ... + ap1xp = a ′1 X of variables. If the variables are associated in a nonlinear manner, the association can
V1 = b11y1 + b21 y2 + ... + bq1 yq = b 1′ Y . not be captured by canonical correlation. Thus, while canonical correlation analysis is
the most generalized multivariate method, it is still constrained to identifying linear
The coefficients are so chosen that the two canonical variables, U1 and V1 have unit
associations. Canonical correlation analysis can accommodate any metric variable
variances and have the largest possible correlation. This maximized correlation
without the strict assumption of normality. Normality is desirable because it
between the two canonical variables is the first canonical correlation. The coefficients
standardizes a distribution to allow for a higher correlation among the variables. But in
of the linear combinations are canonical coefficients or canonical weights. Let the
the strictest sense, canonical correlation analysis can accommodate even non-normal
maximum correlation attained in first pair of canonical variables be Corr(U1 ,V1 ) = ρ1 . variables if the distributional form (e.g., highly skewed) does not decrease the
correlation with other variables. This allows for transformed nonmetric data (in the
The second set of canonical variables, uncorrelated with the first pair of canonical form of dummy variables) to be used as well. However, multivariate normality is
variables is required for the statistical inference test of the significance of each canonical function.
U2 = a12x1 + a22x2 + ... + ap2xp= a ′2 X Because tests for multivariate normality are not readily available, the prevailing
V2 = b12y1 + b22 y2 + ... + bq2 yq = b′2 Y . guideline is to ensure that each variable has univariate normality. Thus, although
The coefficients are so chosen that the two canonical variables, U2 and V2; U2 is normality is not strictly required, it is highly recommended that all variables be
uncorrelated with U1 and V1, V2 is uncorrelated with U1 and V1, and U2 and V2 have evaluated for normality and transformed if necessary. Homoscedasticity, to the extent
the largest possible correlation subject to these constraints. Let the maximum that it decreases the correlation between variables, should also be remedied. Finally,
correlation attained in second pair of canonical variables be Corr(U 2 ,V2 ) = ρ2 . multicollinearity among either variable set will confound the ability of the technique
to isolate the impact of any single variable, making interpretation less reliable.
This process continues until the number of pairs of canonical variables is equal to the Now, we shall discuss nonlinear canonical correlation in the following Section.
p
≤C
qo

number of variables in the set containing smaller number of variables. In this case
number of canonical variables will be p as . Let the maximum correlation
r
r
(
U
,
V
)
=
ρp

26.3 NONLINEAR CANONICAL CORRELATION

p

attained in pth pair of canonical variables be .

Now the question arises, how to compute ρ1 , ρ2 , …, ρp . Nonlinear canonical correlation analysis corresponds to categorical canonical
correlation analysis with optimal scaling. The OVERALS procedure in SPSS (part of
SPSS Categories) implements nonlinear canonical correlation. Independent variables
Through algebraic manipulations, it can easily be seen that the eigenvalues of can be nominal, ordinal, or interval, and there can be more than two sets of variables
−1/2
Σ11 Σ12 Σ −221 Σ 21 Σ 11
−1/2
are ρ12 ≥ ρ22 ≥ … ≥ ρp2 with corresponding p × 1 eigenvectors (more than one independent set and one dependent set). Whereas ordinary canonical
−1/2 −1 correlation maximizes correlations between the variable sets, in OVERALS the sets
as a1 , a 2 ,…, a p . Similarly the eigenvalues of Σ 22 Σ 21 Σ11 Σ12 Σ −221/2 are are compared to an unknown compromise set defined by the object scores
ρ12 ≥ ρ22 ≥ … ≥ ρp2 with corresponding q × 1 eigenvectors as b1 , b 2 ,… , b p . Therefore,
OVERALS makes use of optimal scaling, which quantifies categorical variables and
the maximum correlation, ρ1 is the positive square root of the largest eigenvalue of then treats as numerical variables, including applying nonlinear transformations to find
−1/2
Σ11 Σ12 Σ −221 Σ 21 Σ 11
−1/2 −1/2
or Σ 22 −1
Σ 21 Σ11 Σ12 Σ −221/2 . Similarly the maximum the best-fitting model. For nominal variables, the order of the categories is not retained
correlation between second pair of canonical variables ρ2 is the positive square root of but values are created for each category such that goodness of fit is maximized. For
−1/2 ordinal variables, order is retained and values maximizing fit are created. For interval
the second largest eigenvalue of Σ11 Σ12 Σ −221 Σ 21 Σ 11
−1/2 −1/2
or Σ 22 −1
Σ 21 Σ11 Σ12 Σ −221/2 . variables, order is retained as are equal distances between values.
The canonical variables are obtained by using the coefficient vectors as corresponding
ortho-normalized eigenvectors. This process continues till we get pth pair of canonical Obtain OVERALS from the SPSS menu by selecting Analyze, Data Reduction,
variables and p canonical correlations. In the above discussion, it has been tacitly Optimal Scaling; Select Multiple sets; Select either Some variable(s) not multiple
assumed that Σ ii , i = 1,2 is non-singular. If Σ ii , i=1 or 2 happens to be singular, one nominal or All variables multiple nominal; click Define; define at least two sets of
can use a g-inverse of Σ ii in place of true inverse of Σ ii . variables; define the value range and measurement scale (optimal scaling level) for
each selected variable. SPSS output includes frequencies, centroids, iteration history,
Remark 1: p = q = 1 ⇒ ρ1= usual Pearson's product moment correlation coefficient object scores, category quantifications, weights, component loadings, single and
multiple fit, object scores plots, category coordinates plots, component loadings plots,
between the scalar random variables X and Y; p = 1, q >1 ⇒ρ1= Multiple correlation category centroids plots, and transformation plots.
coefficient between the scalar X and the vector random variable Y. Sample analogues
are trivially defined.
Tip: To minimize output, use the Automatic Recode facility on the Transform menu
to create consecutive categories beginning with 1 for variables treated as nominal or
Remark 2: The canonical correlation also reduces the dimensionality. One feature of
ordinal. To minimize output, for each variable scaled at the numerical (integer) level,
dimensionality reduction is that we have only 2p canonical variables rather than p+q
subtract the smallest observed value from every value and add 1.
original variables. Further, reduction in dimensionality can be achieved by retaining
only those pair of canonical variables, whose correlation coefficient is significantly
different from zero. If the correlation coefficient between (k+1)th pair of canonical Warning: Optimal scaling recodes values on the fly to maximize goodness of fit for
variables is not significantly different from zero, then we can retain only first k pairs the given data. As with any atheoretical, post-hoc data mining procedure, there is a
of canonical variables. This gives only 2k variables rather than p+q original variables. danger of overfitting the model to the given data. Therefore, it is particularly
appropriate to employ cross-validation, developing the model for a training dataset
and then assessing its generalizability by running the model on a separate validation variable. Therefore, instead, Levine recommends interpreting the relations of the
dataset. original variables to a canonical variable in terms of the correlations of the original
variables with the canonical variables - that is, by structure coefficients. This is now
The SPSS manual notes, "If each set contains one variable, nonlinear canonical the standard approach.
correlation analysis is equivalent to principal components analysis with optimal
scaling. If each of these variables is multiple nominal, the analysis corresponds to Canonical correlation places the fewest restrictions on the types of data on which it
homogeneity analysis. If two sets of variables are involved and one of the sets contains operates. Because the other techniques impose more rigid restrictions, it is generally
only one variable, the analysis is identical to categorical regression with optimal believed that the information obtained from them is of higher quality and may be
scaling." presented in a more interpretable manner. For this reason, many researchers view
canonical correlation as a last-ditch effort, to be used when all other higher-level
Redundancy is the percent of variance in one set of variables accounted for by the techniques have been exhausted. But in situations with multiple dependent and
variate of the other set. The researcher wants high redundancy, indicating that independent variables, canonical correlation is the most appropriate and powerful
independent variate accounts for a high percent of the variance in the dependent set of multivariate technique. It has gained acceptance in many fields and represents a useful
original variables. Note this is not the canonical correlation squared, which the percent tool for multivariate analysis, particularly as interest has spread to considering
of variance in the dependent variate is accounted for by the independent variate. multiple dependent variables.

Applications of Canonical Correlation Analysis Now try the following exercises.

E1) Following data was collected on physiological variables (weight in pounds,
waist in inches and pulse rate) and exercise variables (chins, situps and jumps)
• There could be a situation where some of variables have high structure on middle-aged men in a fitness club. Perform the canonical correlation
correlations even though their canonical weights are near zero. This could happen analysis and interpret your results.
because the weights are partial coefficients whereas the structure correlations
(canonical factor loadings) are not: if a given variable shares variance with other Weight Waist Pulse Chins Situps Jumps
independent variables entered in the linear combination of variables used to create 191 36 50 5 162 60
a canonical variable, its canonical coefficient (weight) is computed based on the
189 37 52 2 110 60
residual variance it can explain after controlling for these variables. If an
193 38 58 12 101 101
independent variable is totally redundant with another independent variable, its
partial coefficient (canonical weight) will be zero. Nonetheless, such a variable 162 35 62 12 105 37
might have a high correlation with the canonical variable (that is, a high structure 189 35 46 13 155 58
coefficient). In summary, the canonical weights have to do with the unique 182 36 56 4 101 42
contributions of an original variable to the canonical variable, whereas the 211 38 56 8 101 38
structure correlations have to do with the simple, overall correlation of the original 167 34 60 6 125 40
variable with the canonical variable. 176 31 74 15 200 40
154 33 56 17 251 250
• Canonical correlation is not a measure of the percent of variance explained in the 169 34 50 17 120 38
original variables. The square of the structure correlation is the percent of the
166 33 52 13 210 115
variance in a given original variable accounted for by a given canonical variable
154 34 64 14 215 105
on a given (usually the first) canonical correlation. Note that the average percent
of variance explained in the original variables by a canonical variable (the mean of 247 46 50 1 50 50
the squared structure correlations for the canonical variable) is not at all the same 193 36 46 6 70 31
as the canonical correlation, which has to do with the correlation between the 202 37 62 12 210 120
weighted sums of the two sets of variables. Put another way, the canonical 176 37 54 4 60 25
correlation does not tell us how much of the variance in the original variables is 157 32 52 11 230 80
explained by the canonical variables. Instead, that is determined on the basis of the 156 33 54 15 225 73
squares of the structure correlations. 138 33 68 2 110 43
Source: http://v8doc.sas.com/sashtml/
• Canonical coefficients can be used to explain with which original variables a
canonical correlation is predominantly associated. The canonical coefficients are
E2) The variance covariance matrix between 5 yield attributing parameters (X1, X2,
standardized coefficients and (like beta weights in regression) their magnitudes
X3, X4 and X5) and four quality attributes (Y1, Y2, Y3, Y4) based on a sample
can be compared. Looking at the columns in SPSS output which list the canonical
of size 200 is given as (after standardizing the variables)
coefficients as columns and the variables in a set of variables as rows, some
researchers simply note variables with the highest coefficients to determine which
X1 X2 X3 X4 X5 Y1 Y2 Y3 Y4
variables are associated with which canonical correlations and use this as the basis
X1 1.000 0.754 -0.690 -0.440 0.702 -0.605 -0.480 0.780 -0.152
for inducing the meaning of the dimension represented by the canonical
X2 0.754 1.000 -0.710 -0.515 0.412 -0.722 -0.419 0.542 -0.100
correlation.
X3 -0.690 -0.710 1.000 0.323 -0.444 0.737 0.361 -0.546 0.172
X4 -0.440 -0.515 0.323 1.000 -0.334 0.527 0.461 -0.393 -0.019
However, Levine (1977) argues against the procedure above on the ground that the X5 0.702 0.412 -0.444 -0.334 1.000 -0.383 -0.505 0.737 -0.148
canonical coefficients may be subject to multi-collinearity, leading to incorrect Y1 -0.605 -0.722 0.737 0.527 -0.383 1.000 0.251 -0.490 0.250
judgments. Also, because of suppression, a canonical coefficient may even have a Y2 -0.480 -0.419 0.361 0.461 -0.505 0.251 1.000 -0.434 -0.079
different sign compared to the correlation of the original variable with the canonical
 Y3 0.780 0.542 -0.546 -0.393 0.737 -0.490 -0.434 1.000 -0.163  609.621053 68.800000 − 65.115789
Σ11 =  68.000000 10.252632 − 8.147368  .
Y4 -0.152 -0.100 0.172 -0.019 -0.148 0.250 -0.079 -0.163 1.000

Perform canonical correlation analysis and interpret your results. − 65.115789 − 8.147368 51.989474 

E3) Consider the following variance-covariance matrix Variance-covariance matrix of exercise variables chins, situps and jmps is
 27.944737 230.107895 134.384211 
Σ 22 = 230.107895 3914.576316 2146.984211
X1 X2 Y1 Y2 .
X1 100 0 0 0
X2 0 1 0.95 0 134.384211 2146.984211 2629.378947
Y1 0 0.95 1 0
Y2 0 0 0 100 Covariance matrix between physiological variables and exercise variables is
Obtain the correlation coefficient between first pair of canonical variables.  − 50.863158 − 9.347368 5.742105 
Σ12 =  − 761.715789 − 129.336842 101.521053 .
Now, let us summarize the unit.
 − 286.505263 101.521053 12.915789 

26.4 SUMMARY
In this unit, we have covered the following points. Step 2: To obtain Σ11−1 / 2
, first obtain eigenvalues and eigenvectors of Σ11 . Let
1) Canonical correlation is a technique to identify and quantify the association λi and γ i denote i eigenvalue and ith eigenvector respectively; i = 1,2,3. Now
th

between two sets of variables. 3

2) Canonical correlation analysis actually focuses on the correlation between a

−1/ 2
Σ11 = ∑λ
i =1
−1/ 2
i γ i γ i′ . The eigenvalues Σ11 are 624.93238, 44.487838 and
linear combination of the variables in one set and a linear combination of the
variables in the second set. 2.4429359. The corresponding eigenvectors are: (0.987172, 0.1120006, -
0.113786)’; (0.1150796, -0.005131, 0.993343)’; (-0.110671, 0.9936949,
3) Simple and multiple correlations are special cases of canonical correlation in 0.017954)’. Now
which one or both sets contain a single variable.  0.0488043 − 0.066027 0.0113741
4) The canonical correlation also reduces the dimensionality. −1 / 2
Σ11 = − 0.066027 0.6322628 0.0101406 .
5) Nonlinear canonical correlation analysis corresponds to categorical canonical  0.0113741 0.0101406 0.1486615
correlation analysis with optimal scaling.
−1/2
Step 3: Compute Σ11 Σ12 Σ −221 Σ 21 Σ 11
−1/2
=F (say). In this case it is
26.5 SOLUTIONS/ANSWERS  0.2033438 0.2611853 − 0.044758
F =  0.2611853 0.4540921 − 0.083341 .
E1) Step 1: Obtain the variance covariance matrix of weight, waist, pulse rate, chins, − 0.044758 − 0.083341 0.0210455 
situps and jumps.
Covariance Matrix Step 4: Obtain eigenvalues and eigenvectors of F. The eigenvalues are
0.6332992, 0.040223, 0.005266 and the corresponding eigenvectors are
Weight Waist Pulse Chins Situps Jumps
a1′ = (0.524930, 0.837384, − 0.152437)′ ;
Weight 609.621053 68.800000 -65.115789 -50.863158 - -
761.715789 286.505263 a′2 = (0.847489, 0.497640, 0.184708)′ ;
a′3 = (0.078813, 0.226147, 0.970900)′ respectively.
Waist 68.800000 10.252632 -8.147368 -9.347368 - -31.442105
129.336842
Step 5: On the similar lines of Steps 2, 3 and 4 obtain
Pulse -65.115789 -8.147368 51.989474 5.742105 101.521053 12.915789
Σ −221/2 Σ 21Σ11
−1
Σ12Σ −221/2 = G , its eigenvalues and eigenvectors.
Chins -50.863158 -9.347368 5.742105 27.944737 230.107895 134.384211
 0.2634144 − 0.013700 − 0.001758
Situps - - 101.521053 230.107895 3914.57631 2146.98421
G = − 0.013700 0.0205081 − 0.008377 . The eigenvalues are
761.715789 129.336842 6 1
Jumps - -31.442105 12.915789 134.384211 2146.98421 2629.37894 − 0.001758 − 0.0083771 0.0248539 
286.505263 1 7 0.6332992, 0.040223, 0.005266 and the corresponding eigenvectors are
b1′ = (0.297668, 0.926878, − 0.228672)′ ;
Now variance-covariance matrix of physiological variables weight, waist and b′2 = (−0.247629, 0.3062953, 0.9191642)′ ;
pulse is b′3 = (0.9219943, − 0.216980, 0.3206965)′ respectively
 −1 / 2
Step 6: The first pair of canonical variables is now given by a1′ Σ11 X and  0.2863075 0.0949999 − 0.119406 − 0.084734 0.2775501 
 0.0949999 0.2120403 − 0.238331 − 0.18475 0.0624013 
b1′ Σ −221 / 2 Y .
Here X denotes the matrix of physiological variables and Y denote 
the matrix of exercise variables. Similarly other two pairs of canonical variables F = − 0.119406 − 0.238331 0.2852454 0.1846082 − 0.069907 .
can be obtained.  
− 0.084734 − 0.18475 0.1846082 0.1997659 − 0.089549
−1 / 2
 0.2775501 0.0624013 − 0.069907 − 0.089549 0.3175887 
a1′ Σ11 = [-0.031405 0.4932416 -0.008199];
−1 / 2
a′2 Σ11 = [0.0763195 -0.368723 0.032052]; Step 4: Obtain first four eigenvalues and eigenvectors of F. The eigenvalues are
−1 / 2 0.8265343, 0.4002511, 0.0651783, 0.0089843 and the corresponding
a′3Σ11 = [-0.007735 0.1580337 0.1457322]. eigenvectors are
a1′ = (0.4743674, 0.4263205, − 0.485436, − 0.396704, 0.4474416)′ ;
b1′ Σ −221 / 2 = [0.066114 0.0168462 -0.013972]; a′2 = (0.4682176, − 0.392238, 0.428314, 0.2902513, 0.599352)′ ;
b′2 Σ −221 / 2 = [-0.071041 0.0019737 0.0207141]; a′3 = (0.3723098, 0.0200515, − 0.485523, 0.736616, − 0.287484)′;
b′3Σ −221 / 2 = [0.2452754 -0.019768 0.0081675]. a′4 = (0.6273877, 0.1540741, 0.4326632, − 0.259644, − 0.572742)′ respecti
vely.
The canonical correlations between three pairs of canonical variables can be
obtained by taking the square root of the eigenvalues of Step 5: On the similar lines of Steps 2, 3 and 4 obtain
−1/2
Σ11 Σ12 Σ −221 Σ 21 Σ 11
−1/2
or Σ −221/2 Σ 21Σ11
−1
Σ12 Σ −221/2 .
Here the canonical Σ −221/2 Σ 21Σ11
−1
Σ12Σ −221/2 = G , its eigenvalues and eigenvectors.
correlations are 0.795608, 0.200556 and 0.072570.  1.1354777 − 0.073763 0.2631705 − 0.124552
− 0.073763 1.0984645 0.2394113 0.0889018 
E2) Step 1: Variance-covariance matrix of five yield attributing parameters X1, X2, G= .
 0.2631705 0.2394113 1.1877782 0.0714007 
X3, X4 and X5) is Σ11 and is given by 5× 5 cells in left hand upper corner of  
the given variance covariance matrix. The variance-covariance matrix of four − 0.124552 0.0889018 0.0714007 1.0378877 
quality parameters Y1, Y2, Y3 and Y4 is Σ 22 and is given by 4× 4 cells in
The eigenvalues are 0.8265343, 0.4002511, 0.0651783, 0.0089843 and the
right hand bottom corner of the given variance covariance matrix. The
corresponding eigenvectors are
covariance matrix between yield attributing parameters and quality attributes is
b1′ = (0.6527588, 0.4359061, − 0.616865, 0.0580444)′;
Σ12 and is the 5× 4 cells in right hand upper corner of the given variance
covariance matrix. b′2 = (0.7101565, − 0.072348, 0.6880977, − 0.13025)′;
b′3 = (−0.258398, 0.8350398, 0.279248, − 0.397442)′;
−1 / 2
Step 2: To obtain Σ11 , first obtain eigenvalues and eigenvectors of Σ11 . Let b′4 = (−0.05305, 0.3278111, 0.2608056, 0.90648)′ respectively.
λi and γ i denote i eigenvalue and ith eigenvector respectively; i = 1,2,3. Now
th

−1 / 2
3
Step 6: The first pair of canonical variables is now given by a1′ Σ11 X and
−1/ 2
Σ11 = ∑λ −1/ 2
i γ i γ i′ . The eigenvalues Σ11 are 3.1758116, 0.7417691,
i =1 b1′ Σ −221 / 2 Y . Here X denotes the matrix of yield attributing parameters and Y
0.6572676, 0.275192 and 0.1499598. denote the matrix of quality attributes. Similarly other three pairs of canonical
The corresponding eigenvectors are: variables can be obtained.
(0.5157006 0.2095696 0.0655101 0.3701104 -0.740851)’;
(0.4876162 -0.15341 -0.3956110 0.5454541 0.5335426)’; The canonical correlations between four pairs of canonical variables can be
(-0.457400 -0.206258 0.5068357 0.7007362 0.0181492)’; obtained by taking the square root of the eigenvalues of
(-0.349891 0.8468537 -0.3073330 0.2408609 0.0891511)’ and −1/2 −1
(0.4057652 0.4157429 0.6984732 -0.128269 0.39773710)’.
−1/2
Σ 11 Σ 12 Σ −221 Σ 21 Σ 11
−1/2
or Σ 22 Σ 21Σ11 Σ12 Σ −221/2 . Here the canonical
Now correlations are 0.9091, 0.6327, 0.2553 and 0.0948.
 1.8839857 − 0.564091 0.3180696 0.0793549 − 0.576395
 − 0.564091 1.6560546 0.4178776 0.2766612 0.1107616  E3) Proceeding on the same steps as in E2), one can easily see that the canonical
 correlation between first pair of canonical variables is 0.95. Which is also
−1 / 2
Σ11 =  0.3180696 0.4178776 1.4205354 0.0207793 0.0802538  . obvious.
 
 0.0793549 0.2766612 0.0207793 1.1490046 0.0970125 
− 0.576395 0.1107616 0.0802538 0.0970125 1.334717 

−1/2
Step 3: Compute Σ 11 Σ 12 Σ −221 Σ 21 Σ 11
−1/2
=F (say). In this case it is
 Canonical Correlation Quantitative Methods-II correlation analysis is a powerful analytical tool to analyze the association between
UNIT 14 CANONICAL CORRELATION Analysis two sets of variables belonging to different disciplines. This method measures the
strength of association between the two sets of variables. An attempt is made in this
ANALYSIS method to concentrate a high-dimensional relationship between two sets of variables
into a few pairs of canonical variables. Hence, in this unit, we shall throw light on
various issues relating to canonical correlation like concept and meaning of canonical
Structure correlation, its similarity and differences with multiple regression, procedure involved
14.0 Objectives in the analysis of canonical correlation, and its advantages and limitations.

14.1 Introduction
14.2 CANONICAL CORRELATION ANALYSIS (CCA):
14.2 Canonical Correlation Analysis (CCA): Concept and Meaning
CONCEPT AND MEANING
14.3 Assumptions of Canonical Correlation
14.4 Canonical Correlation Analysis as Generalization of the Multiple Canonical correlation analysis is a multivariate statistical model used to study the
interrelationships among sets of multiple dependent variables and multiple
Regression Analysis
independent variables. This technique is distinct from the multiple regression model in
14.5 Steps and Procedure Involved in Computation of CCA Results the sense that multiple regression predicts a single dependent variable from a set of
multiple independent variables whereas canonical correlation simultaneously predicts
14.6 Illustration of CCA multiple dependent variables from multiple independent variables.
14.7 Interpretation of CCA Results
Let us understand the concept with an example. As a student of economics, you may
14.8 Limitations of Canonical Correlation like to know the association between economic inequality X and political instability
14.9 Let Us Sum Up Y . The economic inequality can be measured by five variables i.e. (i) the division of
farmland X , (ii) the gini coefficient X , (iii) the percentage of tenant farmers
14.10 Key Words X , (iv) the gross national product X , and (v) the percentage of farmers X .
14.11 Some Useful Books Similarly the political instability can be measured by four variables (indicators) i.e. (i)
the instability of leadership Y , (ii) the level of internal group violence Y , (iii) the
14.12 Answers or Hints to Check Your Progress occurrence of internal war Y , (iv) stability of democracy Y . These two
theoretical concepts X and Y can be called two sets of variables or canonical
14.13 Exercises
variables. These can be shown in the following figure:

14.0 OBJECTIVES Y1
X1
After going through this unit, you will be able to:
• explain the concept of canonical correlation analysis;
X2 X* Y* Y2
• state the similarity and difference between multiple regression and canonical
correlation;
. .
• discuss the steps and procedure involved in canonical correlation;
. .
• elucidate how to interpret the results of canonical correlation; X5 Y4
• point out the limitations of canonical correlation analysis.

Fig. 14.1: Canonical Correlation

14.1 INTRODUCTION
Source: The sage encyclopedia of Social Sciences Research Methods vol. 1 (2004) page no. 83.

The first canonical variable X is measured by five variables (P = 5) and can be

The conventional wisdom that the economic agents are rational and are guided by self-
interest in decision making is being questioned as it ignores the psychological and
considered as a linear combination (a weighted sum) of these X variables. The second
canonical variable Y is a linear combination of the q = 4 indicators, Y to Y . The
social factors influencing decision making process. The research findings from many
disciplines like neuroscience, cognitive science, psychology, behavioral economics,
double side curved arrow indicates that the question of casualty remains open.
sociology, anthropology etc. indicate that the decisions made by the individuals in
many aspects of development (like savings, investment, energy consumption, health The purpose of canonical correlation analysis is to find the correlation between a
etc.) are influenced by social contexts, local social networks, cultural factors, social linear combination of the variables in one set and a linear combination of the variables
norms and shared mental models etc. (World Development Report, 2015). Hence, in another set. The idea behind this approach is first to determine the pair of linear
inter-disciplinary perspective is being recognized as research approach to analyze combinations having the largest correlation. Next, we determine the pair of linear
human behavior so as to improve the predictive power of economics. Canonical combinations having the largest correlation among all pairs uncorrelated with the

31
initially selected pair, and so on. As stated in the above para, the pairs of linear Quantitative Methods-II
combinations are called the canonical variables and their correlations are called
Canonical Correlation
Analysis
14.5 STEPS AND PROCEDURE INVOLVED IN
canonical correlations. COMPUTATION OF CCA
Thus, canonical correlation aims to (i) identify the dimensions among the dependent In 1935-36, Hotelling proposed a method, known as Canonical Correlation Analysis to
and independent variables, and (ii) maximize the relationship between the dimensions. investigate “linear” relationship between the two sets of variates.
In this manner, in canonical correlation, we can distinguish three types of correlations: James Press (2005) has expressed the whole idea of this Canonical Correlation
1) Correlation between X variables, the correlation matrix is R . Analysis in the following words:

2) Correlation between Y variables, the correlation matrix is R . “The Canonical correlation” model selects weighted sums of variables from each of
3) Correlation between X and Y variables, the correlation matrix is R R′ .
the two sets to form new variables in each of the sets, so that the correlation between
the new variables in “different sets” is maximized while the new variables within each
set are constrained to be uncorrelated with mean zero and unit variance.
14.3 ASSUMPTIONS OF CANONICAL
We shall adhere to Press’s approach.
CORRELATION
Let
1) The correlation coefficient between any two variables is based on linear α: p 1
relationship.
and
2) The canonical correlation is the linear relationship between the variates.
γ: p 1
3) The distribution of variables is normal.
be two unknown vectors to be determined such that the correlation between α′ Y and
4) Hetro-scedasticity, to the extent it decreases the correlation between variables. γ′ Z be as large as possible.
So, let
14.4 CANONICAL CORRELATION ANALYSIS AS
U α′ Y
A GENERALIZATION OF MULTIPLE
REGRESSION ANALYSIS V γ′ Z
and
Multiple regression analysis is a multivariate technique which can predict the value of U ,V Correlation coefficient between U and V .
a single dependent variable from a linear function of a set of independent variables.
But this is not always the case. There are real life problems, however, when interest The problem of correlation coefficient now amounts to the following:
may not center on a single dependent variable. Rather, the researcher may be maximize U ,V
interested in relationships between sets of multiple dependent and multiple
independent variables. Canonical correlation analysis is a multivariate statistical model Subject to
that facilitates the study of interrelationships among sets of multiple dependent Var U Var V 1
variables and multiple independent variables. Whereas multiple regression predicts a
single dependent variable from a set of multiple independent variables, canonical and
correlation simultaneously predicts multiple dependent variables from multiple E U E V 0.
independent variables. Therefore, canonical correlation analysis is said to be a
generalization of multiple correlation used in multiple regression problems. Hotelling solved this problem using the celebrated method of Lagrange Multipliers.
2
However, we shall just state the final results. Hotelling showed that this maximization
The coefficient of determination R , in regression problems is the proportion of the problem is equivalent to following algorithm:
variability in a dependent variable that is accounted for by a set of predictor variables
and √ is called the multiple correlation coefficient. The multiple correlation Step 1: Solve for the equation

! Σ
coefficient can also be interpreted as a measure of the maximum correlation that is
Σ
0
! Σ
attainable between the dependent variable and any linear combination of the predictor ----------------- (A)
Σ
variable. Canonical correlation places the fewest restrictions on the types of data on
Var Y
which it operates. Because the other techniques impose more rigid restrictions, it is
Here, Σ
generally believed that the information obtained from them is of higher quality and
may be presented in a more interpretable manner. For this reason, many researchers Σ Var Z
Var Y, Z
view canonical correlation as a last effort, to be used when all other higher-level
Σ Σ
techniques have been exhausted. But in situations with multiple dependent and
independent variables, canonical correlation is the most appropriate and powerful Let be the largest positive root of the above equation.
multivariate technique. It has gained acceptance in many fields and represents a useful
tool for multivariate analysis, particularly while considering multiple dependent
variables.

33
Step 2: Now, solve the system of equations: Canonical Correlation Quantitative Methods-II and
! Σ Σ % Analysis
" # $& ' 0
Σ ! Σ
 V1 
for α and γ.  
Mathematically, it is not so simple to solve the above system of linear equations.  V2 
 . 
However, there is an equivalent formulation. To compute α and γ, we solve the pair of V=  V … VP ′
.
equations:  . 
(Σ Σ )
Σ ! Σ *α 0  . 
 
)  Vp 
(Σ Σ Σ ! Σ *γ 0  1
Step 3: We now call these α and γ as α and γ respectively; and The elements of U and V are called Canonical Variates and 2 is corresponding
canonical correlation.
U ′
α γ
and 14.6 ILLUSTRATION OF CCA
V γ′ Z
Let us now understand the canonical correlation analysis technique with the help of an
as First Canonical Variates. illustration.
It will turn out that will be the correlation coefficient between U and V . We,
therefore, write This example is based on 416 observations collected through primary survey for the
purpose of research study entitled “Assessment of Human Well-being in Delhi:
ρ U ,V Multidimensional Approach”. The researcher attempted to study the relationship
and call this as First Canonical Correlation. between economic well-being variables and overall life satisfaction variables. One of
the questions raised in this study was to examine how does economic well-being
Step 4: We now proceed to the next iteration. We now define: influence the overall life satisfaction of the people or more specifically, to know
U α′ γ whether economic well-being indicators are predictive of overall life satisfaction of
people.
V γ′ Z,
where % and & are to be determined. The main characteristic of canonical analysis is the investigation of the relationship
between two sets of variables. One set is the predictor set or, say, analytically, the set
maximize U ,V of independent variables. The second consists of the criteria or dependent variables.
subject to In our example, the set of economic well-being variables constitutes the set of
independent variables. This set of economic well-being variables consist of 5
Var U Var V 1 indicators of economic well-being:
and 1) Annual Income
E U E V 0. 2) Movable Assets
We repeat the above procedure to compute α and γ as solution of 3) Fixed Assets
! Σ Σ % 0 4) Employment Status
" # $& ' $ '
Σ ! Σ 0 5) Educational Attainment
where is the second largest positive root of the equation (A).
Next, the set of criteria variables constitute the overall life satisfaction indicators:
Step 5: We continue this procedure uptil the smallest positive root.
1) My life closer to my ideal life
Step 6: The result is now to be collected in a vector format:
2) Circumstances of my life are best to my choice
 U1 
  3) I am satisfied with my life
 U2  4) I have achieved the things in my life I aspired
 . 
 = (U … ./0 *
′
U = 5) I would not prefer to make any drastic change in my rest of life till I survive
 . 
 .  The data on these two sets of variables was collected through primary survey for the
  above said study.
 Up 
 1

35
Let us now learn to run the Canonical Correlation Analysis for our example using Canonical Correlation Quantitative Methods-II Note that the criterion set of variables are listed before WITH and predictor variables
SPSS Analysis are listed afterwords
Step 1: Click file, new and syntax sequence. Step 3: This command can be implemented using RUN button on toolbar menu.

Step 2: Then type the following syntax :

The results of above illustration using SPSS has been presented in this section. The
table 1 below shows an overall multivariate test of the entire model using different
multivariate criteria.
Table 14.1: Multivariate Tests of Significance
Test Name Value Approx. F Hypoth. DF Error DF Sig. of F
Pillais .14258 2.40693 25.00 2050.00 .000
Hotellings .15845 2.56302 25.00 2022.00 .000
Wilks .86055 2.49153 25.00 1509.72 .000
Roys .11708 .000

Table 14.2: Eigenvalues and Canonical Correlations

Root No. Eigenvalue Pct. Cum. Pct. Canon Cor. Sq. Cor
1 .13260 83.69055 83.69055 .34217 .11708
2 .01720 10.85381 94.54437 .13003 .01691
3 .00698 4.40258 98.94695 .08323 .00693
4 .00167 1.05222 99.99917 .04080 .00166
5 .00000 .00083 100.00000 .00115 .00000

37
In this example, we are not only interested to know whether there is a relationship Canonical Correlation Quantitative Methods-II Table 14.3: Canonical Solution for Economic Well-being Indicator Predicting
between the predicator and criterion variables but also wanted to know which Analysis Overall Life Satisfaction for Function 1
indicators of economic well-being are more or less useful in explaining the
relationship between life satisfaction and economic well-being. This is exactly where Variable Coef 45 45 2 (%)
CCA comes into play. The interpretation and evaluation of the results has been Life closer to ideal life .221 -.502 25.20
discussed in next section. Circumstances of life best to my choice -.692 -.905 81.90
Satisfied with life -.223 -.729 53.14
14.7 INTERPRETATION OF CCA RESULTS Achieved things in life -.400 -.746 55.65
We now come to the conclusive stages of the Canonical Correlation Analysis. Want no drastic change in life -.054 -.427 18.23
To draw the right conclusions amounts to the correct interpretation of the Annual Income -.381 -.857 73.44
results obtained by conducting the CCA. Fixed Assets .029 -.256 6.55

The first step is to evaluate the overall statistical significance of the full Movable Assets -.210 -.696 48.44
canonical model. This is done by testing the Null hypothesis. The Null Educational Attainment -.668 -.895 80.10
hypothesis is that there is no relationship between the two set of variables. The Employment status .219 -.286 8.17
alternative hypothesis is that the two sets of variables are related.
Coef: Standized Canonical Function Coefficient, r3 = Structure Coefficient, r3 2 =
Squared Structure Coefficient
The interpretation of CCA is now accomplished by computing the F-statistic
value using the Wilks- test. Looking at the coefficient, we find that relevant dependent variables are: the
circumstances of life are best to my choice, I am satisfied with my life and I have
We now come to the actual interpretation of the CCA as conducted in our example.
achieved things in my life I aspired, because all these variables high squared structure
Here, the value of the computed value of the F-statistic is high and significant. This coefficient which indicates the amount of variance the observed variable can
is given in table 1 as 0.860. This means that we can reject the Null hypothesis. contributed to new latent criterion variable. Looking at the other side of the equation
That is to say that we accept the alternative hypothesis. Thus, statistically significant on function 1 which involves predicator set, we find that Annual Income, Movable
relationship exists between life satisfaction and economic well-being indicators. Assets and Educational Attainment variables were primary contributors. You must
note that this process for interpretation of function is same as identifying the useful
Let us recall, that the first root (function) is created in such a manner that the canonical predictors in regression analysis with the exception that in canonical correlation
correlation between the new variable is maximized and these new variables within analysis we have two set of equations for consideration.
each set are uncorrelated with zero mean and unit variance.
Check Your Progress I
Let us interpret only those functions which explain reasonable amount of variance
between variable sets. In our illustrations, we interpret only the first function as it 1) What do you mean by the term ‘inter-disciplinary perspective’
explains 12% variance within the function as shown in table no. 2. All other functions ………………………………………………………………………………………
explain less than 10% of variance in their functions, hence we can ignore them.
………………………………………………………………………………………
So, what we have concluded so far is that there is a statically significant relationship ………………………………………………………………………………………
between our two set of variables. Further, this relationship is largely captured by the
first root (function) in the canonical model. Next, we want to identify those variables 2) How is CCA useful to address the inter-disciplinary nature of research questions?
which contribute significantly to explain this relationship between economic well- ………………………………………………………………………………………
being and overall life satisfaction.
………………………………………………………………………………………
In multiple regression analysis, we often look at Beta weights to identify the relative ………………………………………………………………………………………
contribution of one independent variable to explain dependent variable. In CCA, we
look at the structure coefficient to decide which variables are useful for the model. 3) State the steps involved in computation of CCA through SPSS software.
Therefore, we examine the standardized weights and structure coefficients to interpret ………………………………………………………………………………………
the first root (function). Let us underline the point that we are only concerned with the
first function and will ignore other functions as they are not significant. ………………………………………………………………………………………
………………………………………………………………………………………
To understand the pattern among two set of variables, we have created table 3 showing
coefficients which presents the standardized canonical function coefficients (i.e.
weights) and structure coefficients for all variables. The squared structure coefficient 14.8 LIMITATIONS OF CANONICAL
(r3 2) are also given, which represent the percentage of shared variance between CORRELATION
observed variable and the new variable created from the observed variables set.
1) The canonical correlation express the variance shared by the linear composites of
the set of variables and not the variance extracted from the variables.

39
2) Canonical weights derived in computing canonical functions are subject to great Canonical Correlation Quantitative Methods-II Canonical loadings : Measure of the simple linear correlation between
deal of instability. Analysis the independent variables and their respective
canonical variates. These can be interpreted like
3) The interpretation of canonical variates is different due to the efforts to maximize
factor loadings, and are also known as canonical
the relationship.
structure correlations.
4) It is difficult to identify meaningful relationship between the subset of independent
Canonical roots : Squared canonical correlations, which provide an
and dependent variables because precise statistics is yet to be developed.
estimate of the amount of shared variance between
the respective optimally weighted canonical
14.9 LET US SUM UP variates of dependent and independent variables. It
is also known as eigenvalues.
Canonical correlation analysis is a useful and powerful technique for exploring the Canonical variates : Linear combinations that represent the weighted
relationships among multiple dependent and independent variables. The technique is sum of two or more variables and can be defined
primarily descriptive, although it may be used for predictive purposes. In this for either dependent or independent variables. It is
technique, weighted sums of variables are selected from each of the two sets to form also known as linear composites, linear
new variables in each of the sets so that the correlation between the new variables in compounds, and linear combinations.
different sets in maximized while the new variables within each set are constrained to
be uncorrelated with mean zero and unit variance. Results obtained from a canonical Multiple regression : Multiple regression analysis predicts a single
analysis suggest answers to questions concerning the number of ways in which the two analysis dependent variable from a set of multiple
sets of multiple variables are related, the strengths of the relationships, and the nature independent variables.
of the relationships defined. Canonical analysis enables the students to combine into a
composite measure what otherwise might be an unmanageably large number of 14.11 SOME USEFUL BOOKS
bivariate correlations between sets of variables. It is useful for identifying overall
relationships between multiple independent and dependent variables, particularly 1) Alissa Sherry, Robin K. Henson (2005); Conducting and Interpreting Canonical
when we have little a priori knowledge about relationships among the data for two sets Correlation Analysis in Personality Research. Journal of Personality Assessment.
of variables. Essentially, we can apply canonical correlation analysis to a set of
2) Hotelling, H. (1936); Relations between two sets of variables, Biometrika 28, 321-377
variables that appear to be significantly related.
3) Johnson, R.A. & Wichern, D.W. (2002); Applied Multivariate Statistical Analysis,
The CCA is based on two statistical assumptions. First, the correlation coefficient
Pearson Education, Inc.
between any two variables is based on a linear relationship. Second, the parent
population from which the sample has been drawn is normally distributed. CCA has 4) Johnson, Dallas E. (1998); Applied Multivariate Methods for Data Analysis,
several advantages and limitations. International Thomson Publishing Inc.
5) Michael S. Lewis-beck Alan Bryman Tim Futing Liao (2004); The sage
14.10 KEY WORDS encyclopedia of Social Sciences Research Methods vol. 1 page no. 83
Canonical correlation : Measure of the strength of the overall relationships 6) Magnus Borga; Canonical Correlation. A tutotial www.cs.cmu.edu/~tom/10701-
between the linear composites (canonical variates) spll/sides/cca-totorial.pdf
for the independent and dependent variables. In
effect, it represents the bivariate correlation 7) S. Press James (2005): Applied Multivariate Analysis. Dover publication.inc.
between the two canonical variates.
Canonical correlation : It is a multivariate statistical analysis that facilitates
14.12 ANSWERS OR HINTS TO CHECK YOUR
analysis the study of interrelationships among the sets of PROGRESS
multiple dependent variables and multiple
independent variables. Check Your Progress I
Canonical cross-loadings : Correlation of each observed independent or 1) See Section 14.1
dependent variable with the opposite canonical 2) See Section 14.6
variate. For example, the independent variables are
correlated with the dependent canonical variate. 3) See Section 14.6
They can be interpreted like canonical loadings,
but with the opposite canonical variate. 14.13 EXERCISES
Canonical function : Relationship (correlation) between two linear
composites (canonical variates). Each canonical 1) Under what circumstances would you select canonical correlation analysis instead
function has two canonical variates, one for the set of multiple regressions as the appropriate statistical technique?
of dependent variables and one for the set of 2) Discuss in details the procedure for computation of the canonical correlation.
independent variables. The strength of the
relationship is given by the canonical correlation. 3) What are the limitations associated with canonical correlation analysis?

41