C

Canonical The latent variables (one per data

Canonical Correlation Analysis
variates table) computed in CCA (also
called canonical variables,
Hervé Abdi1, Vincent Guillemot2, Aida Eslami3
canonical variable scores, or
and Derek Beaton4
1 canonical factor scores). The
School of Behavioral and Brain Sciences, The
canonical variates have maximal
University of Texas at Dallas, Richardson, TX,
correlation.
USA
2 Canonical The set of coefficients of the linear
Bioinformatics and Biostatistics Hub, Institut
vectors combinations used to compute the
Pasteur (IP), C3BI, USR 3756 CNRS, Paris,
canonical variates, also called
France
3 canonical weights. Canonical
Centre for Heart Lung Innovation, University of
vectors are also sometimes called
British Columbia, Vancouver, BC, Canada
4 canonical loadings.
Rotman Research Institute, Baycrest Health
Latent A linear combination of the
Sciences, Toronto, ON, Canada
variable variables of one data table. In
general, a latent variable is
computed to satisfy some
Synonyms
predefined criterion.
Canonical analysis; Canonical variate analysis;
External factor analysis.
Definition

Canonical correlation analysis (CCA) is a statisti-

cal method whose goal is to extract the informa-
Glossary
tion common to two data tables that measure
quantitative variables on a same set of observa-
Canonical Correlation between two canonical
tions. To do so, CCA creates pairs of linear com-
correlation variates of the same pair. This is the
binations of the variables (one per table) that have
criterion optimized by CCA.
maximal correlation.
Canonical Correlation between the original
loadings variables and the canonical variates.
Sometimes used as a synonym for
canonical vectors (because these
quantities differ only by their
normalization).
# Springer Science+Business Media LLC 2018
R. Alhajj, J. Rokne (eds.), Encyclopedia of Social Network Analysis and Mining,
DOI 10.1007/978-1-4614-7163-9_110191-1
2 Canonical Correlation Analysis

Introduction Historical Background

Originally defined by Hotelling in 1935 After principal component analysis (PCA), CCA
(Hotteling 1935; Hotteling 1936, see also Bartlett is one of the oldest multivariate techniques that
1948), canonical correlation analysis (CCA) is a was first defined in 1935 by Hotelling. In addition
statistical method whose goal is to extract the of being the first method created for the statistical
information common to two data tables that mea- analysis of two data tables, CCA is also of theo-
sure quantitative variables on a same set of obser- retical interest because a very large number of
vations. To do so, CCA computes two sets of linear multivariate analytic tools are particular
linear combinations – called latent variables – cases of CCA. Like most multivariate statistical
(one for each data table) that have maximum techniques, CCA has been really feasible only
correlation. To visualize this common information with the advent of modern computers. Recent
extracted by the analysis, a convenient way is developments involve generalization of CCA to
(1) to plot the latent variables of one set against the case of more than two-tables and cross-
the other set (this creates plots akin to plots of validation approaches to select important vari-
factor scores in principal component analysis); ables and the stability and reliability of the solu-
(2) to plot the coefficients of the linear combina- tion obtained on a given sample.
tions (this creates plots akin to plotting the load-
ings in principal component analysis); and (3) to
plot the correlations between the original vari-
Canonical Correlation Analysis
ables and the latent variables (this creates correla-
tion circle” plots like in principal component
Notations
analysis).
Matrices are denoted in upper case bold letters,
CCA generalizes many standard statistical
vectors are denoted in lower case bold, and their
techniques (e.g., multiple regression, analysis of
elements are denoted in lower case italic. Matrices,
variance, discriminant analysis) and can also be
vectors, and elements from the same matrix all
declined in several related methods that address
use the same letter (e.g., A, a, a). The transpose
slightly different types of problems (e.g., different
operation is denoted by the superscript⊺, the
normalization conditions, different types of data).
inverse operation is denoted by
1. The identity
matrix is denoted I, vectors or matrices of ones are
denoted 1, and matrices or vectors of zeros are
Key Points denoted 0. When provided with a square matrix,
the diag operator gives a vector with the diagonal
CCA extracts the information common to two elements of this matrix. When provided with a
data tables measuring quantitative variables on vector, the diag operator gives a diagonal matrix
the same set of observations. For each data table, with the elements of the vector as the diagonal
CCA computes a set of linear combinations of the elements of this matrix. When provided with a
variables of this table called latent variables or square matrix, the trace operator gives the sum of
canonical variates with the constraints that a the diagonal elements of this matrix.
latent variable from one table has maximal corre- The data tables to be analyzed by CCA of, respec-
lation with one latent variable of the other table tively, size N  I and N  J, are denoted X and
and no correlation with the remaining latent vari- Y and collect two sets of, respectively, I and J
ables of the other table. The results of the analysis quantitative measurements obtained on the same
are interpreted using different types of graphical N observations. Except if mentioned otherwise,
displays that plot the latent variables and the coef- matrices X and Y are column centered and normal-
ficients of the linear combinations used to create ized, and so:
the latent variables.
Canonical Correlation Analysis 3

1⊺ X 5 0, 1⊺ Y 5 0, (1) f ⊺ f ¼ p⊺ X⊺ Xp ¼ p⊺ RX p ¼ 1 ¼ g⊺ g
¼ q⊺ Y⊺ Yq ¼ q⊺ RY q: (6)
(with 1 being a conformable vector of 1 s and 0 a
conformable vector of 0 s), and
With these notations, the maximization prob-


 lem from Eq. 5 becomes:
diag X⊺ , X 51, diag Y⊺ , Y 51: (2)


arg max f ⊺ g ¼ p⊺ Rq under the contraints that
Note that because X and Y are centered and p, q
normalized matrices, their inner products are cor- p⊺ RX p ¼ 1 ¼ q⊺ RY q:
relation matrices that are denoted:
(7)
RX ¼ X⊺ X, RY ¼ Y⊺ Y and Equivalent Optimum Criteria
⊺
(3)
R ¼ X Y: The maximization problem expressed in Eq. 5 can
also be expressed as the following equivalent
minimization problem:

Optimization Problem

arg max j j Xp
Yq22 jj ¼
p, q
In CCA, the problem is to find two latent vari-


ables, denoted f and g obtained as linear combi- arg max trace ðXp
YqÞ⊺ ðXp
YqÞ
p, q
nations of the columns of, respectively, X and Y.
The coefficients of these linear combinations are under the contraints that p⊺RXp ¼ 1 ¼ q⊺RYq:
stored, respectively, in the I  1 vector p and the (8)
J  1 vector q; and, so, we are looking for

f ¼ Xp and g ¼ Yq: (4) Lagrangian Approach

A standard way to solve the type of maximization
In CCA, we want the latent variables to have described in Eq. 7 is to use a Lagrangian approach.
maximum correlation and so we are looking for To do so we need: first to define a Lagrangian that
p and q satisfying: incorporates the constraints; next, to take the deriv-
ative of the Lagrangian respective to the unknown
d ¼ arg max fcorðf, gÞg quantities (here: p and q); and, finally, to set the
p, q derivative to zero in order to obtain the normal
8 9
> > equations. Solving the normal equations will then
< ⊺
f g =
give the values of the parameters that solve the
¼ arg max qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi : (5)
p, q > f ⊺ f  ðg ⊺ g Þ ;
: > optimization problem (i.e., for CCA: maximum
correlation).
Here the Lagrangian (denoted ℒ) that includes
The correlation between latent variables – the constraints with the two Lagrange multipliers
maximized by CCA – is called the canonical a and b is
correlation (Gittins 2012; Hotteling 1936; Horst,
1961; Mardia et al. 1980).  
ℒ ¼ p⊺ Rq
a p⊺ RX p
1
An easy way to solve the maximization prob-
 
lem of CCA is to constraint the latent variables to
b q⊺ RY q
1 : (9)
have unitary norm (because, in this case, the term
f⊺g gives the correlation between f and g). Specif- The derivatives of the Lagrangian for p and q are
ically, we require that:
4 Canonical Correlation Analysis

 
@ℒ respectively, R
1 ⊺
1
Y R RX R , associated with the
¼ Rq
2aRX p (10)
@p first eigenvalue l1 = d2, and that the maximum
correlation (i.e., the canonical correlation) is
@ℒ equal to d. Note that in order to make explicit
¼ R⊺ p
2bRY q: (11)
@q the constraints expressed in Eq. 7, the vectors
p and q are normalized (respectively) in the metric
RX and RY (i.e., p⊺RXp = 1 and q⊺RYq = 1).
The Normal Equations
Setting Eqs. 10 and 11 to zero gives the normal
equations:
Additional Pairs of Latent Variables
Rq ¼ 2aRX p (12)
After the first pair of latent variables has been
⊺ found, additional pairs of latent variables can be
R p ¼ 2bRY q: (13)
extracted. The criterion from Eqs. 5 and 7 is still
used for the subsequent pairs of latent variables,
Solution of the Normal Equations along with the requirement that the new latent
The first step to solve the normal equations is to variables are orthogonal to the previous ones.
show that a = b. This is done by premultiplying Specifically, if f‘ and g‘ denote the ‘-th pair of
Eq. 12 by p and Eq. 13 by q to obtain (using the latent variables, the orthogonality condition
constraints from Eq. 3): becomes:

p⊺ Rq ¼ 2ap⊺ RX p ¼ 2a (14) f ⊺‘ f ‘0 ¼ 0 and g⊺‘ g‘0 ¼ 0 8‘ 6¼ ‘0 : (20)

q⊺ R⊺ p 5 2bq⊺ RY q ¼ 2b (15) This orthogonality condition imposed on the

latent variables is equivalent to imposing an RX-
Equating p⊺Rq and q⊺R⊺p shows that (respectively RY-) orthogonality condition on the
2a = 2b. For convenience, in what follows we eigenvectors p and q, namely that:
set d = 2a = 2b.
Postmultiplying Eq. 12 by R
1
X and Eq. 13 by p⊺‘ RX p‘0 ¼ 0 and q⊺‘ RY q‘0 ¼ 0 8‘ 6¼ ‘0 :

1
RY gives
(21)

R
1
X Rq ¼ dp (16) For convenience, latent variables and eigen-
vectors can be stored in matrices F, G, P, and
R
1 y
Y R p ¼ dq: (17) Q. With these notations, the normalization (from
Eq. 7) and orthogonality (from Eq. 21) conditions
Replacing q (respectively p) in Eq. 16 are written as
(respectively Eq. 17) by its expression from
Eq. 17 (respectively Eq. 16) gives the following F⊺ F5I , P⊺ RX P5I (22)
two eigen-equations (see Abdi 2007b) for a
refresher about the eigen-decomposition): G⊺ G5I , Q⊺ RY Q5I: (23)

R
1
1 ⊺
X RRY R p ¼ d p
2
(18) The matrices of eigenvectors P and Q are
respectively called RX- and RY- orthogonal (the
R
1 ⊺
1
Y R RX Rq ¼ d q,
2
(19) proof of this property is given in the next section,
see Eq. 27).
which shows that p (respectively q) is the eigen- The eigen-decompositions for P and Q can
vector of the nonsymmetric matrix R
1X RRY R

1 ⊺
then be expressed in a matrix form as:
Canonical Correlation Analysis 5

R
1
1 ⊺
1 ⊺
1
X RRY R P5PL and RY R RX RQ5QL:
Solution from One Singular Value
(24)
Decomposition

The eigenvector matrices P and Q can also be

obtained from the singular value decomposition
Solution from the Eigen-Decomposition (SVD, see, e.g., Greenacre (1984); Abdi (2007a))
of Symmetric Matrices of the matrix

The matrices P and Q can also be obtained from X

L


~ Q~⊺¼
1 1

the decomposition of two symmetric matrices. For RX 2 RRY 2 ¼ PD ~‘q

d‘ p ~‘
‘
example, matrix P can be obtained from the fol-
lowing eigendecomposition ~ yP
with P ~ ⊺Q
~ ¼Q ~ ¼ I: (29)

1
RX 2 RR
1 ⊺
2 ~ ~T
1
~ ⊺P
~ ¼ I: where P,~ Q,~ and D denote (respectively) the left,
Y R RX ¼ PLP with P
(25) right singular vectors, and a diagonal matrix of the

1
1
singular values of matrix RX 2 RRY 2 . The matrices
~ ¼ R 2P
1 P and Q (containing the vectors p and q) are then
This can be shown by 1first defining P X

2 ~ computed as
and replacing P by RX P in Eq. 24 and then
simplifying:
~ ¼ R2 P ~ ¼ R2 Q ,
1 1
P X and Q Y
(30)
R
1
1 ⊺
X RRY R P 5 PL ~
1 ~
1
  P ¼ RX 2 P and Q ¼ RY 2 Q:
2~ 2~
1
1
1 ~
R
1
1 ⊺
X RRY R RX P ¼ RX PL because P ¼ RX 2 P
1
2~
1
2~
1

1 From the Eigen-Decomposition to the
R2X R
1
1 ⊺
X RRY R RX P ¼ RX RX PL
2

 1
 Singular Value Decomposition
Multiply both sides by R2X To show that p can be found from the eigen-

1
⊺
2 ~ ~
1 decompositions from Eqs. 18 and 19, we first
RX 2 RR
1
Y R RX P ¼ PL: ~ in Eq. 18. This gives:
replace p by p
(26)

1
This shows that P~ is the matrix of the eigen- R
1
1 ⊺
X RRY R RX p
2
~ ¼ d2 p, (31)

1
1
vectors of the symmetric matrix RX 2 RR
1 Y R⊺ RX 2 ,
~ ⊺P
1
which also implies that P ~ ¼ I. The eigenvectors then premultiplyinig both sides of Eq. 31 by R2X
of the asymmetric matrix R
1
1 ⊺
X RRY R are then
and simplifying gives

12 ~
recovered as P ¼ RX P . A simple substitution
shows that P is RX-orthogonal:
1
1 1
R2X R
1
1 ⊺
X RRY R RX p
2
~ ¼ d2 R2X p

1
1
~ ⊺ R
2 RX R
2 P
P⊺ R X P ¼ P ~ ⊺P
~¼P
1
~ ¼ I:
1
(27) , RX 2 RR
1 ⊺
Y R RX p
2
~ ¼ d2 p
~: (32)
X X

A similar derivation shows that Q can be A similar argument shows that

obtained from the eigen-decomposition

1
1
RY 2 R⊺ R
1
X RRY q
2
~ ¼ d2 q
~: (33)

1
RY 2 R⊺ R
1

12 ~ Q~⊺
X RRY ¼ QL (28)
Combining Eqs. 32 and 33 shows that p ~ and q
~
~ Q ⊺
~ ¼ I, where Q ¼
1 ~ are left and right singular vectors of the matrix
with Q RY 2 Q.
1
1
RX 2 RRY 2 .
6 Canonical Correlation Analysis

An Example: The Colors and Grapes of (Fig. 6) roughly orders the wines according to
Wines their concentration of red pigment (i.e., white,
rosé, and red; similar plots using grape varietal
To illustrate CCA we use the data set presented in or origin of the wines did not show any interesting
Table 1. These data describe thirty-six red, rosé, or patterns and are therefore not shown).
white wines produced in three different countries To understand the contribution of the variables
(Chili, Canada, and USA) using several different of X and Y to the latent variables, two types of
varietal of grapes. These wines are described by plots are used: (1) a plot of the correlation between
two different sets of variables. The first set of latent variables and original variables and (2) a
variables (i.e., matrix X) describes the objective plot of the loadings of the variables. Figure 7
properties of the wines: Price, Acidity, Alcohol (respectively Figure 8) shows the correlation
content, Sugar, and Tannin (in what follows we between the original variables (both X and Y)
capitalize these descriptors). The second set of and respectively F (i.e., the latent variables from
variables (i.e., matrix Y) describes the subjective X) and G (i.e., the latent variables from Y). Figs. 9
properties of the wines as evaluated by a profes- and 10 display the loadings for, respectively,
sional wine taster and consists in ratings on a nine- X and Y (i.e., matrices P and Q) for the first two
point rating scale of eight aspects of taste: fruity, dimensions of the analysis. Together these figures
floral, vegetal, spicy, woody, sweet, astringent, indicate that the first dimension reflects the nega-
acidic, plus an overall evaluation of the hedonic tive correlation between Alcohol (from X) and the
aspect of the wine (i.e., how much the taster liked subjective hedonic evaluation of the wines (from
the wine). Y), whereas the second dimension combines
The analysis of this example was performed (low) Alcohol, (high) Acidity, and (high) Sugar
using the statistical programming language R and (from X) to reflect their correlations with the sub-
is available to download from https://github.com/ jective variables astringent and hedonic. Figs. 7
vguillemot/2Tables. and 8 show very similar pictures (because the
Figures 1, 2, and 3 show heatmaps of the latent variables are very correlated) and this sug-
correlation matrices RX, RY, and R. As shown gests that the first pair of latent variables opposes
in Fig. 3 (for matrix R), the objective variables “bitterness” (i.e., astringent, alcohol, etc.) to
alcohol and tannin are positively correlated with sweetness, whereas the second pair of latent vari-
the perceived qualities of astringent and woody; ables opposes bitterness (from astringent) to the
by contrast, the perceived hedonic aspect of wine “burning” effect of Alcohol.
is negatively correlated with alcohol, tannin (and
price, so our taster liked inexpensive wines) and
positively correlated with the sugar content of the Variations over CCA
wines. Unsurprisingly, the objective amount of
Sugar is correlated with the perceived quality By imposing slightly different orthogonality con-
sweet. ditions than the ones described in Eqs. 14 and 15,
The CCA of these data found five pairs of different (but related) alternative methods can be
latent variables [in general CCA will find a max- defined to analyze two data tables.
imum of min(I, J) pairs of latent variables]. The
values of the canonical correlations are shown in Inter-Battery Analysis (IBA) Et Alia
Fig. 4. The first and second canonical correlations The oldest alternative – originally proposed by
are very high (.98 and.85, see Table 2), and so we Tucker in 1958 – called inter-battery analysis
will only consider them here. As shown in Figs. 5 (IBA) (Tucker 1958), is also known under a vari-
and 6, the latent variables extracted by the analysis ety of different names such as coinertia analysis
are very sensitive to “the color” of the wines: The (Dolédec and Chessel 1994), partial least square
first pair of latent variables (Fig. 5) isolates the red SVD (PLSVD) (Bookstein 1994), partial least
wines, whereas the second pair of latent variables square correlation (PLSC) (Krishnan et al. 2010;
Canonical Correlation Analysis, Table 1 An example for CCA. Thirty-six wines are described by two sets of variables: objective descriptors (Matrix X) and subjective
descriptors (Matrix Y)
Matrix X: Objective Matrix Y: Subjective
Wine Origin Color Varietal Price Acidity Alcohol Sugar Tannin Fruity Floral Vegetal Spicy Woody Sweet Astringent Acidic Hedonic
CH01 Chili Red Merlot 11 5.33 13.80 2.75 559 6 2 1 4 5 3 5 4 2
Canonical Correlation Analysis

CH02 Chili Red Cabernet 5 5.14 13.90 2.41 672 5 3 2 3 4 2 6 3 2

CH03 Chili Red Shiraz 7 5.16 14.30 2.20 455 7 1 2 6 5 3 4 2 2
CH04 Chili red Pinot 16 4.37 13.50 3.00 348 5 3 2 2 4 1 3 4 4
CH05 Chili White Chardonnay 14 4.34 13.30 2.61 46 5 4 1 3 4 2 1 4 6
CH06 Chili White Sauvignon 8 6.60 13.30 3.17 54 7 5 6 1 1 4 1 5 8
CH07 Chili White Riesling 9 7.70 12.30 2.15 42 6 7 2 2 2 3 1 6 9
CH08 Chili White Gewurzt 11 6.70 12.50 2.51 51 5 8 2 1 1 4 1 4 9
CH09 Chili Rose Malbec 4 6.50 13.00 7.24 84 8 4 3 2 2 6 2 3 8
CH10 Chili Rose Cabernet 3 4.39 12.00 4.50 90 6 3 2 1 1 5 2 3 8
CH11 Chili Rose Pinot 6 4.89 12.00 6.37 76 7 2 1 1 1 4 1 4 9
CH12 Chili Rose Syrah 5 5.90 13.50 4.20 80 8 4 1 3 2 5 2 3 7
CA01 Canada Red Merlot 20 7.42 14.90 2.10 483 5 3 2 3 4 3 4 4 3
CA02 Canada Red Cabernet 15 7.35 14.50 1.90 698 6 3 2 2 5 2 5 4 2
CA03 Canada Red Shiraz 20 7.50 14.50 1.50 413 6 2 3 4 3 3 5 1 2
CA04 Canada Red Pinot 25 5.70 13.30 1.70 320 4 2 3 1 3 2 4 4 4
CA05 Canada White Chardonnay 20 6.00 13.50 3.00 35 4 3 2 1 3 2 2 3 5
CA06 Canada White Sauvignon 15 7.50 12.00 3.50 40 8 4 3 2 1 3 1 4 8
CA07 Canada White Riesling 15 7.00 11.90 3.40 48 7 5 1 1 3 3 1 7 8
CA08 Canada White Gewurzt 18 6.30 13.90 2.80 39 6 5 2 2 2 3 2 5 6
CA09 Canada Rose Malbec 8 5.90 12.00 5.50 90 6 3 3 3 2 4 2 4 8
CA10 Canada Rose Cabernet 6 5.60 12.50 4.00 85 5 4 1 3 2 4 2 4 7
CA11 Canada Rose Pinot 9 6.20 13.00 6.00 75 5 3 2 1 2 3 2 3 7
CA12 Canada Rose Syrah 9 5.80 13.00 3.50 83 7 3 2 3 3 4 1 4 7
US01 USA Red Merlot 25 6.00 13.60 3.50 578 7 2 2 5 6 3 4 3 2
US02 USA Red Cabernet 15 6.50 14.60 3.50 710 8 3 1 4 5 3 5 3 2
7

(continued)
 8

Canonical Correlation Analysis, Table 1 (continued)

Matrix X: Objective Matrix Y: Subjective
US03 USA Red Shiraz 25 5.30 13.90 1.99 610 8 2 3 7 6 4 5 3 1
US04 USA Red Pinot 28 6.10 14.00 0.00 340 6 3 2 2 5 2 4 4 2
US05 USA White Chardonnay 15 7.20 13.30 1.10 41 6 4 2 3 6 3 2 4 5
US06 USA White Sauvignon 8 7.20 13.50 1.00 50 6 5 5 1 2 4 2 4 7
US07 USA White Riesling 10 8.60 12.00 1.65 47 5 5 3 2 2 4 2 5 8
US08 USA White Gewurzt 20 9.60 12.00 0.00 45 6 6 3 2 2 4 2 3 8
US09 USA Rose Malbec 3 6.20 12.50 4.00 84 8 2 1 4 3 5 2 4 7
US10 USA Rose Cabernet 4 5.71 12.50 4.30 93 8 3 3 3 2 6 2 3 8
US11 USA Rose Pinot 8 5.40 13.00 3.10 79 6 1 1 2 3 4 1 3 6
US12 USA Rose Syrah 6 6.50 13.50 3.00 89 9 3 2 5 4 3 2 3 5
Canonical Correlation Analysis
Canonical Correlation Analysis 9

Abdi and Williams 2013), singular value decom- (as described in Eqs. 5 and 7) the latent variables
position of the covariance between two fields are required to have maximum covariance. So we
(Bretherton et al. 1992), maximum covariance are looking for vectors p and q satisfying:
analysis (von Storch and Zwiers 2002), or even,
recently, “multivariate genotype-phenotype” d ¼ arg max fcovðf, gÞg
(MGP) analysis (Mitteroecker et al. 2016). It is p, q


particularly popular in brain-imaging and related ¼ arg max f ⊺g ¼ p⊺ Rq : (34)
domains (McIntosh et al. 1996). In IBA (like in p, q

CCA) the latent variables are obtained as linear

combinations of the variables of X and Y but An easy way to solve this problem is to impose
instead of having maximum correlation the constraints that p and q have unitary norm,
namely that:

p⊺ p ¼ 1 ¼ q⊺ q: (35)

A derivation similar to the one used for CCA

1
Sugar
shows that the coefficients P and Q and the opti-
mal linear combination can be directly obtained
Acidity from the singular decomposition of the correlation
0.5
Price
matrix R as:

Alcohol X
L
0
R ¼ PDQ⊺ ¼ d‘ p‘ q⊺‘
Tannin ‘ (36)
⊺ ⊺
P P ¼ Q Q ¼ I:
Sugar

Acidity

Price

Alcohol

Tannin

with
−0.5

where P, Q, and D are (respectively) matrices

Canonical Correlation Analysis, Fig. 1 Heatmap of containing the left, right singular vectors, and dia-
correlation matrix RX (i.e., between the variables of gonal matrix of the singular values of R.
matrix X)

Canonical Correlation
Analysis, Fig. 2 Heatmap
of correlation matrix RY
(i.e., between the variables 1
of matrix Y) spicy
woody
0.5
astringent
hedonic 0
floral
acidic −0.5
vegetal
fruity
sweet
spicy
woody
astringent
hedonic
floral
acidic
vegetal
fruity
sweet
10 Canonical Correlation Analysis

Canonical Correlation
Analysis, Fig. 3 Heatmap
of correlation matrix R (i.e.,
between the variables of
matrices X and Y)
Price

Alcohol 0.5

Tannin
0
Acidity

Sugar −0.5

spicy

woody

astringent

sweet

hedonic

fruity

vegetal

floral

acidic
Canonical Correlation 1.00
Analysis, Fig. 4 Barplot
of the canonical correlations
(i.e., correlations between
Canonical Correlations

0.75
pairs of latent variables for a
given dimension)

0.50

0.25

0.00
1 2 3 4 5
Dimensions

Because IBA maximizes the covariance Asymmetric Two-Table Analysis

between latent variables (instead of the correlation
for CCA), it does not require the inversion of CCA and IBA are symmetric techniques because
matrices RX and RY and therefore, IBA can be the results of the analysis will be unchanged
used with rank deficient or badly conditioned (mutatis mutandis) if X and Y are permuted.
matrices and, in particular, when the number of Other related techniques differentiate the roles of
observations is smaller than the number of vari- X and Y and treat X as a matrix of predictor vari-
ables (a configuration sometimes called N << P). ables and Y as a matrix of dependent variables
This makes IBA a robust technique popular for (i.e., to be “explained” or “predicted” by X).
domains using Big or “Wide” Data such as, for Among these techniques, the two most well-
example, brain imaging (Krishnan et al. 2010; known approaches are redundancy analysis and
Grellmann et al. 2015) or genomics (Beaton partial least square regression (see, also, Takane
et al. 2016). 2013, for alternative methods and review).
 Canonical Correlation Analysis

Canonical Correlation Analysis, Table 2 An example for CCA. Canonical correlations (d‘), loadings for matrices X (objective descriptors, loading matrix P) and Y (subjective
descriptors loading matrix Q) for the five dimensions extracted by CCA
Matrix P: Objective Matrix Q: Subjective
Dimension d‘ Price Acidity Alcohol Sugar Tannin Fruity Floral Vegetal Spicy Woody Sweet Astringent Acidic Hedonic
1 .98 0.026
0.088 0.489
0.134 0.002 0.000 0.150 0.030
0.068
0.109 0.082
0.095 0.059
0.586
2 .85
0.062 0.306
0.927 0.188 0.006 0.274
0.247
0.150 0.114 0.307
0.285 1.688 0.085 1.179
3 .65
0.024 0.678
0.243
0.288 0.001 0.044 0.605 0.366 0.024 0.390
0.133 0.465
0.083 0.327
4 .48
0.057 0.574 1.382 0.428
0.002 0.565 0.368
0.076
0.427
0.505 0.584
0.531
0.423
0.870
5 .22 0.157 0.360
0.131 0.627
0.000 0.744
0.205 0.087
0.877 0.290
0.568 0.296 0.065 0.185
11
12 Canonical Correlation Analysis

Canonical Correlation
Analysis, Fig. 5 CCA.
Latent variables: First latent
variable from X (1st LV

1st LV (Subjective properties)

objective properties) 1
vs. First latent variable from
Y (first LV subjective
properties). One point
wine
represents one wine. Wines
red
are colored according to 0 rose
their types (i.e., red, rosé, or
white
white). Red wines are well
separated from the other
wines
−1

−1 0 1
1st LV (Objective properties)

Canonical Correlation 2
Analysis, Fig. 6 CCA.
Latent variables: Second
2nd LV (Subjective properties)

latent variable from

X (second LV objective 1
properties) vs. Second
latent variable from Y (2nd
LV subjective properties). wine
One point represents one red
0
wine. Wines are colored rose
white
according to their types
(i.e., red, rosé, or white)
−1

−2

−2 −1 0 1 2
2nd LV (Objective properties)

Redundancy Analysis predictors) are required to be unit normalized

Redundancy analysis (RA, (Van Den Wollenberg (just like in IBA). Specifically, RA is solving the
1977)) – originally developed under the name of following optimization problem:
principal component analysis of instrumental var-
iables by Rao in 1964 (Rao 1964) and simulta- d ¼ argmax fcov ðf, gÞg
neous prediction method (by Fortier 1966) – can p ,q


be seen as a technique intermediate between CCA ¼ argmax f ⊺ g ¼ p⊺ Rq
and IBA. In RA, the vectors stored in P (i.e., the p, q

loadings from X, the predictors) are required to be with p⊺ RX p ¼ 1 ¼ q⊺ q: (37)

RX-normalized (just like in CCA) but the vectors
stored in Q (i.e., the loadings from Y, the
Canonical Correlation Analysis 13

Canonical Correlation 1.0

Analysis,
Fig. 7 Correlation circle

2nd LV (Objective properties)

with latent variables from
matrix X 0.5 Ta
Tannin
Sugar
sweet e
astringent
fruity
spicy B
Ac
Acidity
0.0 i
acidic od
woody a Obj
hedonic
a Subj
floral g
vegetal o
Alcohol

Price
−0.5

−1.0

−1.0 −0.5 0.0 0.5 1.0

1st LV (Objective properties)

Canonical Correlation 1.0

Analysis,
Fig. 8 Correlation circle
2nd LV (Subjective properties)

with latent variables from

matrix Y 0.5 ng
g
astringent
Sugar n
spicy Tannin
sweet fruity
B
d
Acidity
0.0 idi
d
acidic o
woody a Obj
hedonic a Subj
g
vegetal Alcoholl
floral
Price
−0.5

−1.0

−1.0 −0.5 0.0 0.5 1.0

1st LV (Subjective properties)

RA can be interpreted as searching for the best latent variable from X) is used, in a regression
predictive linear combinations of the columns of step, to predict Y. After the first latent variable has
X or, equivalently, RA is searching for the sub- been used, its effect is partialled from X and Y and
space of X where the projection of Y has the the procedure is re-iterated to find subsequent
largest variance. latent variables and loadings.

Partial Least Square Regression

Partial Least Square Regression (PLSR) – a tech- Particular Cases
nique tailored to cope with multicollinearity of the
predictors in a regression framework – (see, e.g., Noncentered CCA: Correspondence Analysis
Tenenhaus (1998); Abdi (2010) for reviews) starts An interesting particular case of noncentered
just like IBA and extracts a first pair of latent CCA concerns the case of group matrices. In a
variables with maximal covariance, then f (the group matrix, the rows represent observations
14 Canonical Correlation Analysis

c
Acidity
astringent
en
n
a
Sugar 1.5

2nd Loadings (Subjective properties)

2nd Loadings (Objective properties)

0.0 hedonic
n
Tannin
Price
r 1.0

0.5
−0.5 woody
fruity
ru
spicy

0.0 acidic
a d
vegetal
e
floral
Alcohol
sweet
e
0.0 0.2 0.4 −0.50 −0.25 0.00 0.25
1st Loadings 1st Loadings
(Objective properties) (Subjective properties)

Canonical Correlation Analysis, Fig. 9 Loadings of the Canonical Correlation Analysis, Fig. 10 Loadings of
second LV versus the first LV for matrix X the second LV versus the first LV for matrix Y

(just like in plain CCA) and the columns represent X with itself is equivalent to multiple correspon-
a set of exclusive groups (i.e., an observation dence analysis.
belongs to one and only one group). The group
assigned to an observation has a value of 1 for the
row representing this observation and all the other
Some Other Particular Cases of CCA
columns for this observation (representing the
groups not assigned to this observation) will
CCA is a very general method and so a very large
have a value of 0. When both X and Y are non-
number of methods are particular cases of CCA
centered and nonnormalized group matrices, then
(Abdi 2003). For example, when Y has only one
the CCA of these matrices will give correspon-
column, CCA becomes (simple and multiple) lin-
dence analysis – a technique developed to analyze
ear regression. If X is a group matrix and Y stores
contingency tables (see entry on correspondence
one quantitative variable, CCA becomes analysis
analysis and, e.g., Greenacre (1984)). When
of variance. If Y is a group matrix, CCA becomes
X and Y are composed of the concatenation of
discriminant analysis. This versatility of CCA
several noncentered and nonnormalized group
makes it of particular theoretical interest.
matrices, the CCA of these two tables will be
equivalent to partial least square correspondence
analysis (PLSCA (Beaton et al. 2016)) – a tech-
nique originally developed to analyze the infor- Key Applications
mation shared by two tables storing qualitative
data. In the particular case when X is composed CCA and its derivatives – or variations thereof –
of the concatenation of several noncentered and are used when the analytic problem is to relate two
nonnormalized group matrices, the CCA of data tables and this makes these techniques ubiq-
uitous in almost any domains of inquiry from
Canonical Correlation Analysis 15

marketing to brain imaging and network analysis Abdi, H., and Williams, L. J. (2013). Partial least squares
(see Abdi et al., 2016, for examples). methods: partial least squares correlation and partial
least square regression. Computational Toxicology, II,
549–579.
Abdi H, Vinzi VE, Russolillo G, Saporta G, Trinchera
L (eds) (2016) The multiple facets of partial least
Future Directions squares methods. Springer Verlag, New York
Bartlett MS (1948) External and internal factor analysis. Br
CCA is still a domain of intense research with J Psychol 1:73–81
future developments likely to be concerned with Beaton D, Dunlop J, Abdi H, ADNI (2016) Partial least
squares correspondence analysis: a framework to
multi-table extensions (e.g., Horst 1961;
simultaneously analyze behavioral and genetic data.
Tenenhaus et al. 2014), “robustification,” and Psychol Methods 21:621–651
sparsification (Witten et al. 2009). All these Bookstein F (1994) Partial least squares: a dose response
approaches will make CCA and its related tech- model for measurement in the behavioral and brain
sciences. Psycoloquy 5(23)
niques even more suitable for the analysis of very
Bretherton CS, Smith C, Wallace JM (1992) An
large data sets that are becoming prevalent in intercomparison of methods for finding coupled pat-
analytics. terns in climate data. J Clim 5:541–560
Dolédec S, Chessel D (1994) Co-inertia analysis: an alter-
native method for studying species-environment rela-
tionships. Freshw Biol 31:277–294
Cross-References Fortier JJ (1966) Simultaneous linear prediction.
Psychometrika 31:369–381
Gittins R (2012) Canonical analysis: a review with appli-
▶ Barycentric Discriminant Analysis cations in ecology. Springer Verlag, New York
▶ Correspondence analysis Greenacre MJ (1984) Theory and applications of corre-
▶ Eigenvalues, Singular Value Decomposition spondence analysis. Academic Press, London
Grellmann C, Bitzer S, Neumann J, Westlye LT,
▶ Iterative Methods for Eigenvalues/ Andreassen OA, Villringer A, Horstmann A (2015)
Eigenvectors Comparison of variants of canonical correlation analy-
▶ Least Squares sis and partial least squares for combined analysis of
▶ Matrix Algebra, Basics of MRI and genetic data. NeuroImage 107:289–310
Horst P (1961) Relations among m sets of measures.
▶ Matrix Decomposition Psychometrika 26:129–149
▶ Principal Component Analysis Hotteling H (1935) The most predicable criterion. J Educ
▶ Regression Analysis Psychol 26:139–142
▶ Spectral Analysis Hotteling H (1936) Relation between two sets of variates.
Biometrika 28:321–377
Krishnan A, Williams LJ, McIntosh AR, Abdi H (2010)
Partial least squares (PLS) methods for neuroimaging: a
tutorial and review. NeuroImage 56:455–475
References Mardia KV, Kent JT, Bibby JM (1980) Multivariate anal-
ysis. Academic Press, London
Abdi H (2003) Multivariate analysis. In: Lewis-Beck M, McIntosh AR, Bookstein FL, Haxby JV, Grady CL
Bryman A, Futing T (eds) Encyclopedia for research (1996) Spatial pattern analysis of functional brain
methods for the social sciences. Sage, Thousand Oaks, images using partial least squares. NeuroImage
pp 699–702 3:143–157
Abdi H (2007a) Singular value decomposition (SVD) and Mitteroecker P, Cheverud JM, Pavlicev M (2016) Multi-
generalized singular value decomposition (GSVD). In: variate analysis of genotype-phenotype association.
Salkind NJ (ed) Encyclopedia of measurement and Genetics. doi:10.1534/genetics.115.181339
statistics. Sage, Thousand Oaks, pp 907–912 Rao CR (1964) The use and interpretation of principal
Abdi H (2007b) Eigen-decomposition: eigenvalues and component analysis in applied research. Sankhyā
eigenvectors. In: Salkind NJ (ed) Encyclopedia of mea- A26:329–358
surement and statistics. Sage, Thousand Oaks, von Storch H, Zwiers FW (2002) Statistical analysis in
pp 304–308 climate research. Cambridge University Press,
Abdi H (2010) Partial least squares regression and projec- Cambridge
tion on latent structure regression (PLS regression). Takane Y (2013) Constrained principal component analy-
Wiley Interdisciplinary Reviews: Computational Sta- sis and related techniques. CRC Press, Boca Raton
tistics 2:97–106
16 Canonical Correlation Analysis

Tenenhaus M (1998) La régression PLS: théorie et pra- Van Den Wollenberg AL (1977) Redundancy analysis an
tique. Editions Technip, Paris alternative for canonical correlation analysis.
Tenenhaus A, Philippe C, Guillemot V, Le Cao KA, Grill J, Psychometrika 42:207–219
Frouin V (2014) Variable selection for generalized Witten DM, Tibshirani R, Hastie T (2009) A penalized
canonical correlation analysis. Biostatistics matrix decomposition, with applications to sparse prin-
15:569–583 cipal components and canonical correlation analysis.
Tucker LR (1958) An inter-battery method of factor anal- Biostatistics 10:515–534
ysis. Psychometrika 23:111–136