What is factor analysis?

Factor analysis is the practice of condensing many variables into just a few, so that your
research data is easier to work with.
For example, a retail business trying to understand customer buying behaviours might consider
variables such as ‘did the product meet your expectations?’, ‘how would you rate the value for
money?’ and ‘did you find the product easily?’. Factor analysis can help condense these
variables into a single factor, such as ‘customer purchase satisfaction’.
Factor analysis is also sometimes called “dimension reduction”: you can reduce the
“dimensions” of your data into one or more “super-variables,” also known as unobserved
variables or latent variables. This process involves creating a factor model and often yields a
factor matrix that organises the relationship between observed variables and the factors they’re
associated with.
What is a factor?
In the context of factor analysis, a factor is a hidden or underlying variable that we infer from
a set of directly measurable variables.
Take ‘customer purchase satisfaction’ as an example again. This isn’t a variable you can
directly ask a customer to rate, but it can be determined from the responses to correlated
questions like ‘Did the product meet your expectations?’, ‘How would you rate the value for
money?’ and ‘did you find the product easily?’.
While not directly observable, factors are essential for providing a clearer, more streamlined
understanding of data. They enable us to capture the essence of our data’s complexity, making
it simpler and more manageable to work with, and without losing lots of information.
Objective of factor analysis
1. To understand how many factors are needed to explain common themes amongst a
given set of variables.
2. To determine the extent to which each variable in the dataset is associated with a
common theme or factor.
3. To provide an interpretation of the common factors in the dataset.
4. To determine the degree to which each observed data point represents each theme or
factor.
When to use factor analysis
Factor analysis is a powerful tool when you want to simplify complex data, find hidden
patterns, and set the stage for deeper, more focused analysis. It’s typically used when you’re
dealing with a large number of interconnected variables, and you want to understand the
underlying structure or patterns within this data. It’s particularly useful when you suspect that
these observed variables could be influenced by some hidden factors.

For example, consider a business that has collected extensive customer feedback through
surveys. The survey covers a wide range of questions about product quality, pricing, customer
service and more. This huge volume of data can be overwhelming, and this is where factor
analysis comes in. It can help condense these numerous variables into a few meaningful factors,
such as ‘product satisfaction’, ‘customer service experience’ and ‘value for money’.
Factor analysis doesn’t operate in isolation – it’s often used as a step for further analysis. For
example, once you’ve identified key factors through factor analysis, you might then proceed to
a cluster analysis – a method that groups your customers based on their responses to these
factors. The result is a clearer understanding of different customer segments, which can then
guide targeted marketing and product development strategies. By combining factor analysis
with other methodologies, you can not only make sense of your data but also gain valuable
insights to drive your business decisions.

Factor analysis assumptions

Factor analysis relies on several assumptions for accurate results. Violating these assumptions
may lead to factors that are hard to interpret or misleading.

• Linear relationships between variables This ensures that changes in the values of
your variables are consistent.
• Sufficient variables for each factor Because if only a few variables represent a factor,
it might not be identified accurately.
• Adequate sample size The larger the ratio of cases (respondents, for instance) to
variables, the more reliable the analysis.
• No perfect multicollinearity and singularity No variable is a perfect linear
combination of other variables, and no variable is a duplicate of another.
• Relevance of the variables There should be some correlation between variables to
make a factor analysis feasible.

Features of factor analysis

Some of the methods used to extract the factors could include:

• Principal component analysis (PCA) This is the most commonly used method. The
first factor is extracted by determining the maximum variance. This variance is then
removed and is replaced by the factor. The second factor is then determined by the next
highest variance, and the process continues until there are no more variances.
 • Common factor analysis In this method, the factors are extracted from commonly
occurring variances and do not include the unique variances of all the variables.
• Image factoring Based on the correlation matrix, this process uses predicted variables
using the OLS regression method.

Once the factors are extracted, the questionnaire’s score is assumed to be related to the factors
in a linear manner. The margin of error is also taken into consideration, along with all the
factors to the equation.

Types of factor analysis

There are two main factor analysis methods: exploratory and confirmatory.

1. Exploratory Factor Analysis: In exploratory factor analysis, the researcher does not
make any assumptions about prior relationships between factors. In this method, any
variable can be related to any factor. This helps identify complex relationships among
variables and group them based on common factors.
2. Confirmatory Factor Analysis: The confirmatory factor analysis, on the other hand,
assumes that variables are related to specific factors and uses pre-established theory to
confirm its expectations of the model.

How to perform factor analysis

Performing a factor analysis involves a series of steps, often facilitated by statistical software
packages like SPSS, Stata and the R programming language. Here’s a simplified overview of
the process.

• Prepare your data Start with a dataset where each row represents a case (for example,
a survey respondent), and each column is a variable you’re interested in. Ensure your
data meets the assumptions necessary for factor analysis.
• Create an initial hypothesis If you have a theory about the underlying factors and their
relationships with your variables, make a note of this. This hypothesis can guide your
analysis, but keep in mind that the beauty of factor analysis is its ability to uncover
unexpected relationships.
• Choose the type of factor analysis The most common type is exploratory factor
analysis, which is used when you’re not sure what to expect. If you have a specific
hypothesis about the factors, you might use confirmatory factor analysis.
 • Form your correlation matrix After you’ve chosen the type of factor analysis, you’ll
need to create the correlation matrix of your variables. This matrix, which shows the
correlation coefficients between each pair of variables, forms the basis for the extraction
of factors. This is a key step in building your factor analysis model.
• Decide on the extraction method Principal component analysis is the most commonly
used extraction method.
• Determine the number of factors Various criteria can be used here, such as Kaiser’s
criterion or scree plot method. The choice depends on your data and your goals.
• Interpret and validate your results Each factor will be associated with a set of your
original variables, so label each factor based on how you interpret these associations.
These labels should represent the underlying concept that ties the associated variables
together.
Validation can be done through a variety of methods, like splitting your data in half and
checking if both halves produce the same factors.

Let’s try to understand it with an example

You conduct a survey and collect responses about people’s anxiety about using SPSS. Do all
these items actually measure what we call “SPSS Anxiety”? Let’s proceed with our
hypothetical example of the survey which termed as the SPSS Anxiety Questionnaire. For
simplicity, we will use the so-called “SAQ-8” which consists of the first eight items in
the SAQ. The SAQ-8 consists of the following questions:

1. Statistics makes me cry

2. My friends will think I’m stupid for not being able to cope with SPSS
3. Standard deviations excite me
4. I dream that Pearson is attacking me with correlation coefficients
5. I don’t understand statistics
6. I have little experience of computers
7. All computers hate me
8. I have never been good at mathematics

Pearson Correlation of the SAQ-8

From this table, we can see that most items have some correlation with each other. Due to
relatively high correlations among items, this would be a good candidate for factor analysis.
Since the goal of factor analysis is to model the interrelationships among items, we focus
primarily on the variance and covariance rather than the mean. Factor analysis assumes that
variance can be partitioned into two types of variance: common and unique.

• Common variance is the amount of variance that is shared among a set of items. Items
that are highly correlated will share a lot of variance. Common variance that ranges
between 0 and 1. Values closer to 1 suggest that extracted factors explain more of the
variance of an individual item.
• Unique variance is any portion of variance that’s not common. There are two parts:
There are two types:
1. Specific variance: is a variance that is specific to a particular item (e.g., Item 4 “All
computers hate me” may have variance that is attributable to anxiety about computers
in addition to anxiety about SPSS).
2. Error variance: comes from errors of measurement and basically anything
unexplained by common or specific variance (e.g., the person got a call from her
babysitter that her two-year old son ate her favorite lipstick).

The total variance is made up of common variance and unique variance, and unique variance
is composed of specific and error variance. If the total variance is 1, then the communality
is h2 and the unique variance is 1−h2. Let’s take a look at how the partition of variance applies
to the SAQ-8 factor model.
Here you see that SPSS Anxiety makes up the common variance for all eight items, but within
each item, there is specific variance and error variance. Take the example of Item 7 “All
computers hate me”. Although SPSS Anxiety explains some of this variance, there may be
systematic factors such as technophobia and non-systemic factors that can’t be explained by
either SPSS anxiety or technophobia, such as getting a speeding ticket right before coming to
the survey center (error of measurement).

Now after the partitioning of variance, we can move on to performing our first factor analysis.

Performing Factor Analysis

As a data analyst, the goal of a factor analysis is to reduce the number of variables to explain
and interpret the results. This can be accomplished in two steps:

1. factor extraction
2. factor rotation

Factor extraction involves making a choice about the type of model as well the number of
factors to extract. Factor rotation comes after the factors are extracted, with the goal of
achieving a simple structure in order to improve interpretability.

Extracting Factors
There are two approaches to factor extraction which stems from different approaches to
variance partitioning: a) principal components analysis and b) common factor analysis.

a) Principal Components Analysis

Unlike factor analysis, principal components analysis or PCA makes the assumption
that there is no unique variance, the total variance is equal to the common variance.
Recall that variance can be partitioned into common and unique variance. If there is no
unique variance then common variance takes up total variance (see figure below).
Additionally, if the total variance is 1, then the common variance is equal to the
communality.

Running a PCA with 8 components in SPSS

The goal of a PCA is to replicate the correlation matrix using a set of components that are fewer
in number and linear combinations of the original set of items.

First, go to Analyze – Dimension Reduction – Factor. Move all the observed variables over the
Variables: box to be analyze.
Under Extraction – Method, pick Principal components and make sure to Analyze the
Correlation matrix. We also request the Unrotated factor solution and the Scree plot. Under
Extract, choose Fixed number of factors, and under Factor to extract enter 8. We also bumped
up the Maximum Iterations of Convergence to 100.

Before we get into the SPSS output, let’s understand a few things about eigenvalues and
eigenvectors.

Eigenvalues represent the total amount of variance that can be explained by a given principal
component. They can be positive or negative in theory, but in practice they explain variance
which is always positive.

• If eigenvalues are greater than zero, then it’s a good sign.

• Since variance cannot be negative, negative eigenvalues imply the model is ill-
conditioned.
 • Eigenvalues close to zero imply there is item multicollinearity, since all the variance
can be taken up by the first component.

Eigenvalues are also the sum of squared component loadings across all items for each
component, which represent the amount of variance in each item that can be explained by the
principal component.

Eigenvectors represent a weight for each eigenvalue. The eigenvector times the square root of
the eigenvalue gives the component loadings which can be interpreted as the correlation of
each item with the principal component. For this particular PCA of the SAQ-8, the eigenvector
associated with Item 1 on the first component is 0.377, and the eigenvalue of Item 1 is 3.057.
We can calculate the first component as

(0.377) √3.057=0.659

In this case, we can say that the correlation of the first item with the first component is 0.659.

Component Matrix

The components can be interpreted as the correlation of each item with the component. Each
item has a loading corresponding to each of the 8 components. For example, Item 1 is
correlated 0.659 with the first component, 0.136 with the second component and −0.398 with
the third, and so on.

The square of each loading represents the proportion of variance (think of it as an R2 statistic)
explained by a particular component. For Item 1, (0.659)2=0.434 or 43.4% of its variance is
explained by the first component. Subsequently, (0.136)2=0.018 or 1.8% of the variance in
Item 1 is explained by the second component. The total variance explained by both components
is thus 43.4%+1.8%=45.2%. If you keep going on adding the squared loadings cumulatively
down the components, you find that it sums to 1 or 100%. This is also known as
the communality, and in a PCA the communality for each item is equal to the total variance.
Summing the squared component loadings across the components (columns) gives you the
communality estimates for each item, and summing each squared loading down the items
(rows) gives you the eigenvalue for each component. For example, to obtain the first
eigenvalue we calculate:

(0.659)2+(−.300)2+(−0.653)2+(0.720)2+(0.650)2+(0.572)2+(0.718)2+(0.568)2=3.057

You will get eight eigenvalues for eight components, which leads us to the next table.

Total Variance Explained in the 8-component PCA

Recall that the eigenvalue represents the total amount of variance that can be explained by a
given principal component. Starting from the first component, each subsequent component is
obtained from partialling out the previous component. Therefore, the first component explains
the most variance, and the last component explains the least.

Looking at the Total Variance Explained table, you will get the total variance explained by
each component. For example, Component 1 is 3.057, or (3.057/8)%=38.21% of the total
variance. Because we extracted the same number of components as the number of items, the
Initial Eigenvalues column is the same as the Extraction Sums of Squared Loadings column.

Choosing the number of components to extract

Since the goal of running a PCA is to reduce our set of variables down, it would useful to have
a criterion for selecting the optimal number of components that are of course smaller than the
total number of items. One criterion is the choose components that have eigenvalues greater
than 1. Under the Total Variance Explained table, we see the first two components have an
eigenvalue greater than 1. This can be confirmed by the Scree Plot which plots the eigenvalue
(total variance explained) by the component number.

The first component will always have the highest of the total variance and the last component
will always have the least, but where do we see the largest drop? If you look at Component 2,
you will see an “elbow” joint. This is the marking point where it’s perhaps not too beneficial
to continue further component extraction.
There are some conflicting definitions of the interpretation of the scree plot but some say to
take the number of components to the left of the “elbow”. Following this criterion, we would
pick only one component. A more subjective interpretation of the scree plots suggests that any
number of components between 1 and 4 would be plausible and further corroborative evidence
would be helpful.

Some criteria say that the total variance explained by all components should be between 70%
to 80% variance, which in this case would mean about four to five components. Picking the
number of components is a bit of an art and requires input from the whole research team. Let’s
suppose we talked to the principal investigator and she believes that the two-component
solution makes sense for the study, so we will proceed with the analysis.

Running a PCA with 2 components in SPSS

Running the two component PCA is just as easy as running the 8 component solution. The only
difference is under the Fixed number of factors – Factors to extract you enter 2.
We will focus the differences in the output between the eight and two-component solution.
Under Total Variance Explained, we see that the Initial Eigenvalues no longer equals the
Extraction Sums of Squared Loadings. The main difference is that there are only two rows of
eigenvalues, and the cumulative percent variance goes up to 51.54%.

Similarly, you will see that the Component Matrix has the same loadings as the eight-
component solution but instead of eight columns, it’s now two columns.

Communalities of the 2-component PCA

The communality is the sum of the squared component loadings up to the number of
components you extract. In the SPSS output, you will see a table of communalities.
Since PCA is an iterative estimation process, it starts with 1 as an initial estimate of the
communality (since this is the total variance across all 8 components), and then proceeds with
the analysis until a final communality is extracted. Notice that the Extraction column is smaller
Initial column because we only extracted two components. As an exercise, let’s manually
calculate the first communality from the Component Matrix. The first ordered pair
is (0.659,0.136) which represents the correlation of the first item with Component 1 and
Component 2. Recall that squaring the loadings and summing down the components (columns)
gives us the communality:

h2=(0.659)2+(0.136)2=0.453

Going back to the Communalities table, if you sum down all 8 items (rows) of the Extraction
column, you get 4.123. If you go back to the Total Variance Explained table and summed the
first two eigenvalues you also get 3.057+1.067=4.124. Is that surprising? Basically, it’s saying
that the summing the communalities across all items is the same as summing the eigenvalues
across all components.

How factor analysis is used

• Business marketing In a business model, factor analysis is used to explain complex

variables or data using the matrix of association. It studies the interdependencies of data
and assumes that complex variables can be reduced to a few important dimensions. This
is possible because of some of the relationships between variables and their dimensions.
The attribute of one variable might sometimes be the result of the dimension of another.
It breaks down the initial rating, using statistical algorithms on various components and
uses these partial scores to extract various factors.
• Automotive industry The use of factor analysis in the automotive industry was
mentioned as far back as 1997 in an article by Professor Emeritus Richard B. Darlington
of Cornell University. He explained how a study could be used to identify all the
variables that apply to the decision-making of purchasing a car—size, pricing, options,
accessories, and more. The study could then be used to arrive at a few key variables
that actually close a purchase decision. Automotive dealers can then tailor their
offerings to cater to the market.
• Investing The key to a productive investment portfolio is diversification. To ensure a
diverse portfolio, investment professionals use factor analysis to predict movement
across a wide sector of industries and provide insights on factors that may be under the
radar. For example, the average portfolio contains stocks of industries like technology
and commodities. A look at the rise in stock prices of a related industry, like oil, will
give investment professionals a good idea on what to sell and retain.
• Human resources There are many factors that go into a company’s hiring process.
With statistics, human resource professionals will be able to create a comfortable and
productive working environment. Several variables can be compared and analyzed to
see which combination in terms of the number of team members, varied skill sets, and
contractual or in-house talent works, improving the overall functioning of the
organization.
• Restaurants For restaurants, factor analysis can be used to understand demographics
and target diners in the creation of menus. A fast-food restaurant opening next to a
university campus will have to plan its menu differently than if it was placed in a high-
 end shopping location. Factors such as surrounding competition, foot-traffic, age-
groups, and location all determine success.
• Education When hiring teachers and deciding on a curriculum for the school year,
factor analysis plays a huge role. It is used to determine classroom sizes, staffing limits,
salary distribution, and a wide range of other requirements necessary for the school year
to run smoothly.

Ordinary least squares (OLS) regression is a statistical technique that uses a straight line to
estimate the relationship between a dependent variable and one or more independent variables.
Kaiser's criterion is based on the concept of eigenvalues, which are measures of how much variance
each factor accounts for in your data. The higher the eigenvalue, the more important the factor is.
Kaiser's criterion suggests that you should only keep the factors that have eigenvalues greater than
one, and discard the rest. This means that you are only retaining the factors that explain more variance
than a single variable would on its own.

The scree plot method: a graph that helps determine the number of factors or components to keep
in an analysis. It's used in multivariate statistics, such as exploratory factor analysis (EFA) and
principal component analysis (PCA).