Rise of the Machines: Application

of Machine Learning to Mortgage

Prepayment Modeling
Glenn M. Schultz and Frank J. Fabozzi

Glenn M. Schultz
is the director of mortgage
KEY FINDINGS
prepayment modeling for
MUFG Securities, in n Machine learning mortgage prepayment models are proving competitive, if not superior,
New York, NY.
to the traditional modular mortgage prepayment model.
glenn.schultz@
mufgsecurities.com n Key to training a machine learning prepayment model is managing the bias variance
tradeoff through the proper selection of the machine hyperparameters.
Frank J. Fabozzi
is a professor of finance at n One of the most powerful techniques to improve performance of a machine learning
EDHEC Business School prepayment model is “boosting”; an ensemble method that improves the predictive
in Nice, France. accuracy of the model by combining the output of many “weak leaners” into a “strong
[email protected]
committee.”

ABSTRACT
Key to the valuation of agency residential mortgage-backed securities (MBSs) is the modeling
of voluntary prepayment and default behaviors of the underlying borrowers in the mortgage
pool. The proliferation of pool- and loan-level data coupled with access to advanced machine
learning algorithms has opened the door to the application of machine learning to mortgage
prepayment modeling. The modular prepayment model, one that relies on defined functions
to predict mortgage prepayment, has dominated the MBS market nearly since its inception.
However, machine learning models are beginning to make inroads and, in some cases, are
replacing traditional modular prepayment models. The modular and machine learning model
differ in the following ways: In the case of modular prepayment models, either added or multi-
plicative, the modeler defines the functional form of each feature as well as the “tuning” of the
parameters passed to each. Machine learning or “second generation” mortgage prepayment
models differ in the sense that the modeler “tunes” the hyperparameters that determine the
bias variance tradeoff while the machine determines the functional form of each feature of
the model. In this article, the authors propose a machine learning mortgage prepayment
model using a boosted gradient classifier, trained at the loan level and generalized to the
pool level. A gradient boosted classifier is a tree-based model using an ensemble of weak
learners to create a strong committee for prediction.

T
he residential mortgage-backed securities (MBSs) market comprises three
subsectors based on the type of issuer/guarantee: US government backed
securities (Government National Mortgage Association [Ginnie Mae GNMA]),
government-sponsored enterprises or GSEs (Fannie Mae/FNMA and Freddie Mac/
FHLMC), and private-label MBSs. Central to the valuation of residential MBS is the
modeling of the prepayment and default behaviors of homeowners. The data for

Downloaded from https://pm-research.com/content/iijfixinc/31/3, at University of Illinois on January 21, 2025. Copyright 2021 With Intelligence LLC.
It is illegal to make unauthorized copies, forward to an unauthorized user, post electronically, or store on shared cloud or hard drive without Publisher permission.
 Winter 2022 The Journal of Fixed Income | 7

modeling prepayment and default behaviors are obtained from information provided
by the master servicer of MBS issuers on a regular basis. That information is in the
form of a remittance report that details the performance of the collateral (i.e., pool
of loans) backing the security. In the relatively recent past, loan-level data were not
readily available for residential MBS valuation. As a result, prepayment modeling was,
for the most part, based on the remittance data provided by the issuers.
In December 2004, the Securities and Exchange Commission (SEC) adopted
a comprehensive set of regulations for disclosure and reporting of publicly offered
asset-backed securities (ABSs). These regulations, which took effect on January 1,
2006, are commonly referred to as Regulation AB. Since MBSs are the major type
of ABS, Regulation AB applied to issuers of MBSs. However, Regulation AB applies
to private-label MBSs only and not MBSs issued by Ginnie Mae, Fannie Mae, and
Freddie Macs which are exempt from the reporting requirements of the Securities
Exchange Act of 1934. In August 2014, the SEC adopted the final Regulation AB
rules, referred to as Regulation AB II. These rules included asset-level information
requiring issuers to provide standardized loan-level information for ABSs collateral-
ized by residential mortgages, commercial mortgages, auto loans and leases, debt
securities, and re-securitizations.
Coincident with the effective date of Regulation AB in 2006, Freddie Mac began
releasing the loan-level data collateralizing all of its outstanding mortgage pools.
Prior to the adoption of Regulation AB II, Fannie Mae and Ginnie Mae began releasing
loan-level data starting around January 2013.
Each data set differs in terms of the historical data offered: (1) Freddie Mac’s
data set reports loan-level data on all securities outstanding and issued as of January
2006, (2) Fannie Mae’s data set provides loan-level data on those securities issued
on or after January 2013, and (3) GNMA’s data set reports loan-level data on all
securities outstanding and issued as of January 2013. In addition, Fannie Mae and
Freddie Mac provide historical credit data sets, derived from their loan-level data,
reporting delinquency, default, and voluntary repayment as of January 2000.
The purpose of this article is to apply machine learning tools to the modeling
of prepayment behavior of homeowners whose loans are securitized via the GSEs.
We provide modeling examples using the Freddie Mac loan-level data set because it
represents the more comprehensive release of loan-level data of the two GSEs. We
use the eMBS loan and pool-level data sets. eMBS, a leading provider of agency MBS
data, and Kodiak Advisors have developed an AWS Redshift implementation of the
eMBS database. In addition to the eMBS data, Bond Lab LLC maintains a database
of the Federal Housing Finance Agency (FHFA) home price indexes and FHLMC Primary
Mortgage Market Survey rates (PMMS®).

COHORT MODELING VERSUS LOAN-LEVEL MODELING

Cohort analysis is a direct result of the data limitations extant in the mortgage
market prior to the adoption of the final rules under Regulation AB, namely, reliance
on remittance data. Remittance data include cash flow due the investor each month,
scheduled principal, interest, prepaid principal, default recovery, and pool factor (i.e.,
a measure of the amount of the original loan that remains for the collateral). The
information provided in the remittance data allows the modeler to create data sets
reporting a measure of prepayments for an MBS: the single monthly mortality rate
(SMM). Thus, the modeling exercise becomes one of time-series analysis. Loan-level
data, in contrast, report and track the performance of each loan on a monthly basis.
A loan is either active or terminated. As a result, the modeling exercise becomes one
of classification—an accurate representation of the problem.

Downloaded from https://pm-research.com/content/iijfixinc/31/3, at University of Illinois on January 21, 2025. Copyright 2021 With Intelligence LLC.
It is illegal to make unauthorized copies, forward to an unauthorized user, post electronically, or store on shared cloud or hard drive without Publisher permission.
 8 | Rise of the Machines: Application of Machine Learning to Mortgage Prepayment Modeling Winter 2022

Traditional cohort modeling is based on a modular approach. That is, each feature
of the model is specified via a function. Often, the parameter(s) of each function are
exposed to the user. The exposure of the function parameter(s) facilitates model user
adjustments (tuning) of the model, thereby allowing the user to express an opinion
with respect to the influence of each function—essentially regression analysis or
constrained optimization.

THE MORTGAGE PREPAYMENT MODEL

Central to all mortgage prepayment models are the following features that are
well known to impact mortgage prepayments:1

§ External time-dependent variables: housing turnover rate, seasonal factors

and updated borrower loan-to-value ratio.
§ Internal time-dependent variables: loan age (seasoning) and borrower incentive
(prevailing mortgage rate minus borrower mortgage rate).
§ Continuous variables: debt-to-income ratio and credit score.

Continuous variables may be either static, measured at the time of origination,

or time dependent. Often the state of these variables, static or time dependent, is
a function of data availability.
In addition to the above features, categorical variables are included. Typically,
categorical variables relate to borrower or loan characteristics such as loan purpose,
property type, property location, and servicer. Often these are represented as multi-
pliers on either housing turnover or borrower incentive.
Related to the borrower incentive, an internal time-dependent variable, is the
concept of burnout. Burnout refers to the tendency of those borrowers who have
bypassed previous refinancing opportunities to remain unresponsive to subsequent
refinancing opportunities. Burnout is often modeled via a fast/slow payer framework.
That is, as each refinancing epoch passes, those borrowers that have not responded
are said to be slow payers, exhibiting burnout. The transition from fast to slow payer
and slow to fast payer is modeled as borrower state transition between the two,
further complicating the modeling process.
The structure of most prepayment models is additive and specified as follows:

SMM = (turover × seasonal factor) + (borrower incentive × burnout)

SMM expresses the percentage of prepaid principal as a function of the out-

standing balance returned to the investor in the current month after giving credit
to scheduled principal due in the current month. Below we outline a basic modular
prepayment model.

§ Turnover: The modeler may choose to express turnover (denoted by l) as a

function of predictor variables or may choose to simply assume a long-term
average. In either case, it is apparent that the prepayment model may take
the form of an exponential survivorship model in that l, the baseline survival
function is housing turnover.2
§ Loan seasoning: Loan seasoning is a three-parameter asymptotic function
acting as a multiplier on the turnover rate. The function is
1
See Chapter 8 in Schultz (2016).
2
See Chapter 9 in Schultz (2016).

Downloaded from https://pm-research.com/content/iijfixinc/31/3, at University of Illinois on January 21, 2025. Copyright 2021 With Intelligence LLC.
It is illegal to make unauthorized copies, forward to an unauthorized user, post electronically, or store on shared cloud or hard drive without Publisher permission.
 Winter 2022 The Journal of Fixed Income | 9

α − β × e−θ × loanage

where a is the function’s asymptote, β is the function’s intercept, and θ is

the point of maximum curvature.
§ Seasonality: Mortgage prepayments follow the home selling season in that
they increase in the spring and summer months and decline in the fall and
winter months. The function is

 π month + θ − 3 
1 × α sin    ×  
 2   3−1

where a sets the function’s maximum value, and q sets the point at which
the function reaches its maximum value.
§ Borrower incentive: Borrower incentive is calibrated twice. Once to describe
the borrower incentive of the fast payer cohort and second to calibrate the
slower payer borrower cohort. The borrower incentive function is

  η − arctan ( x ) 
arctan  [ x × π ] × β  
  π

where x is the borrower incentive, β an integer that defines the slope of the
response function, and h an integer that defines the location of the response
function.
§ Burnout: Burnout determines the population distribution between fast and
slower payer. The function for burnout is

eβ1 × loanage+β2 ×max(incentive, start value)

The analyst may choose to “fit” the model via constrained optimization. Alterna-
tively, the analyst may choose to “tune” the model based on the analyst’s unique
outlook on the future of prepayment rates.
The above is intended to provide context and a point of reference for the current
state of prepayment modeling. The choice of the functional form of each feature
depends on the analyst’s preference. Nonetheless, mortgage prepayment modeling
follows the general framework above—it is both labor intensive and prone to bias.
Indeed, one can simply adjust the model in the way one sees proper irrespective
of the rationality behind each adjustment. This is often referred to as “turning the
knobs”—hardly an endorsement of the practice.
Our goal is to employ machine learning, in this case stochastic gradient boosting,
to train a mortgage prepayment model suitable for production in an analytic frame-
work. The prepayment model is trained at the loan level and generalized to the pool
level, returning an SMM vector given a forward interest rate path. Exhibit 1 outlines
the target variable and features of the model.

OVERVIEW OF MACHINE LEARNING

Machine learning is the process of determining the appropriate algorithm to

apply to a collection of inputs thereby producing the desired result. Machine learning
encompasses both simple and deep learning. Examples of simple learning include
decision trees, random forest, and support vector machines. A neural net is an
example of deep learning. Often, the choice of which model to use depends on the

Downloaded from https://pm-research.com/content/iijfixinc/31/3, at University of Illinois on January 21, 2025. Copyright 2021 With Intelligence LLC.
It is illegal to make unauthorized copies, forward to an unauthorized user, post electronically, or store on shared cloud or hard drive without Publisher permission.
 10 | Rise of the Machines: Application of Machine Learning to Mortgage Prepayment Modeling Winter 2022

EXHIBIT 1
Prepayment Model Target and Features

Target
Loan Status Integer Internal Discrete
Features
Loan Age Integer Internal Continuous
Seasonality Integer External Discrete
Borrower Incentive Decimal Internal Continuous
Incentive Lagged 1-mo. Decimal Internal Continuous
Incentive Lagged 2-mo. Decimal Internal Continuous
Mortgage Curve Decimal External Continuous
Mortgage Curve Lagged 1-mo. Decimal External Continuous
Mortgage Curve Lagged 2-mo. Decimal External Continuous
Spread at Origination Decimal Internal Continuous
Original Loan to Value Decimal Internal Continuous
Home Price Appreciation Decimal External Continuous

type of problem and the modeler’s determination of the proper algorithm to apply to
the problem. According to Garath et al. (2017), machine learning, simple or deep,
can be classified as follows:

§ Supervised Learning, “for each observation of the predictor measurement

xi , i = 1, …, n there is an associated response measurement yi.” (Garath
et al., 2017). The goal is to fit a model that relates the response to the pre-
dictor variables as accurately as possible, the purpose of which is predicting
the response to future observations (prediction) or to better understand the
relationship between the response and predictor variables (inference).
§ Unsupervised Learning, “for every observation of i = 1, …, n we observe a
vector of measurements xi but no associated yi. It is not possible to fit a lin-
ear regression model since there is no response variable to predict” (Garath
et al., 2017). In this case, the modeler is seeking to understand the relation-
ship between variables or observations. The canonical example of unsuper-
vised learning is cluster analysis. The objective in the application of cluster
analysis is to determine based on xi , …, xn , whether the observations can
be placed into relatively distinct groups (Garath et al. 2017).

MACHINE LEARNING MORTGAGE PREPAYMENT MODEL

Machine learning applied to the mortgage prepayment modeling problem has

tended to focus on the use of neural networks3 (deep learning) to either pool-level data
(regression) or loan-level data (classification). The machine learning algorithm of choice
for our agency mortgage prepayment model is the gradient boosting classifier—a
boosted tree model. Tree models partition the feature space into rectangles. With
each partition a simple model is fit. Tree-based methods are both conceptually simple
and powerful. Furthermore, trees possess the advantage of interpretability in that
the feature space partition is fully described by a single tree. According to Zhang
et al. (2019) tree ensembles may outperform deep learning; indeed, tree models are

3
See Zhang et al. (2019).

Downloaded from https://pm-research.com/content/iijfixinc/31/3, at University of Illinois on January 21, 2025. Copyright 2021 With Intelligence LLC.
It is illegal to make unauthorized copies, forward to an unauthorized user, post electronically, or store on shared cloud or hard drive without Publisher permission.
 Winter 2022 The Journal of Fixed Income | 11

considered one of the better “off-the-shelf” data mining tools (Hastie 2012) for the
following reasons:

§ Interpretability: They are fast to construct and provided the tree is relatively
small, they produce interpretable models,
§ Robustness: They easily incorporate mixtures of numerical and categorical
variables and missing data and are immune to the influence of predictor
outliers.
§ Internal Feature Selection: They perform internal feature selection and thus
are resistant to the inclusion of irrelevant predictor variables.

Despite their strengths, tree models suffer from the following limitations according
to Hastie (2012):

§ Inaccuracy: They seldom provide predictive accuracy comparable to that which

can be best achieved by employing other methods.
§ Instability: They tend to exhibit a high degree of variance (i.e., a small change
in the training data can result in a different series of splits, which makes the
interpretability somewhat challenging). The instability is a direct result of the
hierarchical nature of the tree-building process.
§ Lack of Smoothness: Given that trees partition the feature space into
rectangles, the prediction surface lacks smoothness—a major drawback
when predicting mortgage prepayments.
§ Difficulty in Capturing Additive Structures: Trees may choose the appropriate
split but in the case of complex additive structures, such as mortgage pre-
payments, a degree of ‘lucky’ splits may be required to recreate the proper
interaction—due to the binary structure of the tree.

Boosting
Given the weakness of tree models with respect to predictive accuracy, why
would an analyst choose to use a tree model for prediction? Boosting. Boosting
tree models dramatically improves their predictive accuracy while preserving their
desirable properties for data mining. However, the predictive accuracy of boosting
requires a tradeoff between speed and interpretability. Boosting combines the output
of many “weak” classifiers to produce a strong “committee.” Within this context a
weak classifier is one whose error rate is marginally better than a random guess.
Boosting sequentially applies the weak classification algorithm to repeatedly modified
versions of the training data. In turn, this produces a sequence of weak classifiers.
The predictions from the ensemble of the weak learners are then combined through
a weighted majority vote to produce the final prediction.
The “boosting” process is also referred to as a “strong committee” machine.
Strong committee machines may be of one of two types of ensemble averaging,
typically used with neural nets, or boosting, typically used with tree-based methods.
The difference between the two being that boosting trains the weak learners on data
sets with entirely different distributions. Thus, boosting serves as a method that can
improve the performance of machine learning algorithms.
Boosting was originally designed for classification problems and later extended
to regression problems. In the context of mortgage prepayment modeling, we focus
on boosting as a classification problem and train the model at the loan level using
monthly transaction data. That is, we will classify a loan, in any given period, as either
active (negative observation assigned a value of 0) or terminated (positive observation
assigned a value of 1).

Downloaded from https://pm-research.com/content/iijfixinc/31/3, at University of Illinois on January 21, 2025. Copyright 2021 With Intelligence LLC.
It is illegal to make unauthorized copies, forward to an unauthorized user, post electronically, or store on shared cloud or hard drive without Publisher permission.
 12 | Rise of the Machines: Application of Machine Learning to Mortgage Prepayment Modeling Winter 2022

EXHIBIT 2
Learning Rate and Number of Learners

0.765

0.760
AUC

0.755

0.750

500 1000 1500 2000

Number of Learners

Learning Rate 0.1 0.2 0.3 Tree Depth = 2

Hyperparameters
Most boosting models offer a number of hyperparameters that can be adjusted to
control the bias variance tradeoff of the model. The most significant are learning rate,
number of learners (trees), tree depth, and subsampling (row wise and column wise).
The learning rate is a regularization parameter. The simplest application of which
is to scale the contribution of each tree by a factor between 0 and 1. Smaller values
of the learning rate increase the training risk, holding the number of learners con-
stant. Consequently, both the learning rate and the number of learners control the
prediction risk given the training data. Further, the learning rate and the number of
learners are not independent of one another. That is, smaller values for the learning
rate lead to a larger number of learners.
Exhibit 2 shows the tradeoff between learning rate and the number of learners
using a tree depth of two. The objective function maximized is the area under the
receiver operating (ROC) curve, discussed in greater detail later in this article. The
grid search is conducted setting tree depth equal to two and testing the learning rate
over a number of learners from 250 to 2,000. As the number of learners increases
the area under the curve (AUC) generally increases. In this case monotonically, indi-
cating diminishing improvement in the model as the number of learners increases.
Generally speaking, the ROC curve that lies above is considered the “best” value for
the hyperparameter under consideration, in this case 0.30. However, research indi-
cates that a smaller learning rate and greater number of learners produce superior
results. Following the literature, we select a learning rate of 0.10.
Tree depth controls the interaction level. That is, no interaction levels greater
than tree depth—1 are possible. Thus, setting the tree depth equal to two results in
a boosted model with only main interactions; setting the tree depth equal to three
results in a boosted model with two variable interactions. The tree depth should

Downloaded from https://pm-research.com/content/iijfixinc/31/3, at University of Illinois on January 21, 2025. Copyright 2021 With Intelligence LLC.
It is illegal to make unauthorized copies, forward to an unauthorized user, post electronically, or store on shared cloud or hard drive without Publisher permission.
 Winter 2022 The Journal of Fixed Income | 13

EXHIBIT 3
Tree Depth

0.765

0.760
AUC

0.755

0.750

500 1000 1500 2000

Number of Learners

3 5 7
Tree Depth
4 6 8
Learning Rate = 0.1

reflect the number of dominant interactions—which is typically not known at the

start of the modeling project. Hastie (2012) suggests that a tree depth between
four and eight generally works well in the context of boosting. Exhibit 3 shows the
impact of tree depth holding the learning rate at a constant value of 0.10. A tree
depth of three and four results in increasing monotonic ROC values as the number
of learners increases, while tree depths greater than four result in decreasing ROC
values as the number of learners increases. The analysis of tree depth is consistent
with the literature, which indicates a tree depth of three or four is generally sufficient
to capture variable interaction.
Stochastic gradient boosting samples a fraction of the training data (h) at each
iteration without replacement and grows the next tree using the subsample selected.
Subsampling reduces the computing time by the same fraction (h) and in many
cases produces a more accurate model. Exhibit 4 shows the impact on model per-
formance given row subsampling values of 0.30, 0.50, and 0.80. A row subsam-
pling value of 0.80 produces a ROC value that lies above both 0.30 and 0.50.
In the case of the 0.30 row subsampling strategy, the ROC values are below both 0.50
and 0.80 subsampling values, and the ROC value begins to decline at 2,000 learners.
In all, there are five hyperparameters to “tune”: learning rate, number of trees, tree
depth, row wise subsampling, and column wise subsampling. Of these, the number
of trees is the primary parameter.
Thus far, the traditional modular mortgage prepayment model and the
second-generation machine learning prepayment share commonality in that the
modeler must “tune” each. In the case of the modular mortgage prepayment model,
the modeler is both determining the functional form of each feature of the model
and providing the value of the parameters used for each feature. In the case of the
modular prepayment model outlined above, the modeler must provide a total of nine
coefficients to produce the best estimate and provide the best out-of-sample fit.

Downloaded from https://pm-research.com/content/iijfixinc/31/3, at University of Illinois on January 21, 2025. Copyright 2021 With Intelligence LLC.
It is illegal to make unauthorized copies, forward to an unauthorized user, post electronically, or store on shared cloud or hard drive without Publisher permission.
 14 | Rise of the Machines: Application of Machine Learning to Mortgage Prepayment Modeling Winter 2022

EXHIBIT 4
Row and Column Subsampling
Panel A: Row Subsampling
0.766

0.764

0.762
AUC

0.760

0.758

500 1000 1500 2000

Number of Learners

Subsample 0.3 0.5 0.8 Learning Rate = 0.1

Panel B: Column Subsampling

0.766

0.764

0.762
AUC

0.760

0.758

0.756
500 1000 1500 2000
Number of Learners

Subsample 0.5 0.75 1 Learning Rate = 0.1

In the case of the machine learning prepayment model, the modeler does not
determine the functional form of each feature (i.e., the task of the machine learning
algorithm). Still, the modeler must provide up to five hyperparameters balancing the bias
variance tradeoff such that the model generalizes well to the unseen or test data set.

Downloaded from https://pm-research.com/content/iijfixinc/31/3, at University of Illinois on January 21, 2025. Copyright 2021 With Intelligence LLC.
It is illegal to make unauthorized copies, forward to an unauthorized user, post electronically, or store on shared cloud or hard drive without Publisher permission.
 Winter 2022 The Journal of Fixed Income | 15

EXHIBIT 5 PREPAYMENT MODEL

Stochastic Gradient Boost Hyperparameter
Our prepayment model is trained using the hyper-
Hyperparameter Value parameter values outlined in Exhibit 5. The model is
Objective Binary:Logistic trained as a classification problem with the active loan
Number of Learners 3,500 in the period classified as 0 and the terminated loan in
Learning Rate 0.1 the period classified as 1. The classifier is trained on
Tree Depth 4.0 101, 202, and 381 transaction records covering the
Row Subsampling 0.80 period January 2006 to December 2020. The data are
Column Subsampling 1.0 right censored and left truncated (see Schultz 2016).
Early Stopping 100 Iterations The objective function maximizes the AUC score.
The ROC curve is plotted with the true positive
rate (vertical axis) versus the false positive rate
(horizontal axis).

True Positive
True Positive Rate =
True Positive + False Negative

True Negative
Specificity =
True Negative + False Positive

False Positive Rate

False Positive Rate =
True Negative Rate + False Positive Rate

Exhibit 6 is the ROC curve for the prepayment model. The true positive rate and
false positive rate are calculated at different thresholds. A model with an AUC score
of 1.0 is a perfect model with complete separability between the negative and positive
classes. A model with an AUC score of 0 inverts the negative and positive classes,

EXHIBIT 6
AUC ROC Curve

1.00

AUC = 0.76

0.75
True Positive

0.50

0.25

0.00

0.00 0.25 0.50 0.75 1.00

False Positive

Downloaded from https://pm-research.com/content/iijfixinc/31/3, at University of Illinois on January 21, 2025. Copyright 2021 With Intelligence LLC.
It is illegal to make unauthorized copies, forward to an unauthorized user, post electronically, or store on shared cloud or hard drive without Publisher permission.
 16 | Rise of the Machines: Application of Machine Learning to Mortgage Prepayment Modeling Winter 2022

EXHIBIT 7
Feature Importance

incentive_1

loanage

mtgcurve_1

hpiratio

mtgcurve_2 Cluster
Features

1
mtgcurve_3 2
month

origltv

incentive_3

incentive_2

0.0 0.1 0.2 0.3 0.4

Importance

and a model with an AUC score of 0.50 cannot separate the negative and positive
classes. The model’s AUC ROC score, 76%, indicates that the model correctly clas-
sifies 76% of the positive classes, loan termination in the period.
Exhibit 7 presents the feature importance of the model. Each feature of the
model is scored according to its contribution to the model’s predictive accuracy. The
purpose of feature importance is to assist in the determination of the relationship
between the target variable and the feature. The dominate features of the model are
incentive_1, loan age, mortgage curve_1, and hpi ratio. Each feature is known to be
dominate drivers of voluntary mortgage termination.
Thus far, we have focused on assessing the model as a classifier. Model back-
testing is generalized to aggregate prepayments. That is, from the loan-level test
data we compute the current balance weighted feature values and SMM, which is
annualized to a conditional prepayment rate (CPR), thereby replicating the model’s
expected performance on pool-level data.
Exhibit 8 is a backtest of the test data for origination years 2011 to 2020. Over-
all, the model performs well and captures differences in the seasoning ramp of each
cohort. The notable exceptions are 2011, 2012, and 2015. The model overshot the
2011, 2012, and 2015 cohorts’ peak in the seasoning ramp. Nonetheless, overall,
the model appears to perform well across the vintage cohorts examined.
To assess the degree to which the classifier model generalizes to the cohort data,
we perform standard least squares regression in the following form:

SMMpredicted = α + β SMMactual

A good prepayment model will produce an a (intercept) and a β (slope) that are
not statistically different from either 0.0 or 1.0, respectively. Exhibit 9 summarizes the
analysis. With respect to all origination years, the intercept’s 2.5% and 97.5% confi-
dence interval is -0.0009 and -0.0002, respectively. Similarly, the 2.5% and 97.5%
confidence interval around the slope coefficient is 1.030 and 1.061, respectively. The
slope coefficient, for all origination years, indicates that for every unit change in the

Downloaded from https://pm-research.com/content/iijfixinc/31/3, at University of Illinois on January 21, 2025. Copyright 2021 With Intelligence LLC.
It is illegal to make unauthorized copies, forward to an unauthorized user, post electronically, or store on shared cloud or hard drive without Publisher permission.
 Winter 2022 The Journal of Fixed Income | 17

EXHIBIT 8
Test Data Backtest

2011 2012

60%

40%

20%

0%
2013 2014

60%

40%

20%

0%
2015 2016

60%
CPR (%)

40%

20%

0%
2017 2018

60%

40%

20%

0%
2019 2020

60%

40%

20%

0%
01-2011
07-2011
01-2012
07-2012
01-2013
07-2013
01-2014
07-2014
01-2015
07-2015
01-2016
07-2016
01-2017
07-2017
01-2018
07-2018
01-2019
07-2019
01-2020
07-2020
01-2021

01-2011
07-2011
01-2012
07-2012
01-2013
07-2013
01-2014
07-2014
01-2015
07-2015
01-2016
07-2016
01-2017
07-2017
01-2018
07-2018
01-2019
07-2019
01-2020
07-2020
01-2021

Date

Actual Model

Downloaded from https://pm-research.com/content/iijfixinc/31/3, at University of Illinois on January 21, 2025. Copyright 2021 With Intelligence LLC.
It is illegal to make unauthorized copies, forward to an unauthorized user, post electronically, or store on shared cloud or hard drive without Publisher permission.
 18 | Rise of the Machines: Application of Machine Learning to Mortgage Prepayment Modeling Winter 2022

EXHIBIT 9 actual SMM, the model SMM changes by 1.04. That

Prepayment Model Test is, our prepayment model is modestly fast, reporting a
slope coefficient greater than 1.0 relative to the actual
Coefficients Estimate Std. Error t Value Pr (>|t|) data. A review of Exhibit 9 shows this is generally the
All Orig. Year case across the origination cohorts examined.
Intercept –0.0005 0.0001 –3.159 0.0016 Exhibit 10 is a scatter plot of the predicted CPR
SMM 1.0456 0.0078 133.179 <2e-16 (vertical axis) versus the actual CPR (horizontal axis).
Orig. Year 2011 Overall, the data fit well on a 45 degree line whose
Intercept –0.0008 0.0007 –1.176 0.2420 origin is [0,0). The majority of the errors occur between
SMM 1.0667 0.0327 32.589 <2e-16 25 CPR and 35 CPR, reflecting the 2011 and 2012
Orig. Year 2012 origination cohorts’ aggressive seasoning ramps (see
Intercept 0.0013 0.0003 3.948 0.0001 Exhibit 8). Both the 2011 and 2012 cohorts were orig-
SMM 0.9622 0.0224 42.779 <2e-16 ination into a persistent rally in the 30-year mortgage
Orig. Year 2013
rate, which saw the 30-year mortgage decline 150
Intercept 0.0002 0.0001 1.504 0.1360 basis points from 4.96% to 3.40% according to data
SMM 0.8606 0.0143 59.877 <2e-16 from the Federal Reserve Bank of St. Louis.
Orig. Year 2014
Intercept –0.0009 0.0005 –1.789 0.0774
SMM 1.0562 0.0204 51.690 <2e-16 CONCLUSION
Orig. Year 2015
The modular prepayment model approach has
Intercept 0.0008 0.0004 2.059 0.0432
SMM 0.9334 0.0185 50.332 <2e-16
been the favored approach for modeling mortgage
prepayment risk since the inception of the agency
Orig. Year 2016
MBS market. However, improvements in computation
Intercept –0.0000 0.0000 –0.128 0.899
speed, memory, and data storage, increased access
SMM 1.0280 0.0179 57.273 <2e-16
to advanced machine learning algorithms, and the pro-
Orig. Year 2017
liferation of both loan-level and pool-level data have
Intercept –0.0012 0.0003 –3.567 0.0008
opened the door to the application of machine learning
SMM 1.0716 0.0130 82.290 <2e-16
to mortgage prepayment modeling.
Orig. Year 2018
The machine learning approach shifts the mod-
Intercept –0.0017 0.0008 –2.196 0.0353
eler’s focus from functions that describe prepay-
SMM 1.0902 0.0184 59.073 <2e-16
ment risk to designing robust machine learning data
Orig. Year 2019
pipelines, feature engineering, and hyperparameter
Intercept –0.0044 0.0029 –1.535 0.1400
tuning. The machine learning prepayment model is a
SMM 1.1593 0.0617 18.773 1.32e-14
data-driven model whereas the modular approach is
Orig. Year 2020 informed by data analysis but nonetheless allows the
Intercept 0.0025 0.0021 1.178 0.2689 modeler to incorporate a “view” on the future evolu-
SMM 0.6881 0.1194 5.759 0.0002
tion of prepayment risk. Irrespective of the method
chosen, domain knowledge remains a requirement to
successfully developing and deploying either mortgage
prepayment model.

Downloaded from https://pm-research.com/content/iijfixinc/31/3, at University of Illinois on January 21, 2025. Copyright 2021 With Intelligence LLC.
It is illegal to make unauthorized copies, forward to an unauthorized user, post electronically, or store on shared cloud or hard drive without Publisher permission.
 Winter 2022 The Journal of Fixed Income | 19

EXHIBIT 10
Scatter Plot: Predicted vs. Actual

60%

40%
Predicted

20%

0%

0% 20% 40% 60%

Actual

2011 2013 2015 2017 2019

2012 2014 2016 2018 2020

REFERENCES
Garath, J., D. Witten, T. Hastie, and R. Tibshirani. An Introduction to Statistical Learning: With
Applications in R. New York, NY: Springer, 2017.

Hastie, T. The Elements of Statistical Learning. New York, NY: Springer, 2012.

Schultz, G. M. Investing in Mortgage Backed and Asset Backed Securities: Financial Modeling with
R and Open-Source Analytics. Hoboken, NJ: John Wiley, 2016.

Schultz, G. M., and F. J. Fabozzi. “Agency Mortgage Passthrough Securities.” The Handbook of
Fixed Income Securities: 9th edition. New York, NY: McGraw Hill, 2016.

Zhang, J., X. Zhao, J. Zhang, F. Teng, S. Lin, and H. Li. 2019. “Agency MBS Prepayment Model
Using Neural Networks.” The Journal of Structured Finance 24 (4): 17–33.

Downloaded from https://pm-research.com/content/iijfixinc/31/3, at University of Illinois on January 21, 2025. Copyright 2021 With Intelligence LLC.
It is illegal to make unauthorized copies, forward to an unauthorized user, post electronically, or store on shared cloud or hard drive without Publisher permission.