Multi-state transition models with actuarial applications

by James W. Daniel

c Copyright 2004 by James W. Daniel Reprinted by the Casualty Actuarial Society and the Society of Actuaries by permission of the author.

1
Multi-state transition models for actuarial applications
Actuaries regularly use probability models to analyze situations involving risk. These models often involve some entity and the various states in which it might bealive or dead, intact or failed, et cetera. This chapter introduces a general type of model that can be applied in many such situations. Section 1 reviews some models of this type that youve probably already seen and then goes on to describe some practical applications for which those models are inadequate. Section 2 then introduces a more general probability model appropriate for these new cases.

1.1 Introduction
What are multi-state transition models? Probability models that descrbe the random movements of a subject among various states. Often the subject is a person, but it could just as well be a piece of machinery or a loan contract in whose survival or failure you are interested. Youre probably already familiar with some special cases of such situations. (1.1) Example (basic survival models). In a basic survival model for a status (x)possibly a person aged xfor which you study the failure time T (x) or K(x), youre considering two states: alive (or, more generally, Intact) and dead (or Failed). Models describe the probability of moving from the State Intact to the State Failed at various points in time. Example (multiple-decrement survival models). In multiple-decrement models, youre interested not only in the time of failure of a status (x) as in Example 1.1 but also in which of m causes #1, #2, . . . , #m was to 2

(1.2)

1.1 Introduction

blame. Models describe the probabilities of moving from the State Intact to one of the States Failed for Cause #1 or Failed for Cause #2 or . . . or Failed for Cause #m at various points in time. (1.3) Example (multiple-life models). In multiple-life models you consider the failure time of complex statuses constructed from other statuses. For example, on a pair of statuses (x) and (y), you might be interested in the joint status x : y that fails when the rst of (x) and (y) fails or in the lastsurvivor status x : y that fails when the last of (x) and (y) fails. [More 1 complicated statuses include x:y that fails at the failure of (x) provided that (x) fails before (y), but these will not be treated here.] Our subject is the pair of statuses, and the possible states are: 1) both are intact, 2) (x) is intact but (y) has failed, 3) (y) is intact but (x) has failed, and 4) both have failed. Models describe the probabilities of moving among these states at various points in time.

All three preceding multi-state transition models share a common characteristic: once the subject leaves a state it cannot return to that state. For instance, in Example 1.1 once the state is Failed it stays Failed forever. But there are important applications in which subjects move back and forth among states, possibly returning to states they have previously left. (1.4) Example (disability). In modeling workers eligibility for various employee benets, you might want to consider such states as Active, Temporarily Disabled, Permanently Disabled, and Inactive (which might include retirement, death, and withdrawalalthough these could also be used as distinct states). Models describe the probabilities of moving among these various states, including the possibility of moving back and forth between Active and Temporarily Disabled several times. Example (driver ratings). In modeling insured automobile drivers ratings by the insurer, you might want to consider states such as Preferred, Standard, and Substandard. Models describe the probabilities of moving back and forth among these states. [You might also include a state Gone for those no longer insured.] Example (Continuing Care Retirement Communitiesor CCRCs). In a Continuing Care Retirement Community (CCRC), residents may move among various states such as Independent Living, Temporarily in the Health Center, Permanently in the Health Center, and Gone. Models describe the probabilities of moving among these states at various points in time.

(1.5)

(1.6)

4 Chapter 1 Multi-state transition models for actuarial applications To deal with the sort of applications in these last three examples, actuaries need models that allow for moving back and forth among states. Section 1.2 presents one approach to such models.

1.2 Non-homogeneous Markov Chains

In dening models that allow the subject to move back and forth among various states, Im going to make some simplifying restrictions and consider only models with: 1) discrete time (meaning that the states are described at times 0, 1, 2, . . . ); 2) a nite number of states in which the subject may be; and 3) history independence (meaning that the probability distribution of the state of the subject at time n + 1 may depend on the time n and on the state at time n but does not depend on the states at times prior to n). Such a model is called a non-homogeneous Markov Chain, although the general non-homogeneous Markov Chain does not require restrictions 1) and 2) above. When the probability distribution in 3) does not depend on the time n, the model is called a homogeneous Markov Chain or often simply a Markov Chain. The following denition makes this more precise: (1.7) Denition (non-homogeneous Markov Chain). M is a non-homogeneous Markov Chain when M is an innite sequence of random variables M0 , M1 , with the following properties. 1) Mn denotes the state number of a subject at time n. 2) Each Mn is a discrete-type random variable over r values (usually 1, 2, , r but sometimes 0, 1, , m with r = m + 1). 3) The transition probabilities Q(i,j) = Pr[Mn+1 = j Mn = i and various other previous values of Mk ] n = Pr[Mn+1 = j Mn = i] are history independent. (i,j) If the transition probabilities Qn pronounced q-sub-n i-to-j do not in fact depend on n, then they are denoted by Q(i,j) and the Chain is a homogeneous Markov Chain. Note that history independence implies the important and useful fact that the probability of moving from State #i to #j and then to #k is simply the product of the probability of moving from #i to #j with the probability of moving from #j to #kthat is, successive transitions are independent events. At this point you should re-examine the Examples in Section 1.1 to see how they can be formulated as non-homogeneous Markov Chains. Heres what youll nd.

1.2 Non-homogeneous Markov Chains

(1.8)

Example (basic survival models as in Example 1.1). Let State #0 be that (x) is Intact and State #1 be that (x) is Failed. [The numbering was chosen so that this single-decrement case is consistent with the multipledecrement case in the next Example.] You should check that the transition (0,0) (0,1) (1,0) (1,1) probabilities are Qn = px+n , Qn = qx+n , Qn = 0, and Qn = 1. Example (multiple-decrement survival models as in Example 1.2). Let State #0 be that (x) is Intact, and State #j be that (x) has Failed for Cause #j, for j = 1, 2, . . . , m. You should check that the transition prob(0,0) ( ) (0,j) (j) (j,j) abilities are Qn = px+n , Qn = qx+n for j = 1, 2, . . . , m, Qn = 1 for (i,j) j = 1, 2, . . . , m, and Qn = 0 for all other values of i and j. Example (multiple-life models as in Example 1.3). Let State #1 be that both (x) and (y) are intact, #2 that (x) is intact but (y) has failed, #3 that (y) is intact but (x) has failed, and #4 that both have failed. Assuming for simplicity that (x) and (y) are independent lives, you should check that (1,1) (1,2) the transition probabilities are Qn = px+n:y+n = px+n py+n , Qn = (1,3) (1,4) px+n qy+n , Qn = py+n qx+n , and Qn = qx+n:y+n = qx+n qy+n ; also Qn = px+n and Qn = qx+n and similarly for Qn (4,4) (i,j) Qn = 1; and all other Qn = 0.
(2,2) (2,4) (3,3)

(1.9)

(1.10)

and Qn

(3,4)

(1.11)

Example (disability as in Example 1.4). Let State #1 stand for the employees being Active, #2 for Temporarily Disabled, #3 for Permanently (3,1) (3,2) Disabled, and #4 for Inactive. Clearly we must have Qn = Qn =0 since #3 denotes permanent disability. Unless we wish to model situations (4,4) (4,j) allowing a return from the Inactive status, Qn = 1 and Qn = 0 for j = 1, 2, 3. The other transition probabilities would be chosen to reect observations. Example (driver ratings as in Example 1.5). Let State #1 stand for the drivers being classied as Preferred, #2 for Standard, and #3 for Substandard. All the transition probabilities would be chosen to reect observations, and presumably all could be positive. Example (Continuing Care Retirement Communities as in Example 1.6). Let State #1 stand for the residents being in Independent Living, State #2 for Temporarily in the Health Center, #3 for Permanently in the Health (3,1) (3,2) Center, and #4 for Gone. Clearly we must have Qn = Qn = 0 since #3 denotes being Permanently in the Health Center. Unless we wish to

(1.12)

(1.13)

6 Chapter 1 Multi-state transition models for actuarial applications model situations allowing a return from the Gone status, Qn = 1 and (4,j) Qn = 0 for j = 1, 2, 3. The other transition probabilities would be chosen to reect observations.
(4,4)

More probabilities
Actuarial notation often uses q to denote failure probabilities [such as moving from State #0 (Intact) to the dierent State #1 (Failed) in the basic survival model] and p to denote success probabilities [remaining in State #0 in the basic survival model]. Analogously, it is sometimes convenient to use: (1.14) Notation. Pn = Qn is the success probability of remaining in State #i at the next time step. Even more convenient is to place the probabilities Qn (1.15)
(i,j) (i) (i,i)

in a matrix:

Denition (transition probability matrix). The transition probability matrix Qn is the r-by-r matrix whose entry in row i and column jthe (i,j) (i,j)-entryis the transition probability Qn .

Using this notation, the probabilities in Example 1.8, for instance, on the basic survival model could have been written as Qn =
(i,j)

px+n 0

qx+n 1

The transition probabilities Qn and the transition probability matrix Qn only provide information about the probability distribution of the state one time step in the future. In practice it is often important to know about longer periods of timewitness the importance of k px+n versus just px+n in basic survival models. For non-homogeneous Markov Chains, the corresponding notation is: (1.16) Notation. k Qn = Pr[Mn+k = j Mn = i], with k Qn used for the r-by-r (i,j) matrix whose (i, j)-entry is k Qn .
(i,j)

For basic survival models, of course, k px+n can be computed from the one-year probabilities as k px+n = px+n px+n+1 px+n+k1 . The same approach works in our more complicated setting. (1.17) Example (longer-term probabilities). Consider a simple example of a homogeneous Markov Chain with r = 2 states #1 and #2 and with transition probability matrix 0.4 0.6 Q= . 0.8 0.2

1.2 Non-homogeneous Markov Chains

Suppose that you want to compute 2 Q(1,2) , the probability that the subject, now in State #1, will be in State #2 after two time periods. The subject can be in State #2 after two transitions in either of two waysby moving #1 #1 #2 or by moving #1 #2 #2. So the probability is the sum of those two probabilities. Thanks to history independence, the events #1 #1 and #1 #2 are independent, and so Pr[#1 #1 #2] = Pr[#1 #1] Pr[#1 #2] = Q(1,1) Q(1,2) = 0.4 0.6. Similarly Pr[#1 #2 #2] = 0.6 0.2. Thus 2 Q(1,2) = 0.4 0.6 + 0.6 0.2. Butand here is the important observation, so check itthis is the same as the (1,2)-entry of the matrix Q Q. A similar argument shows that (i,j) is in general the (i,j)-entry of the matrix QQ. That is, 2 Q = Q2 . 2Q The argument used in Example 1.17 extends easily to the general case of longerterm probabilities for non-homogeneous Markov Chains, resulting in (1.18) Theorem (longer-term probabilities). In non-homogeneous Markov Chains (i,j) the longer-term probability k Qn can be computed as the (i,j)-entry of the matrix Qn Qn+1 Qn+k1 that is,
k Qn

= Qn Qn+1 Qn+k1 .

For a homogeneous Markov Chain, this matrix is just Qk . Calculation by hand of matrix products can be tedious, even in the Examples below. Fortunately, spreadsheet programs and other mathematical software can perform these calculations easily. (i,j) Warning on interpretation: Note that k Qn gives the probability of the subjects being in State #j after k time periods, not the probability of arriving there exactly k steps in the future. The subject might have reached State #j previously, left it, returned, et cetera. This is of course also true for the special case i = j, (i,i) that is, k Qn . The event for which this is the probability allows the subject to have drifted away from State #i, so long as the subject is back again after k time (i,i) periodsthus, k Qn is not analogous to the smaller survival probability of remaining in State #i throughout the k steps. For that, using Notation 1.14 we easily get (check this): (1.19) Theorem. The probability that a subject in State #i at time n remains in that state through time n + k is
(i) k Pn

= Pr[Mn+1 = Mn+2 = = Mn+k = i Mn = i]

(i) = Pn Pn+1 Pn+k1 (i) (i)

(1.20)

Example. For the homogeneous Markov Chain dened in Example 1.17, (1,1) (1) lets compute both 2 Qn and 2 Pn . According to Theorem 1.18 on

8 Chapter 1 Multi-state transition models for actuarial applications longer-term probabilities, 2 Qn Qn Qn+1 = Q2 =
(1,1) (1,1)

is the (1,1)-entry of 0.4 0.6 0.64 = 0.8 0.2 0.48 0.36 , 0.52
(1)

0.4 0.8

0.6 0.2

and so 2 Qn = 0.64. From Theorem 1.19 and Notation 1.14, 2 Pn (1) (1) (1,1) (1,1) Pn Pn+1 = Qn Qn+1 = [Q(1,1) ]2 = (0.4)2 = 0.16. (1.21)

Example. Consider a Continuing Care Retirement Community (CCRC) with four states: Independent Living, Temporarily in the Health Center, Permanently in the Health Center, and Gone, with the states numbered 1, 2, 3, 4, respectively. Suppose that the transition-probability matrices for a new entrant (at time 0) are as given in the Illustrative Matrices in Section 3.1. Given that this entrant is in Independent Living at time 2, lets nd the probability of being there at time 5 and also the probability of remaining there from time 2 through time 5. (1,1) The rst probability is 3 Q2 , which is the (1,1)-entry of Q2 Q3 Q4 = [note that I only write the entries in the matrices that I actually need in the calculation, writing elsewhere] 0.60 0.15 0.15 0.10 0.50 0.20 0.20 0.10 0.40 0.20 0.30 0.35 0.15 0.10 0 0 0.50 0.50 0 0 0 0 1 0 0.60 0.15 0.15 0.10 0.22 0.1485 0.11 = = 0 0 and so the probability 3 Q2 is 0.1485. (1) (1) (1) (1) (1,1) (1,1) (1,1) The second probability is 3 P2 = P2 P3 P4 = Q2 Q3 Q3 = (0.60)(0.50)(0.40) = 0.12. Example. Consider a driver-ratings model in which drivers move among the two classications Preferred and Standard at the end of each year. Each year: 60% of Preferred are reclassied as Preferred and 40% as Standard; and 70% of Standard are reclassied as Standard and 30% as Preferred. Lets nd the probability that a drtiver, known to be classied as Standard at the start of the rst year, will be classied as Standard at the start of the fourth year. Let Preferred be State #1 and Standard be State #2. Then the transition-probability matrix Q for this homogeneous Markov Chain is 0.6 0.3 0.4 . 0.7
(1,1)

(1.22)

1.2 Non-homogeneous Markov Chains

We seek 3 Q0 , which is the (2, 2)-entry of Q3 . Rather than proceeding as in the preceding Example, consider the following approach. Note that if ej denotes an n 1 column matrix with 1 as its j th entry and 0 as its other entries, then for any k n matrix M the product Mej is just the (2,2) j th column of M. Therefore the desired 3 Q0 , which is the (2, 2)-entry of Q3 , is just the bottom entry of Q3 e2 = Q2 (Qe2 ) = = 0.6 0.4 0.3 0.7 0.6 0.3 0.4 0.7 0.6 0.3 0.4 0.7 0.4 0.7

(2,2)

0.52 0.556 = , 0.61 0.583

giving 0.583 for the answer. Theres another probability that will prove central to computations in Section 2.2: for a subject in State #s at time n, the probability of making the transition from State #i at time n + k to State #j at time n + k + 1. In order to possibly make this transition, the subject rst must be in State #i at time n+k. Since the subject (s,i) is now in State #s at time n, the probability of this is k Qn . The probability of (i,j) (s,i) (i,j) the transition then from State #i to State #j is Qn+k . The product k Qn Qn+k of these two probabilities gives the probability of the transition in question. That is, (1.23) Theorem (future transition probabilities). Given that a subject is in State #s at time n, the probability of making the transition from State #i at (s,i) (i,j) time n + k to State #j at time n + k + 1 is given by k Qn Qn+k .

Problems
1. A basic aggregate survival model as in Example 1.1 follows the DeMoivre Law with ultimate age = 100. As in Example 1.8, nd the matrix Q30 for a person aged x = 60. [Answer: the rst row contains 0.9 and 0.1, the second 0 and 1.] 2. Consider a multiple-life model as in Example 1.10 for independent lives aged x = 60 and y = 75 subject to a DeMoivre Law with = 100. As in Example (1,2) 1.10, nd Q10 . 29 [Answer: 450 .] 3. For the model in Example 1.17, nd 3 Q(2,1) . [Answer: 0.608.] 4. As in Example 1.5, consider a driver-ratings model in which drivers move among the classications Preferred, Standard, and Substandard at the end of each

10 Chapter 1 Multi-state transition models for actuarial applications year. Each year: 60% of Preferreds are reclassied as Preferred, 30% as Standard, and 10% as substandard; 50% of Standards are reclassied as Standard, 30% as Preferred, and 20% as Substandard; and 60% of Substandards are reclassied as Substandard, 40% as Standard, and 0% as Preferred. Find the probability that a driver, classied as Standard at the start of the rst year, will be classied as Standard at the start of the fourth year. [Answer: 0.409.] Consider the situation in Problem 4 again. Find the probability that a driver, classied as Standard at the start of the rst year, will be classied as Standard at the start of each of the rst four years. [Answer: 0.125.] Consider the CCRC model in Example 1.21. Find the probability that a resident, in Independent Living at time 1, will not be Gone at time 3. [Answer: 0.8175.] Consider a disability model with four states, numbered in order: Active, Temporarily Disabled, Permanently Disabled, and Inactive. Suppose that the transition-probability matrices for a new employee (at time 0) are as given in the Illustrative Matrices in Section 3.1. For an Active employee at time 1, nd the probability the employee is Inactive at time 4. [Answer: 0.3535.] Consider a four-state non-homogeneous Markov Chain with transition probability matrices given by the Illustrative Matrices in Section 3.1. For a subject in State #2 at time 3, nd the probability that the subject transitions from State #1 at time 5 to State #3 at time 6. [Answer; 0.033.] (Theory.) Extend Example 1.17 in general for homogeneous Markov Chains with two states to prove that 2 Q = Q2 . (Theory.) Extend Problem 9 to non-homogeneous Markov Chains with r states to prove that 2 Qn = Qn Qn+1 . (Theory.) Extend Problem 10 to prove Theorem 1.18 on longer-term probabilities.

5.

6. 7.

8.

9. 10. 11.

2
Cash ows and their actuarial present values
Actuaries usually arent interested in a probability model for its own sake. Rather, they want to use the model to analyze the nancial impact of the events being modeled. Section 1 gives some simple examples of nancial consequences (cash ows) associated with some introductory examples from Section 1.1. Section 2 specializes to cash ows associated with transitions between states, while Section 3 treats cash ows that occur while the subject is in a particular state; both Sections examine computing the actuarial present value of cash ows. Finally, Section 4 introduces benet premiums and benet reserves in the context of the general non-homogeneous Markov Chain.

2.1 Introduction
Actuaries are not simply interested in modeling the future states of a subject. They also need to model cash ows associated with future states. (2.1) Example (insurances and annuities). In the basic survival models, multipledecrement models, and multiple-life models of Examples 1.1, 1.2, and 1.3, actuaries are concerned about insurancespayments made upon failure of a statusand about annuitiespayments made while a status is intact. [An annuity of course either could represent payments made by an annuity company to an annuitant, or could represent payments (premiums) paid to an insurer by an insured.] In our non-homogeneous Markov Chain models, insurance payments correspond to payments made upon transition from one state to another, while annuities represent payments made while the subject is in a particular state. Example (disability). In the disability model of Example 1.4, actuaries may be concerned about payments made to an employee while Temporarily or Permanently Disabled, and about administrative costs (possibly minor) associated with a change of status. In our non-homogeneous Markov Chain models, these correspond to cash ows while the subject is in a particular state or upon transition from one state to another. 11

(2.2)

12 Chapter 2 Cash ows and their actuarial present values (2.3) Example (driver ratings). In the driver-ratings model of Example 1.5, actuaries may be concerned about expected claims payable and premiums collected while a driver is in a particular classication, or about administrative costs (possibly minor except when a signicant change occursa Driving While Intoxicated conviction, for examplethat requires special underwriting) associated with a change of classication. In our non-homogeneous Markov Chain models, these correspond to cash ows while the subject is in a particular state or upon transition from one state to another. Example (CCRCs). In the CCRC models of Example 1.6, actuaries may be concerned about expenses to be paid and payments collected while a resident is in a particular classication, or about costs associated with a change of classication (such as moving a resident or cleaning an apartment). In our non-homogeneous Markov Chain models, these correspond to cash ows while the subject is in a particular state or upon transition from one state to another.

(2.4)

These examples make it clear that actuaries are concerned with cash ows while the subject is in a particular state or upon transition from one state to another in nonhomogeneous Markov Chains.

2.2 Cash ows upon transitions

I could easily let C (i,j) denote a cash oweither positive or negativethat occurs when a subject transitions from State #i to State #j (including the possibility that i = j). But that does not account for the possibility of the amount depending on the time at which it occurs. Since actuaries usually account for the time value of money, I also need to make an assumption about when the transition from the State M at time to the State M +1 at time + 1 occurs. By analogy with discrete insurances whose benets are paid at the end of the year of death, Ill here assume that the cash ow occurs at time + 1, although other assumptions (such as at mid-year) are certainly possible. This leads to the following (2.5) Notation (cash ows at transitions). +1 C (i,j) denotes the cash ow at time + 1 if the subject is in State #i at time and State #j at time + 1. Now, how to account for the time value of money? To be general, Ill use the following (2.6) Notation (discounting). k vn denotes the value at time n of one unit certain to be paid k periods in the future at time n + k.

1 In the case of compound interest at rate i, of course, k vn = v k , where v = 1+i is the one-period discount factor. Im using the more general notation to allow for computations with varying, or even random, interest rates each period.

2.2 Cash ows upon transitions

13

Actuarial present values

We now have the tools to calculate actuarial present valuesthe expected value of the present value of cash ows. How? By the usual triple-product summation approach (the 3 approach): Sum up, over all times at which a cash ow might occur, the product of three terms: the probability that the cash ow occurs then; the amount of the cash ow; and the discounting from the time of the cash ow back to the present time (n). [Its also possible to compute the variance of the present value of cash ows, but Ill not address that here.] We have at hand all the ingredients to compute the actuarial present value of the cash ows, upon transitions, in Notation 2.5 for a subject now in State #s at time n: the times at which the ows can occur are times n + k + 1 for k 0; the amount of the cash ow at that time is denoted by n+k+1 C (i,j) ; the discounting from time n + k + 1 to the present time n is k+1 vn ; and the probability of making the transition from State #i at time n + k to State #j (s,i) (i,j) at time n + k + 1 is, by Theorem 1.23, k Qn Qn+k . (2.7) Theorem (actuarial present value of cash ows at transitions). As in Notation 2.5, let +1 C (i,j) denote the cash ow at time + 1 if the subject is in State #i at time and State #j at time + 1. Suppose that the subject is now in State #s at time n. Then the actuarial present value, as seen from time n, of these cash ows is given by the triple-product summation (3) AP Vs@n (C
(i,j)

)=

k=0

[k Q(s,i) Qn+k ] [n+k+1 C (i,j) ] [k+1 vn ]. n

(i,j)

The general formula above is a mess. Spreadsheet software can easily perform such summations, however, and in simple illustrative cases the computations are fairly straightforward. (2.8) Example. Consider the simple example of a homogeneous Markov Chain in Example 1.17, with states numbered 1 and 2 and with Q= 0.4 0.8 0.6 . 0.2

Suppose that the subject is now in State #1 and that there is a cash ow of 1 for a transition from State #2 to State #1 any time in the next 3 periods. [Recall that this Markov Chain is homogeneous and so probabilities do not depend on timeso we can treat now as, say, time 0.] Finally, suppose that interest is constant at 25%, so that v=0.8 and k vn = (0.8)k . Note that a transition from State #2 to State #1 is impossible the rst year since the subject is in State #1, not State #2. That means that there are only

14 Chapter 2 Cash ows and their actuarial present values two possible transitionsin the second year from time 1 to time 2 or in the third year from time 2 to time 3. To compute the actuarial present value of these cash ows (as seen from now at time 0), we need the probabilities of being in State #2 at times 1 or 2 and the probability Q(2,1) = 0.8 of then having a transition from State #2 to State #1. The rst probability is the (1, 2)-entry of Q, which is just 0.6. The second probability is the (1, 2)-entry of Q2 , which you can easily compute to be 0.36 (check this). The actuarial present value from the 3 is then [(0.6)(0.8)](1)(v 2 ) + [(0.36)(0.8)](1)(v 3 ) = 0.45456. Since this is a homogeneous Markov Chain, the value of course does not actually depend on the time n. (2.9) Example. Consider the preceding Example 2.8 again, but this time suppose that the interest rate is varying: 10% in the rst year from time n to time n + 1, 15% the second year, and 20% the third year. Then discounting is accomplished by 1 1 1 1 vn = 1.1 = 0.90909, 2 vn = 1 vn 1.15 = 0.79051, and 3 vn = 2 vn 1.20 = 0.65876. The actuarial present value from the 3 is then [(0.6)(0.8)](1)(2 vn ) + [(0.36)(0.8)](1)(3 vn ) = 0.56917.

(2.10)

Example. Consider a CCRC with the usual four States as in Example 1.13. Suppose that the transition-probability matrices for a new entrant are as given in Section 3.1, and suppose that the cash ows upon transitions are as given in Section 3.2. Suppose that a resident is in Independent Living at time 5. Lets compute the actuarial present value (as seen from now at time 5) of the cash ows upon transition from Independent Living (State #1) either to Permanently in the Health Center (State #3) or to Gone (State #4), using a constant interest rate of 25%. Clearly, the actuarial present value of the cash ows from these two types of transitions is the sum of the actuarial present values for each type. Consider the rst type of transitionfrom State #1 to State #3. Such a transition can only occur from time 5 to time 6, from time 6 to time 7, or from time 7 to time 8. (Why? Look at Q8 , for example.) The probabilities of the (1,1) resident being in State #1 at the start of those years are: 1; Q5 = 0.3; and (1,1) , which is the (1,1)-entry of Q5 Q6 and is easily computed to be 0.08 (check 2 Q5 this). The probabilities of the rst type of transition in each of those years, once the (1,3) (1,3) (1,3) resident has reached State #1, are Q5 = 0.3, Q6 = 0.3, and Q7 = 0.3. The cash ows in each year are 53, 63, and 73 (check this). The 3 then computes the actuarial present value for the rst type of transition as [(1)(0.3)](53)v + [(0.3)(0.3)](63)v 2 + [(0.08)(0.3)](73)v 3 = 17.246.

2.3 Cash ows while in states

15

Now consider the second type of transitionfrom State #1 to State #4. Such a transition can only occur from time 5 to time 6, from time 6 to time 7, from time 7 to time 8, or from time 8 to time 9. (Why?) The probabilities of the resident being in State #1 at the start of the rst three of those years are again: 1; 0.3; (1,1) and 0.08. For the fourth year, it is 3 Q5 , which can be computed to be 0.012 (check this). The probabilities of the second type of transition in each of those years, once (1,4) (1,4) (1,4) the resident has reached State #1, are Q5 = 0.2, Q6 = 0.3, Q7 = 0.5, and (1,4) Q8 = 1. The cash ows in each year are 54, 64, 74, and 81 (check this). The 3 then computes the actuarial present value for the second type of transition as [(1)(0.2)](54)v + [(0.3)(0.3)](64)v 2 + [(0.08)(0.5)](74)v 3 + [(0.012)(1)]81v 4 = 14.201. The combined actuarial present value for the two types of transitions together is 17.426 + 14.201 = 31.447.

Problems
1. Consider a CCRC model with cash ows from the Illustrative Cash Flows in Section 3.2 as in Example 2.10. Suppose that a subject is in Independent Living (State #1) at time 4 and then in States #2, #2, #1, #3, and #4 at times 5, 6, 7, 8, and 9, respectively. Using 5% interest, nd the present value of the cash ows for these transitions. [Note that no probability is involved here.] [Answer: 272.03.] 2. Consider a homogeneous Markov Chain with two states and transition probability matrix as in Example 1.17. The subject is now in State #2 at time 3. There are possible cash ows of + 1 for transition from State #2 at time to State #1 at time + 1, for 5. Find the actuarial present value of these cash ows using 25% interest. [Answer: 4.3500.] 3. Solve Problem 2 again, but this time assume that the interest rates for the three future years are, in order, 10%, 15%, and 20%. [Answer: 5.1858.] 4. Consider a CCRC model with transition probabilities from the Illustrative Matrices in Section 3.1 and cash ows from the Illustrative Cash Flows in Section 3.2, exactly as in Example 2.10. The subject is in Independent Living at time 5. Find the actuarial present value of the cash ows resulting from future transitions from Temporarily in the Health Center to Permanently in the Health Center, using 25% interest. [Answer: 4.3766.] 5. Solve Problem 4 again, but this time assume that the interest rate from time n to time n + 1 is 0.05|n 4|. [Answer: 6.0320.]

16 Chapter 2 Cash ows and their actuarial present values

2.3 Cash ows while in states

The rst thing that you should notice about this Section is that it looks a great deal like the preceding Section 2.2. The only change is that the cash ows now occur while the subject is in a state rather than upon transition between states. In computing actuarial present values using the triple-product summation approach (the 3 approach), the main dierence lies in the probability that the cash ow occurs. I again need to make an assumption about when the cash ow for being in State #i at time occurs. By analogy with annuities-due whose payments fall at the start of each period, Ill here assume that the cash ow occurs at time , although other assumptions (such as at mid-year) are certainly possible. This leads to the following (2.11) Notation (cash ows while in states). C (i) denotes the cash ow at time subject is in State #i at time . if the

Actuarial present values

We again now have the tools to calculate actuarial present values by the 3 approach: Sum up, over all times at which a cash ow might occur, the product of three terms: the probability that the cash ow occurs then; the amount of the cash ow; and the discounting from the time of the cash ow back to the present time (n). [Its also possible to compute the variance of the present value of cash ows, but Ill not address that here.] We have at hand all the ingredients to compute the actuarial present value of the cash ows, for being in a state, in Notation 2.11 for a subject now in State #s at time n: the times at which the ow can occur are times n + k for k 0; (s,i) the probability of being in State #i at that time is k Qn ; the amount of the cash ow at that time is denoted by n+k C (i) ; and the discounting from time n + k to time n is k vn . The computation is straightforward: (2.12) Theorem (actuarial present value of cash ows while in states). As in Notation 2.11, let C (i) denote the cash ow at time if the subject is in State #i at time . Suppose that the subject is now in State #s at time n. Then the actuarial present value, as seen from time n, of these cash ows is given by the tripleproduct summation (3) AP Vs@n (C ) =
(i) k=0

[k Q(s,i) ] [n+k C (i) ] [k vn ]. n

The general formula above is again a mess. Spreadsheet software can easily perform such summations, however, and in simple illustrative cases the computations are fairly straightforward.

2.3 Cash ows while in states

17

(2.13)

Example. Consider the simple example of the homogeneous Markov Chain in Example 1.17, with States numbered 1 and 2 and with Q= 0.4 0.6 . 0.8 0.2

Suppose that the subject is now (time n) in State #1 and that there is a cash ow of 1 for being in State #1 now or the next two times (n + 1 or n + 2). Finally, suppose that interest is constant at 25%, so that v=0.8 and k vn = (0.8)k . Note that the cash ow at time n is certain. The probability of the cash ow (1,1) at time n + 1 is Qn = Q(1,1) = 0.4. The probability of the cash ow at time (1,1) n + 2 is 2 Qn which is just the (1,1)-entry of Q2 and is easily found to be 0.64 (check this). The actuarial present value (as seen from now at time n) by the 3 is then (1)(1)(v 0 ) + (0.4)(1)(v 1 ) + (0.64)(1)(v 2 ) = 1.7296. Since this is a homogeneous Markov Chain, the value of course does not actually depend on the time n. (2.14) Example. Consider the preceding Example 2.13 again, but this time suppose that the interest rate is varying: 10% in the rst year from time n to time n + 1 and 15% the second year. Then discounting is accomplished by 0 vn = 1, 1 1 1 vn = 1.1 = 0.90909 and 2 vn = 1 vn 1.15 = 0.79051. The actuarial present value from the 3 is then (1)(1)(0 vn ) + (0.4)(1)(1 vn ) + (0.64)(1)(2 vn ) = 1.8696.

(2.15)

Example. Consider a CCRC with the usual four states as in Example 1.13. Suppose that the transition-probability matrices for a new entrant are as given in Section 3.1, and suppose that the only cash ow is 1 for a resident in Independent Living (State #1). Suppose that a resident is in Independent Living at time 5. Lets compute the actuarial present value of the cash ows using a constant interest rate of 25%. The cash ows can only occur at times 5 (which is certain), 6, 7, and 8. (Why?) The probabilities of the resident being in State #1 at the start of those (1,1) (1,1) years are: 1; Q5 = 0.3; 2 Q5 , which is the (1,1)-entry of Q5 Q6 and is easily (1,1) computed to be 0.08 (check this); and 3 Q5 , which is the (1,1)-entry of Q5 Q6 Q7 and is somewhat-less-easily computed to be 0.012 (check this). The 3 then computes the actuarial present value (as seen from now at time 5) as (1)(1)v 0 + (0.3)(1)v 1 + (0.08)(1)v 2 + (0.012)(1)v 3 = 1.2973.

18 Chapter 2 Cash ows and their actuarial present values (2.16) Example. Consider a driver-ratings model in which drivers move among the two classications Preferred and Standard at the end of each year. Each year: 60% of Preferred are reclassied as Preferred and 40% as Standard; and 70% of Standard are reclassied as Standard and 30% as Preferred. Suppose that the insurer decides to provide a refund of $100 now at the start of the year to each Preferred driver and to continue to do so at the start of each year so long as the driver continually remains classied as Preferred; lets nd the actuarial present value of the current and future payments using 25% interest for one currently Preferred driver. Let Preferred be State #1 and Standard be State #2. Then the transitionprobability matrix Q for this homogeneous Markov Chain is 0.6 0.4 . 0.3 0.7 The 3 gives the desired actuarial present value as 100 k=0 k P0 v k , since (1) denotes the probability of remaining in State #1 from time 0 through time k P0 (1) k. But by Theorem 1.19 and Notation 1.14, k P0 = (Q(1,1) )k = (0.6)k . Thus the actuarial present value is 100
k=0 (1)

1 k (0.6) ( ) = 100 1.25

k

k=0

(0.48)k =

100 = $192.31. 1 0.48

Problems
1. Consider a CCRC as in Example 2.15. Suppose that a subject is in Independent Living (State #1) at time 4 and then in States #2, #2, #1, #3, and #4 at times 5, 6, 7, 8, and 9, respectively. Using 5% interest, nd the present value of cash ows of 10 for any time the subject is in State #1 and 30 for any time the subject is in State #3. [Note that no probability is involved here.] [Answer: 43.319.] 2. Consider a homogeneous Markov Chain with two states and transition probability matrix as in Example 2.13. The subject is now in State #2 at time 3. There are possible cash ows of 1 for being in State #2 at times 3, 4, or 5. Find the actuarial present value of these cash ows using 25% interest. [Answer: 1.4928.] 3. Solve Problem 2 again, but this time assume that the interest rates for the three future years are, in order, 10%, 15%, and 20%. [Answer: 1.5929.] 4. Consider a CCRC model with transition probabilities from the Illustrative Matrices in Section 3.1 as in Example 2.15. The subject is in Independent Living at time 5.

2.4 Benet premiums and reserves

19

There is a possible cash ow of 1 at the start of each future period the resident is Temporarily in the Health Center. Find the actuarial present value of these cash ows, using 25% interest. [Answer: 0.21734.] 5. Solve Problem 4 again, but this time assume that the interest rate from time n to time n + 1 is 0.05|n 4|. [Answer: 0.26877.] 6. As in Example 1.5, consider a driver-ratings model in which drivers move among the classications Preferred, Standard, and Substandard at the end of each year. As in Problem 4 of Section 1.2, each year: 60% of Preferreds are reclassied as Preferred, 30% as Standard, and 10% as substandard; 50% of Standards are reclassied as Standard, 30% as Preferred, and 20% as Substandard; and 60% of Substandards are reclassied as Substandard, 40% as Standard, and 0% as Preferred. A driver now Preferred at the start of the year will receive a premium refund of 100 now and at the start of each year so long as the driver remains continuously in the Preferred classication. Find the actuarial present value of these refunds, using 25% interest. [Answer: 192.31.]

2.4 Benet premiums and reserves

Youre probably already familiar with benet premiums for insurance or annuity policiespremiums determined by the Equivalence Principle that, at the time of issue of the policy, the actuarial present value of the premiums should equal the actuarial present value of the benets. This principle can be applied in the context of multi-state transition models. (2.17) Example (benet premiums). Consider a CCRC with the usual four states as in Example 1.13, with the transition-probability matrices as given in Section 3.1 and the cash ows at transitions as given in Section 3.2. Suppose that a resident is in Independent Living (State #1) at time 5. Using 25% interest, lets compute the benet premium to be paid at the start of each period in which the resident is in Independent Living in order to nance the future cash ows at transition from Independent Living to Permanently in the Health Center. Example 2.10 computed the actuarial present value, for a resident in Independent Living at time 5, of the cash ows at transition from Independent Living to Permanently in the Health Center, using 25% interest. The result was 17.246. Example 2.15 computed the actuarial present value, for a resident in Independent Living at time 5, of a cash ow of 1 whenever the resident is in Independent Living, using 25% interest. The result was 1.2973. If a premium P is paid instead of the cash ow of 1, the actuarial present value will be 1.2973P . To determine the benet premium, we use the Equivalence Principle and require 1.2973P = 17.246, yielding P = 13.294 for the benet premium.

20 Chapter 2 Cash ows and their actuarial present values

Benet reserves
Youre also probably already familiar with benet reserves: the actuarial present valueat the time the policy was issued or at some later time for a person still insuredof the present value of the future loss (benets out minus benet premiums in). The same concept is relevant in the context of multi-state transition models. (2.18) Example (benet reserves). Consider the preceding Example 2.17 again, with the same assumptions. Suppose that at time 6 the resident is in Temporarily in the Health Center (State #2). Lets calculate the benet reserve at this point. First we need the actuarial present value, as seen from time 6 with the subject in State #2, of the future cash ows for transition from State #1 to State #3. This can only happen (why?) if the resident is in State #1 at time 7 (2,1) (with probability Q6 = 0.1) and then transitions to State #3 (with probability (1,3) Q7 = 0.3) at time 8. Since the cash ow for that transition is 8 C (1,3) = 73, the actuarial present value is [(0.1)(0.3)](73)v 2 = 1.4016. Next we need the actuarial present value, as seen from time 6 with the subject in State #2, of the future cash ows of the benet premium 13.294 for being in State #1. The subject can only be in State #1 at times 7 and 8, with respective (2,1) (2,1) probabilities Q6 = 0.1 and 2 Q6 = 0.015 (check these). So the actuarial present value of the premiums is (0.1)(13.294)v + (0.015)(13.294)v 2 = 1.1911. This makes the benet reserve (as seen from time 6 with the resident in Temporarily in the Health Center) equal 1.4016 1.1911 = 0.2105.

Problems
1. Consider a homogeneous Markov Chain with two states and transition probability matrix as in Example 2.13. The subject is now in State #2 at time 3. As in Problem 2 of Section 2.2, there are possible cash ows of + 1 for transition from State #2 at time to State #1 at time + 1, for 5. A benet premium P will be paid at each of the times 3, 4, and 5, provided that the subject is in State #2 at that time (see Problem 2 of Section 2.3). Find P using 25% interest. [Answer: 2.9140.] 2. Solve Problem 1 again, but this time assume that the interest rates for the three future years are, in order, 10%, 15%, and 20%. [Answer: 3.2556.] 3. Consider a CCRC model with transition probabilities given by the Illustrative Matrices in Section 3.1. As in Problem 4 of Section 2.2, a resident is in Independent Living at time 5, and is subject to cash ows resulting from future transitions from Temporarily in the Health Center to Permanently in the Health Center, with the values given by the Illustrative Cash Flows in Section 3.2. A benet premium P will be paid at the start of each future period in which the resident is Temporarily in the Health Center (see Problem 4 of Section 2.3). Find P using 25% interest.

2.4 Benet premiums and reserves

21

[Answer: 20.137.] 4. Solve Problem 3 again, but this time assume that the interest rate from time n to time n + 1 is 0.05|n 4|. [Answer: 22.443.] 5. Consider again the situation in Problem 1. Given that the subject is in State #2 at time 4, nd the benet reserve. [Answer: 0.43416.] 6. Consider again the situation in Problem 3. Given that the resident is in Independent Living at time 6, nd the benet reserve. [Answer: 0.6518.]

3
An illustrative non-homogeneous Markov Chain
This chapter presents a set of illustrative transition-probability matrices and cash ows for use in examples and problems.

3.1 Illustrative transition-probability matrices

Consider a non-homogeneous Markov Chain with four states numbered 1, 2, 3, 4. The transition-probability matrices given below have 0 or 1 in certain positions so that the models make sense for disability models as in Example 1.11 and Continuing Care Retirement Community models as in Example 1.13. The other probabilities have been chosen arbitrarily and of course are unlikely to be appropriate for real-life situations. Transition-probability matrices are given at times 0, 1, 2, 3, 4, 5, 6, and 7, with the same matrix for all times n 8; this nal matrix is chosen so that the subject is certain to reach State #4 by time 9 and then remain there forever. 0.80 0.10 0.05 0.05 0.20 0.60 0.10 0.10 Q0 = , 0 0 0.80 0.20 0 0 0 1 0.60 0.20 Q2 = 0 0 0.40 0.10 Q4 = 0 0 0.15 0.15 0.10 0.40 0.25 0.15 , 0 0.60 0.40 0 0 1 0.20 0.20 0.20 0.30 0.30 0.30 , 0 0.40 0.60 0 0 1 0.70 0.20 Q1 = 0 0 0.50 0.20 Q3 = 0 0 0.30 0.10 Q5 = 0 0 0.15 0.50 0 0 0.20 0.30 0 0 0.20 0.20 0 0 0.10 0.05 0.20 0.10 , 0.70 0.30 0 1 0.20 0.10 0.35 0.15 , 0.50 0.50 0 1 0.30 0.20 0.40 0.30 , 0.30 0.70 0 1 22

3.2 Illustrative cash ows upon transitions

23

0.20 0.20 0.30 0.30 0.10 0.10 0.40 0.40 Q6 = , 0 0 0.20 0.80 0 0 0 1

0.10 0.10 0.30 0.50 0.05 0.05 0.30 0.60 Q7 = , 0 0 0.10 0.90 0 0 0 1 0 0 0 1 0 0 0 1 and, for n 8, Qn = . 0 0 0 1 0 0 0 1

3.2 Illustrative cash ows upon transitions

This section presents some illustrative cash ows upon transitions between states in the non-homogeneous Markov Chain described in Section 3.1. The particular values for the cash ows are not intended to be meaningfulrather they were chosen to be easily distinguishable from one another so that you can see from where values come in Examples. For convenience in displaying the values, Ive entered the cash ow +1 C (i,j) that occurs at time + 1 as the (i,j)-entry of a matrix +1 C. 1 5 1C = 0 0 21 25 3C = 0 0 41 45 5C = 0 0 61 65 7C = 0 0 4 11 12 13 14 8 15 16 17 18 , 2C = , 10 0 0 19 20 0 0 0 0 0 22 23 24 31 32 33 34 26 27 28 35 36 37 38 , 4C = , 0 29 30 0 0 39 40 0 0 0 0 0 0 0 42 43 44 51 52 53 54 46 47 48 55 56 57 58 , 6C = , 0 49 50 0 0 59 60 0 0 0 0 0 0 0 62 63 64 71 72 73 74 66 67 68 75 76 77 78 , 8C = , 0 69 70 0 0 79 80 0 0 0 0 0 0 0 0 0 0 81 0 0 0 82 and, for 8, +1 C = . 0 0 0 83 0 0 0 0 2 6 0 0 3 7 9 0