The Journal of Positive Psychology

Dedicated to furthering research and promoting good practice

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/rpos20

Predicting individual response to a web-based

positive psychology intervention: a machine
learning approach

Amanda C. Collins, George D. Price, Rosalind J. Woodworth & Nicholas C.

Jacobson

To cite this article: Amanda C. Collins, George D. Price, Rosalind J. Woodworth & Nicholas
C. Jacobson (2023): Predicting individual response to a web-based positive psychology
intervention: a machine learning approach, The Journal of Positive Psychology, DOI:
10.1080/17439760.2023.2254743

To link to this article: https://doi.org/10.1080/17439760.2023.2254743

View supplementary material

Published online: 03 Sep 2023.

Submit your article to this journal

View related articles

View Crossmark data

This article has been awarded the Centre

for Open Science 'Open Data' badge.

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=rpos20
THE JOURNAL OF POSITIVE PSYCHOLOGY
https://doi.org/10.1080/17439760.2023.2254743

Predicting individual response to a web-based positive psychology intervention:

a machine learning approach
a,b,c a,d e a,d,f,g
Amanda C. Collins , George D. Price , Rosalind J. Woodworth and Nicholas C. Jacobson
a
Center for Technology and Behavioral Health, Geisel School of Medicine, Dartmouth College, Lebanon, NH, USA; bDepartment of Psychiatry,
Dartmouth-Hitchcock Medical Center, Lebanon, NH, USA; cDepartment of Psychology, Mississippi State University, Starkville, MS, USA;
d
Quantitative Biomedical Sciences Program, Dartmouth College, Lebanon, NH, USA; eIWK Health Centre, Halifax, Canada; fDepartment of
Psychiatry, Geisel School of Medicine, Dartmouth College, Lebanon, NH, USA; gDepartment of Biomedical Data Science, Geisel School of
Medicine, Dartmouth College, Lebanon, NH, USA

ABSTRACT ARTICLE HISTORY

Positive psychology interventions (PPIs) are effective at increasing happiness and decreasing Received 22 September 2022
depressive symptoms. PPIs are often administered as self-guided web-based interventions, but Accepted 13 July 2023
not all persons benefit from web-based interventions. Therefore, it is important to identify whether KEYWORDS
someone is likely to benefit from web-based PPIs, in order to triage persons who may not benefit Positive psychology;
from other interventions. In the current study, we used machine learning to predict individual depression; machine
response to a web-based PPI, in order to investigate baseline prognostic indicators of likelihood of learning; digital intervention
response (N = 120). Our models demonstrated moderate correlations (happiness: rTest = 0.30 ± 0.09;
depressive symptoms: rTest = 0.39 ± 0.06), indicating that baseline features can predict changes in
happiness and depressive symptoms at a 6-month follow-up. Thus, machine learning can be used
to predict outcome changes from a web-based PPI and has important clinical implications for
matching individuals to PPIs based on their individual characteristics.

Happiness is often emphasized as a goal to strive toward two outcomes, demonstrating small to medium effect sizes
given that it is typically associated with increased posi­ (Bolier et al., 2013; Carr et al., 2021). Given that traditional
tive emotions, life satisfaction, and a meaningful life interventions for depression, including cognitive beha­
(Diener et al., 2009; Myers & Diener, 2018; Seligman vioral therapy, target enhanced negative biases, PPIs can
et al., 2005). In addition, happier individuals often have serve as a standalone or add-on intervention to target
more social support, are more successful, and have fewer depressed individuals’ reduced positive biases and increase
physical and psychological symptoms (Cohn et al., 2009; happiness. Seligman et al. (2005) created a series of five
Lyubomirsky et al., 2005; Myers & Diener, 2018). Indeed, web-based exercises aimed to increase overall happiness
previous research has suggested that a happiness- with common themes of (1) gratitude, (2) identifying posi­
success link exists. Specifically, a reciprocal relationship tivity in one’s daily life, (3) reflecting on one’s ‘best self’, (4)
exists between happiness and success such that indivi­ identifying one’s signature strengths, and (5) using one’s
duals who experience happiness and positive affect are signature strengths in new ways. Overall, individuals
more likely to achieve goals and be successful or accom­ demonstrated improvements in happiness and reductions
plished, thus resulting in them feeling happier (Boehm & in depressive symptoms after one week, with two of the PPI
Lyubomirsky, 2008; Lyubomirsky et al., 2005; Walsh et al., exercises resulting in improvements for 6-months. Thus,
2018). Thus, finding ways to capitalize on and improve their findings provided initial evidence that engaging in
happiness can provide several benefits for individuals’ a PPI for one week can lead to improvements in well-being.
personal life (e.g. work), subjective well-being, and psy­ Potentially as important, however, is that participants com­
chological symptoms (e.g. depression). pleted these exercises independently via an online website
Positive psychology interventions (PPIs) have gained (i.e. they completed a web-based intervention). Moreover,
popularity over the past two decades due to their emphasis these findings are in line with meta-analytic findings indi­
on increasing happiness and reducing depressive symp­ cating that short-term (i.e. 1 week) self-guided PPIs are
toms (Bolier et al., 2013). Indeed, findings from two meta- effective in improving well-being and depressive symp­
analyses suggest that PPIs are overall effective for these toms (Bolier et al., 2013).

CONTACT Amanda C. Collins [email protected] Center for Technology and Behavioral Health, Dartmouth College, 46 Centerra
Parkway Suite 300, Office # 338S, Lebanon, NH 03766, USA
Supplemental data for this article can be accessed online at https://doi.org/10.1080/17439760.2023.2254743
© 2023 Informa UK Limited, trading as Taylor & Francis Group
2 A. C. COLLINS ET AL.

Web-based interventions predict treatment outcomes in depressed individuals.

However, these studies used machine learning to predict
Although PPIs can be delivered in a traditional in-person
treatment outcomes immediately after completion of
format, PPIs have gained popularity as a web-based
the intervention and not in the long-term after comple­
intervention (e.g. self-guided), given their increased
tion. Given that many individuals experience a relapse in
accessibility and the decreased cost to deliver them
their depressive symptoms within 6-months of complet­
(Bolier et al., 2013; Davies et al., 2020). Indeed, there
ing treatment (Delgadillo et al., 2018; Wojnarowski et al.,
are several advantages to web-based interventions
2019), it is also important to match individuals to the
over traditional in-person interventions. First, web-
most appropriate treatment, based on their demo­
based interventions can be delivered to larger popula­
graphics and current psychiatric presentation, to max­
tions and include fewer resources, including clinicians, to
imize the effects of treatment and reduce symptom
operate (Wilhelm et al., 2019). Second, they are often
relapse.
free or of low cost. Third, they can be completed on an
individual’s own time from their home instead of during
the traditional working day at a clinic, resulting in poten­ Rationale
tial solutions for low access areas (Davies et al., 2020;
Given the rise in popularity of PPIs and web-based inter­
Teachman, 2014; Woodworth et al., 2017). Fourth,
ventions, the purpose of the current study was to inves­
regarding PPIs specifically, web-based interventions
tigate if baseline features would predict individual
demonstrate efficacy for improving well-being and
changes in overall happiness and depressive symptoms
depressive symptoms; however, not all participants
for individuals who engaged in a week-long web-based
may benefit from a web-delivery format.
PPI. Furthermore, we investigated whether these base­
Web-based interventions aimed at targeting well-
line features could predict sustained changes over vari­
being and depressive symptoms demonstrate effective­
able periods of time, potentially providing valuable
ness (compared to control conditions) with small to
information for who is more or less susceptible to symp­
medium effect sizes (Carr et al., 2021; Cowpertwait &
tom relapse.
Clarke, 2013); however, approximately 21–33% of indivi­
To accomplish this aim, we utilized data from a publicly
duals do not complete the intervention (i.e. drop out)
available dataset (Woodworth et al., 2017, 2018) that aimed
(Cowpertwait & Clarke, 2013; Wantland et al., 2004). In
to replicate findings from Seligman et al. (2005). The origi­
addition, meta-analytic findings suggest that longer and
nal findings from this data indicated that all participants
more intensive in-person PPIs demonstrate larger effects
demonstrated improvements in their happiness and
than shorter, web-based interventions (Bolier et al.,
depressive symptoms when assessed one week after com­
2013; Carr et al., 2021; Sin & Lyubomirsky, 2009). Thus,
pleting an online intervention (Woodworth et al., 2017).
it is important to determine when it is appropriate to
Importantly, a rapid improvement occurred in symptoms
deliver a PPI online, particularly when considering acces­
after one week of the intervention, with minimal change in
sibility, and for whom do specific web-based interven­
symptoms observed in the subsequent follow-ups.
tions, including PPIs, work best for in order to assess for
However, findings from the multilevel models indicated
individuals who may be less likely to respond to a web-
that differences in the improvement of happiness and
based PPI and triage them to other interventions.
depressive symptoms over time could not be distinguished
between the three intervention and one placebo groups
(each group is described in more detail below). Thus, it may
Applying machine learning to interventions
be likely that the PPIs themselves did not lead to the
Fortunately, computational approaches, including improvements in symptoms given that individuals in the
machine learning, are commonly used in psychiatric neutral, therapeutic placebo group also demonstrated
research to answer this question and predict treatment improvements. Instead, it is more plausible that the ‘client
outcomes (Meinlschmidt et al., 2020; Price et al., 2022). factor’, or individual participant characteristics, influenced
The ability to predict treatment outcomes for individuals the degree to which an individual experienced improve­
who are seeking treatment is especially promising to ments in their happiness and depressive symptoms over
optimize treatment matches. Existing studies have uti­ time, as suggested by Woodworth et al. (2017).
lized machine learning approaches to predict individual Thus, to better understand who may respond better
changes in depressive symptoms following a digital to PPIs and demonstrate symptom improvements long-
intervention from participants’ baseline features term, we utilized machine learning to predict individual
(Hornstein et al., 2021; Jacobson & Nemesure, 2021; changes from baseline features (e.g. individual symp­
Pearson et al., 2019), demonstrating that it is feasible to toms and demographics) in happiness and depressive
 THE JOURNAL OF POSITIVE PSYCHOLOGY 3

symptoms at the end of a week-long PPI. Specifically, we Measures

investigated the changes in happiness and depressive
Authentic happiness inventory
symptoms from pretest to posttest, 1-week follow-up,
The Authentic Happiness Inventory (AHI) is a 24-item
1-month follow-up, 3-month follow-up, and 6-month
self-report measure that assesses for happiness and life
follow-up after completing the web-based PPI.
satisfaction (Park et al., 2010; Proyer et al., 2017). Items
Moreover, given that improvement in symptoms may
on the AHI consist of different statements ranging from
be due to ‘client factors’ rather than the specific ‘ingre­
1 (negatively worded) to 5 (positively worded). An exam­
dients’ of the PPIs themselves, we also examined which
ple item assesses one’s mood and ranges from 1 (‘I am
individual characteristics at baseline were most influen­
usually in a bad mood’) to 5 (‘I am usually in an unbelie­
tial in predicting improvements in happiness and
vably great mood’). Total scores are created by summing
depressive symptoms over time following completion
individual items, and higher scores indicate greater
of the PPI.
levels of happiness.

Method Center for epidemiologic studies depression scale

The Center for Epidemiologic Studies Depression Scale
Participants
(CES-D) is a 20-item self-report measure that assesses for
Participants were recruited through the Australian com­ depressive symptoms (Radloff, 1977). Items on the CES-D
munity via newspaper, radio station, television, and are scored on a 4-point Likert scale from 0 (‘rarely or
internet articles to participate in a ‘happiness study’. none of the time’) to 3 (‘most or almost all the time’).
Two-hundred and ninety-five (N = 295) participants Four items are positively worded, so we reverse coded
enrolled in the study and completed the pre-test (i.e. these items for the current analyses. Total scores are
baseline) measures (Mage = 43.76, SDage = 12.43; 39.66% created by summing individual items, and higher scores
postgraduate degree, 35.22% Bachelor’s degree; 46.10% indicate greater severity of depressive symptoms.
‘average’ income; 85.06% female). The CES-D has four subscales (Radloff, 1977; Shafer,
2006): depressed affect (DA), positive affect (PA), somatic
complaints (SC), and interpersonal problems (IP). The DA
Intervention subscale represents general symptoms of depression
Participants were assigned to one of four groups, includ­ and negative emotion (e.g. ‘I felt depressed). The PA
ing three intervention groups (Gratitude Visit, Signature subscale represents a lack of positive affect or anhedonia
Strengths, and Three Good Things), and one control group (e.g. ‘I was happy’), and all items in this subscale are
(Early Memories). These groups were based on the three reverse scored. The SC subscale represents physical
strongest interventions and the control group from symptoms of depression (e.g. ‘My sleep was restless’).
Seligman et al. (2005). Participants assigned to the The IP subscale represents difficulties interacting with
Gratitude Visit intervention group were instructed to other people or negative beliefs about other people (e.g.
write a one-page letter to one person from their past ‘People were unfriendly’).
who had been a positive influence but never been
thanked. Then, participants were instructed to deliver
Procedure
the letter to the person and reminisce on previous,
positive events together. Participants assigned to the The initial study was approved by the Tasmanian Social
Signature Strengths intervention group were instructed Sciences Research Ethics Committee and followed ethi­
to complete a questionnaire that would assess their top cal guidelines by the Australian Government National
five character strengths (Woodworth et al., 2017). Health and Medical Research Council (Woodworth
Participants then received an email containing these et al., 2017, 2018). Participants interested in the study
strengths and were instructed to read each description were instructed to visit the study website to enroll them­
and choose one strength to use in a different way selves. After giving their informed consent, participants
every day over the next week. Participants assigned to completed pre-test questionnaires, including their con­
the Three Good Things intervention group were tact information, demographics, AHI, and CES-D.
instructed to write three good things that happened Participants were then randomly assigned to one of
and provide an explanation as to why they happened the four groups described above by researchers,
every day. Participants assigned to the Early Memories received an email with initial instructions for their
control group were instructed to write their early mem­ assigned intervention prior to beginning the interven­
ories at the end of each day. tion (and after completing pre-test questionnaires), and
4 A. C. COLLINS ET AL.

then received a mid-week email to remind them of their reported correlative strength based on established
instructions. Both of these emails were individualized for thresholds for behavioral sciences research (r ≥0.3;
the intervention that the receiving participant was moderate association) (Cohen, 2013).
assigned.
After completion of the one-week intervention, all
Machine learning model introspection
participants received a general email thanking them for
their participation and instructing them to complete the To address the decreased interpretability of machine
follow-up questionnaires, which included the AHI, CES- learning models as compared to traditional statistical
D, and a question confirming that they completed the models, a recent methodological approach based upon
intervention. Participants also received this email one the Shapley values of game theory has been developed
week, one month, three months, and six months after (Arrow et al., 1953). SHAP (SHapley Additive
completing the intervention. Thus, participants could exPlanations) calculates the relative contribution of
complete the AHI and CES-D at baseline and five sepa­ a feature to the model’s outcomes predictions by per­
rate follow-up occasions. Participants did not receive turbing the values of the model’s input features and
compensation for their involvement in the study but assessing their respective influence on the model’s pre­
were entered into a drawing to receive a book voucher dictions. The result is a set of SHAP values that corre­
if they completed all follow-up questionnaires. spond to the relative magnitude and direction by which
a given feature influences the model’s prediction out­
comes (Lundberg & Lee, 2017; for a more detailed tutor­
Data analytic plan
ial on the implementation of SHAP in predicting
Machine learning modeling approach individual response to digital interventions, see Lekkas
A leave-one-out cross-validation (LOOCV) (Webb et al. (2021)). We used this approach to determine the
et al., 2011) approach (70%), with a completely held- top five most influential features for predicting change
out test set (30%) was developed and implemented in happiness and depressive symptoms.
in Python (v3.9) (Van Rossum & Drake, 2009) using
the scikit-learn package (for further detail on the
Results
implementation and utility of cross-validation within
machine learning see Lekkas et al. (2021)). We used We used a machine learning approach with a completely
an Extreme Gradient Boosted Tree (xgboost) (Ramraj held-out test set to assess for change in happiness and
et al., 2016) regressor, a well-documented machine depressive symptoms at posttest, 1-week follow-up,
learning algorithm for detecting mental health con­ 1-month follow-up, 3-month follow-up, and 6-month fol­
structs (Jacobson & Chung, 2020; Jacobson et al., low-up, respectively. The modeling approach found
2019a, 2019b; Sharma & Verbeke, 2020), to detect 1) a moderate correlation for both change in total happiness
change in total AHI and 2) change in total CES-D and depressive symptoms at 6-month follow-up but found
from pretest to posttest (7 days after pretest), weak correlations for both change in total happiness and
1-week follow-up (14 days after pretest), 1-month fol­ depressive symptoms at posttest, 1-week follow-up,
low-up (38 days after pretest), 3-month follow-up (98 1-month follow-up, and 3-month follow-up (i.e. r ﹤0.3;
days after pretest), and 6-month follow-up (189 days see Table 1). We examined variance of the outcome at
after pretest), respectively, resulting in 10 separate the five different timepoints and found that, generally,
models. We used demographic variables (age, the variance for change in happiness and depressive symp­
income, education level, and sex), intervention toms increased as time from baseline increased.1 Thus, the
group, elapsed days, individual AHI items, individual reduced variability in score distribution at the earlier time
CES-D items, and the CES-D subscales as input fea­ points may reflect weaker signal in the outcome metric and
tures for the xgboost model, resulting in the inclusion subsequently worse performance by the model. Given the
of 54 input features. The features were subsequently weak correlations, we did not examine the most influential
scaled, bounding their values from [0,1], as feature features in the posttest, 1-week follow-up, 1-month follow-
scaling has been shown to increase model perfor­ up, and 3-month follow-up models for the present
mance and efficiency (Shahriyari, 2019). We assessed analyses.
model performance across three random seeds and One-hundred and twenty-participants (N = 120) com­
reported the average and standard deviation of cor­ pleted the 6-month follow-up and were included in our
relative strength (r) and the Coefficient of analyses. Overall, participants’ happiness scores
Determination (R2) for each questionnaire outcome increased (Mbaseline = 70.60, SDbaseline = 13.53; Mfollow-up
and time point of interest. We interpreted and = 75.83, SDfollow-up = 14.53) and their depressive
 THE JOURNAL OF POSITIVE PSYCHOLOGY 5

Table 1. Modeling performance for changes in happiness and depressive symptoms.

Pretest to Posttest Pretest to 1-Week Pretest to 1-Month Pretest to 3-Month Follow- Pretest to 6-Month Follow-
Change Follow-Up Change Follow-Up Change Up Change Up Change
r ± SD r ± SD r ± SD
Modeling r ± SD (Validation r ± SD (Validation r ± SD (Validation r ± SD r ± SD r ± SD r ± SD
Approach (Test Set) Set) (Test Set) Set) (Test Set) Set) (Test Set) (Validation Set) (Test Set) (Validation Set)
Happiness 0.12 ± 0.12 0.13 ± 0.05 0.21 ± 0.04 0.13 ± 0.03 0.04 ± 0.05 0.07 ± 0.07 0.14 ± 0.10 0.11 ± 0.05 0.30 ± 0.09 0.32 ± 0.04
(AHI)
Depressive 0.14 ± 0.06 0.16 ± 0.08 0.13 ± 0.04 0.06 ± 0.07 0.01 ± 0.03 0.05 ± 0.05 0.21 ± 0.03 0.20 ± 0.09 0.39 ± 0.06 0.37 ± 0.03
Symptoms
(CES-D)
XGBoost Regressor performance for detecting change in Authentic Happiness Inventory (AHI) and Center for Epidemiologic Studies – Depression Scale (CES-D)
total scores from pretest to posttest, 1-month follow-up, and 6-month follow-up. Values are reported for the validation and held-out test set(s) for the three
model types as average correlation ± standard deviation.

symptoms (Mbaseline = 15.03, SDbaseline = 11.36; Mfollow-up considering the top five most influential features for
= 12.83, SDfollow-up = 12.93) decreased from baseline to the model’s predictions, lower scores of Bothered
the 6-month follow-up. Moreover, our sample’s charac­ (CES-D) and Crying (CES-D) and higher scores of
teristics at both baseline and the 6-month follow-up Keeping Score (AHI), Hopeful (CES-D), and Enthusiastic
demonstrated significant heterogeneity, with depressive (AHI) predicted decreases in depressive symptoms 6
symptoms varying from non-clinical to mild severity of months after completion of the intervention (see
depressive symptoms (Radloff, 1977; Santor et al., 1995). Figure 1). No baseline demographic features were
See Table S1 for a full breakdown of participant observed amongst the top five most influential
characteristics. features.

Happiness modeling performance & feature Discussion

influence
In the current study, we utilized machine learning to
Using baseline demographic variables, individual AHI predict individual changes in happiness and depres­
and CES-D items, and CES-D subscales incorporated sive symptoms from baseline characteristics after
into an xgboost regressor modeling approach, we were a one-week web-based PPI. We found a moderate
able to moderately detect change in AHI total score from correlation in predicting change in happiness and
baseline to 6-month follow-up (rTest = 0.30 ± 0.09, depressive symptoms between baseline and the
rValidation = 0.32 ± 0.04; see Table 1). We assessed feature 6-month follow-up. However, the present approach
importance and directionality via SHAP. When consider­ found weak correlations for the models assessing
ing the top five most influential features for the models change in happiness and depressive symptoms at
predictions, higher scores of Successful (AHI), Enthusiastic posttest, 1-week follow-up, 1-month follow-up, and
(AHI), and Accomplished (AHI) and lower scores of Sleep 3-month follow-up.
(CES-D), predicted increases in happiness 6 months after Taken together, our findings indicate that ‘client fac­
completion of the intervention. Although age emerged tors’, or individual characteristics, were only able to
as a significant feature, there was no clear directionality moderately predict improvements in happiness and
to interpret how age influences overall happiness over depressive symptoms 6 months after completing the
time (see Figure 1). PPIs, and not immediately after the PPI. Thus, although
it is possible that the specific ingredients of each PPI
contributed to the long-term improvements in happi­
Depressive symptoms modeling performance &
ness and depressive symptoms, it may be more likely
feature influence
that the individual characteristics contributed more to
Using baseline demographic variables, individual AHI the improvements at 6 months, regardless of interven­
and CES-D items, and CES-D subscales incorporated tion type, as suggested by Woodworth and colleagues
into an xgboost regressor modeling approach, we were (2017) in their original findings. We discuss the most
able to moderately detect change in CES-D total score influential features in further detail below to give better
from baseline to 6-month follow-up (rTest = 0.39 ± 0.06, insight into what individual characteristics may be more
rValidation = 0.37 ± 0.03; see Table 1). We assessed feature important in increasing happiness and decreasing
importance and directionality via SHAP. When depressive symptoms.
6 A. C. COLLINS ET AL.

Figure 1. Top feature importance for change in happiness and depressive symptoms from pretest to 6-month follow-up. The machine
learning model(s) actual versus predicted values plotted with respective correlative strength and the top five most influential features
for the models predictions. The dot color on the SHAP plots correspond to the value of the listed feature, and position of the dot on
the x-axis corresponds to the relative impact on the model prediction. For example, a low score on the AHI Successful item strongly,
positively influences the model’s prediction of lower happiness at the 6-month follow-up. (a) Baseline demographic variables,
individual AHI and CES-D items, and CES-D subscales to predict change in AHI total score from baseline to 6-month follow-up. (b)
Baseline demographic variables, individual AHI and CES-D items, and CES-D subscales to predict change in CES-D total score from
baseline to 6-month follow-up. AHI = Authentic Happiness Inventory; CES-D = Center for Epidemiological Studies Depression Scale.

Happiness modeling performance & feature Moreover, individuals who are able to use their
influence resources to work toward goals are more likely to
Given that our results indicated a moderate correla­ benefit from PPIs (Sheldon & Lyubomirsky, 2021).
tion for detecting long-term change in happiness Thus, we find similar results here, which suggest
from baseline features, we implemented SHAP to that those who feel more successful, accomplished,
assess the top five most influential features in the and enthusiastic (i.e. motivated) compared to other
model’s prediction of outcome change. When exam­ people are more likely to experience sustained
ining AHI items that influenced outcome change for increases in happiness for 6 months after a web-
happiness, Successful, Enthusiastic, and Accomplished based PPI.
emerged as the top predictors of all AHI items. When examining CES-D items that influenced out­
Specifically, higher scores of Successful, Enthusiastic, come change for happiness, Sleep emerged as the top
and Accomplished at baseline predicted increases in predictor of all CES-D items. Specifically, higher scores of
happiness 6 months after completion of the inter­ restless sleep at baseline predicted decreases in happi­
vention. This is in line with prior research indicating ness 6 months after completion of the intervention.
a happiness-success link (Boehm & Lyubomirsky, Sleep has been found to be an important individual
2008; Lyubomirsky et al., 2005; Walsh et al., 2018). factor in maximizing treatment benefit (Dolsen et al.,
 THE JOURNAL OF POSITIVE PSYCHOLOGY 7

2017). Given that sleep impacts learning and memory, including that they are doing well in life (if they were
individuals who experienced poorer sleep may not have to keep score), and are and feel that they are more
been as engaged in the PPI and thus not experienced its enthusiastic about doing anything are more likely to
full benefits. Indeed, our results provide support for this: experience sustained decreases in depressive symptoms
individuals who feel like their sleep is restless at baseline for 6 months after a web-based PPI. Interestingly,
are more likely to experience decreases in happiness 6 Enthusiastic was the only item that emerged as a top
predictor in both models, suggesting that individuals
months after a web-based PPI.
who are more enthusiastic, and thus motivated, in their
life may be the most likely to reap the benefits of web-
based PPIs and experience increases in happiness and
Depressive symptoms modeling performance &
decreases in depressive symptoms.
feature influence
Given that our results indicated a moderate correlation
Clinical implications
for detecting long-term change in depressive symptoms
from baseline features, we were able to assess for feature Our findings indicate baseline features can predict out­
importance for outcome change. When examining CES- come changes following a web-based PPI, providing
D items that influenced outcome change for depressive important clinical implications. Given that individuals
symptoms, Bothered, Crying, and Hopeful emerged as the overall experienced sustained increases in happiness
top predictors of all CES-D items. Specifically, lower and decreases in depressive symptoms after 6 months,
scores of Bothered and Crying and higher scores of and, coupled together with the original findings indicat­
Hopeful predicted decreases in depressive symptoms 6 ing significant changes both immediately (i.e. 1 week)
months after completion of the intervention. The and long-term (i.e. 6 months) in happiness and depres­
impaired disengagement hypothesis suggests that sive symptoms (Woodworth et al., 2017), it appears that
depressed individuals have difficulties disengaging the web-based PPI is feasible for treating individuals
from negative information (Collins et al., 2021; Koster who experience reduced happiness in their life and/or
et al., 2011), so, in line with this hypothesis, our findings individuals with depressive symptoms. Moreover, 48
suggest that individuals who are able to disengage from participants (40%) endorsed clinically significant depres­
negative information at baseline experienced decreases sive symptoms (i.e. equal to or above the CES-D cutoff of
in depressive symptoms later on. Moreover, this may be 16; Radloff, 1977), providing important clinical implica­
due to individuals being asked to engage more with tions for depression.
happiness during the PPI, resulting in them redirecting However, it is important to note that we did not
their attention from negative to positive information. investigate other factors that could have resulted in
Thus, individuals who feel less bothered by things at long-term changes in happiness and depressive symp­
baseline are more likely to experience sustained toms, including medication, therapy, engagement in the
decreases in depressive symptoms for 6 months after intervention after the immediate study period (e.g. 1
a web-based PPI. week) ended. In addition, original findings from
Individuals who experience elevated depressive Woodworth et al. (2017) indicated that all participants
symptoms experience more negative and less positive experienced changes in happiness and depressive symp­
affect (Bylsma et al., 2008). Crying and Hopeful represent toms, including individuals in the neutral, placebo
items from the negative affect and positive affect sub­ group. Thus, it is likely that these ‘client factors’ may be
scale, respectively (Radloff, 1977), so our findings indi­ more effective than the PPI ingredients. Moreover, it may
cate that individuals who experience lower negative be that the interventions utilized in the current study are
affect (i.e. fewer crying spells) and increased positive helpful for people to establish habits to help improve
affect (i.e. more hopeful about the future) at baseline their mood, and these habits may become more fre­
experienced sustained decreases in depressive symp­ quent for individuals with these specific characteristics
toms for 6 months after a web-based PPI. (e.g. individuals who are more accomplished and moti­
When examining AHI items that influenced outcome vated), so we present our clinical implications below and
change for depressive symptoms, Keeping Score and focus on who would specifically benefit web-based PPIs,
Enthusiastic emerged as the top predictors of all AHI rather than the implications of the specific ingredient of
items. Specifically, higher scores of Keeping Score and each PPI.
Enthusiastic predicted lower levels of depressive symp­ Further investigation of feature importance revealed
toms 6 months after completion of the intervention. As specific happiness beliefs and depressive symptoms that
noted above, individuals who feel more successful, had a significant impact on outcome changes. First,
8 A. C. COLLINS ET AL.

those who experienced higher levels of happiness at considered a strength of the current study. The ability
baseline, including feeling enthusiastic, accomplished, to accurately predict symptom changes at the 6-month
and successful, were more likely to experience increased follow-up is particularly important given that individuals
happiness and decreased depressive symptoms. Indeed, typically experience a relapse in depressive symptoms
these specific characteristics are often associated with within that time period. Thus, our findings suggest that it
increased goal motivation and self-esteem, so they may may be possible to identify baseline features that
have been more motivated to complete the intervention increase treatment efficacy for a web-based PPI while
and capitalize on their preexisting higher levels of hap­ decreasing the likelihood for depressive symptom
piness, in line with prior research (Lyubomirsky et al., relapse.
2005; Sheldon & Lyubomirsky, 2021). Thus, individuals Although our findings provide important information
who may feel depressed but are able to experience as to who may benefit from a web-based PPI, the current
happiness in their daily lives and are motivated to sample demonstrated non-clinical levels of depressive
increase their happiness levels may be a good fit for symptoms. Thus, no clear conclusions can be drawn as
week-long, web-based interventions, including PPIs. to how moderately or severely depressed individuals
Second, individuals who reported more sleep difficul­ would respond to this intervention. Moreover, some
ties at baseline experienced less happiness overtime. individuals with clinical depression actually devalue or
Sleep plays a big role in cognitive flexibility, learning, avoid positivity and happiness (Jordan et al., 2021; Winer
and memory (Stickgold & Walker, 2013) and has been & Salem, 2016), so it is possible that PPIs may not address
found to be associated with treatment compliance the underlying devaluation of positivity. Future research
(Dolsen et al., 2017). Thus, individuals who have more should investigate how these individuals respond to
sleep difficulties may not benefit from a week-long, web- web-based PPIs, which emphasize valuing happiness,
based interventions, including PPIs. Instead, they may in order to maximize treatment match for them. In addi­
benefit from targeting their sleep difficulties first before tion, we did not examine other factors (e.g. intervention
engaging in a self-guided online intervention targeting engagement) that may influence changes in partici­
happiness to increase treatment compliance and max­ pants’ symptoms as this was not collected in the original
imize the potential benefits. dataset. Given that the current intervention was only
Third, individuals who are easily bothered by things, active for one week, it is important to investigate with
have crying spells, and do not feel hopeful about the future research whether participants continued enga­
future may not benefit from a short, web-based inter­ ging with the intervention after the study ended as this
vention, including PPIs. In particular, being easily both­ may give further insight into how and why some parti­
ered and having crying spells are often associated with cipants experienced long-term improvements in their
increased negative affect. It is likely that these indivi­ symptoms.
duals may benefit from a more traditional therapy, Our sample size was also relatively small for machine
including cognitive behavioral therapy, rather than learning analyses, so we utilized different approaches to
a self-guided online intervention. Thus, it may be bene­ handle this potential issue. Specifically, we used LOOCV
ficial to triage individuals presenting with these symp­ and completely separated our training data and test
toms to CBT instead of a web-based PPI to better data, which is considered sufficient at any sample size
address their negative biases. to provide unbiased estimates (Vabalas et al., 2019;
Varma & Simon, 2006).

Strengths and limitations

Conclusion
One important strength of the current study is the novel
use of a machine learning approach coupled with feature Our current findings provide evidence that machine
introspection via SHAP to investigate what baseline fea­ learning can use baseline features to predict changes
tures predicted outcome changes following a web-based in happiness and depressive symptoms following
PPI. Our findings provide initial evidence of who would a week-long, web-based PPI. Importantly, the baseline
respond well to this type of intervention, potentially max­ features can predict sustained changes long term (i.e.
imizing treatment match. Moreover, our findings could 6 months), which provides relevant clinical information
help inform clinicians’ decisions when choosing the best regarding depressive symptom relapse. Individuals
treatment for someone who experiences reduced happi­ who demonstrate positive views about themselves,
ness and elevated depressive symptoms. their past, and their future at baseline are more likely
The inclusion of multiple timepoints to model to experience sustained increases in happiness and
changes in happiness and depressive symptoms is decreases in depressive symptoms. Whereas
 THE JOURNAL OF POSITIVE PSYCHOLOGY 9

individuals who are less hopeful or have sleep difficul­ Shiffman, M. . . . Thrall, R. M. (1953). Contributions to the
ties may not respond well to the PPI and may benefit theory of games (AM-28), volume II. Princeton University
from a different intervention. Future work should Press; JSTOR. http://www.jstor.org/stable/j.ctt1b9x1zv
Boehm, J. K., & Lyubomirsky, S. (2008). Does happiness pro­
investigate other individual factors that may impact
mote career success? Journal of Career Assessment, 16(1),
how individuals respond to web-based PPIs. with 101–116. https://doi.org/10.1177/1069072707308140
a particular emphasis on how moderately or severely Bolier, L., Haverman, M., Westerhof, G. J., Riper, H., Smit, F., &
depressed individuals respond, in order to optimize Bohlmeijer, E. (2013). Positive psychology interventions: A
treatment match and increase patient compliance. meta-analysis of randomized controlled studies. BMC Public
Health, 13(1), 119. https://doi.org/10.1186/1471-2458-13-119
Bylsma, L. M., Morris, B. H., & Rottenberg, J. (2008). A
Note meta-analysis of emotional reactivity in major depressive
disorder. Clinical Psychology Review, 28(4), 676–691. https://
1. The variance of change in outcome measures are as doi.org/10.1016/j.cpr.2007.10.001
follows: 57.98, 58.57, 115.27, 123.36, and 123.81 for Carr, A., Cullen, K., Keeney, C., Canning, C., Mooney, O.,
AHI, and 64.78, 49.62, 78.14, 95.33, and 103.78 for CES- Chinseallaigh, E., & O’Dowd, A. (2021). Effectiveness of posi­
D, for posttest, 1-week follow-up, 1-month follow-up, tive psychology interventions: A systematic review and
3-month follow-up and 6-month follow-up, respectively. meta-analysis. The Journal of Positive Psychology, 16(6),
749–769. https://doi.org/10.1080/17439760.2020.1818807
Cohen, J. (2013). Statistical power analysis for the behavioral
Disclosure statement Sciences (0 ed.). Routledge. https://doi.org/10.4324/
No potential conflict of interest was reported by the author(s). 9780203771587
Cohn, M. A., Fredrickson, B. L., Brown, S. L., Mikels, J. A., &
Conway, A. M. (2009). Happiness unpacked: Positive emo­
Funding tions increase life satisfaction by building resilience.
Emotion, 9(3), 361–368. https://doi.org/10.1037/a0015952
This work was partially funded by the National Institute of Collins, A. C., Lass, A. N. S., Jordan, D. G., & Winer, E. S. (2021).
Mental Health (NIMH) and the National Institute Of General Examining rumination, devaluation of positivity, and depres­
Medical Sciences (NIGMS) under R01 MH123482-01. sive symptoms via community‐based network analysis.
Journal of Clinical Psychology, 77(10), 2228–2244. https://
doi.org/10.1002/jclp.23158
ORCID Cowpertwait, L., & Clarke, D. (2013). Effectiveness of web-based
psychological interventions for depression: A meta-analysis.
Amanda C. Collins http://orcid.org/0000-0002-8258-2272 International Journal of Mental Health and Addiction, 11(2),
George D. Price http://orcid.org/0000-0002-9164-4973 247–268. https://doi.org/10.1007/s11469-012-9416-z
Rosalind J. Woodworth http://orcid.org/0000-0002-5454- Davies, F., Shepherd, H. L., Beatty, L., Clark, B., Butow, P., &
659X Shaw, J. (2020). Implementing web-based therapy in routine
Nicholas C. Jacobson http://orcid.org/0000-0002-8832-4741
mental health care: Systematic review of health profes­
sionals’ perspectives. Journal of Medical Internet Research,
22(7), e17362. https://doi.org/10.2196/17362
Data availability statement Delgadillo, J., Rhodes, L., Moreea, O., McMillan, D., Gilbody, S.,
The data that support the findings of this study are openly Leach, C., Lucock, M., Lutz, W., & Ali, S. (2018). Relapse and
available in Journal of Open Psychology Data at https://doi. recurrence of common mental health problems after low
org/10.5334/jopd.35. intensity cognitive behavioural therapy: The WYLOW long­
itudinal cohort study. Psychotherapy and Psychosomatics, 87
(2), 116–117. https://doi.org/10.1159/000485386
Open Scholarship Diener, E., Napa Scollon, C., & Lucas, R. E. (2009). The evolving
concept of subjective well-being: The multifaceted nature of
happiness. In E. Diener (Ed.), Assessing well-being (Vol. 39, pp.
67–100). Springer Netherlands. https://doi.org/10.1007/978-
90-481-2354-4_4
This article has earned the Center for Open Science badge for
Dolsen, M. R., Soehner, A. M., Morin, C. M., Bélanger, L.,
Open Data. The data are openly accessible at https://doi.org/
Walker, M., & Harvey, A. G. (2017). Sleep the night before
10.5334/jopd.35
and after a treatment session: A critical ingredient for treat­
ment adherence? Journal of Consulting and Clinical
References Psychology, 85(6), 647–652. https://doi.org/10.1037/
ccp0000184
Arrow, K. J., Barankin, E. W., Blackwell, D., Bott, R., Dalkey, N., Hornstein, S., Forman-Hoffman, V., Nazander, A., Ranta, K., &
Dresher, M., Gale, D., Gillies, D. B., Glicksberg, I., Gross, O., Hilbert, K. (2021). Predicting therapy outcome in a digital
Karlin, S., Kuhn, H. W., Mayberry, J. P., Milnor, J. W., mental health intervention for depression and anxiety:
Motzkin, T. S., Von Neumann, J., Raiffa, H., Shapley, L. S., A machine learning approach. Digital Health, 7,
10 A. C. COLLINS ET AL.

205520762110606. https://doi.org/10.1177/ Psychological Medicine, 49(14), 2330–2341. https://doi.org/

20552076211060659 10.1017/S003329171800315X
Jacobson, N. C., & Chung, Y. J. (2020). Passive sensing of pre­ Price, G. D., Heinz, M. V., Nemesure, M. D., McFadden, J., &
diction of moment-to-moment depressed mood among Jacobson, N. C. (2022). Predicting symptom response and
undergraduates with clinical levels of depression sample engagement in a digital intervention among individuals
using smartphones. Sensors, 20(12), 3572. https://doi.org/ with schizophrenia and related psychoses. Frontiers in
10.3390/s20123572 Psychiatry, 13, 807116. https://doi.org/10.3389/fpsyt.2022.
Jacobson, N. C., & Nemesure, M. D. (2021). Using artificial 807116
intelligence to predict change in depression and anxiety Proyer, R. T., Gander, F., Wellenzohn, S., & Ruch, W. (2017). The
symptoms in a digital intervention: Evidence from authentic happiness inventory revisited: Addressing its psy­
a transdiagnostic randomized controlled trial. Psychiatry chometric properties, validity, and role in intervention
Research, 295, 113618. https://doi.org/10.1016/j.psychres. studies. Journal of Well-Being Assessment, 1(1–3), 77–96.
2020.113618 https://doi.org/10.1007/s41543-018-0006-0
Jacobson, N. C., Weingarden, H., & Wilhelm, S. (2019a). Digital Radloff, L. S. (1977). The CES-D scale: A self-report depression
biomarkers of mood disorders and symptom change. NPJ scale for research in the general population. Applied
Digital Medicine, 2(1), 3. https://doi.org/10.1038/s41746-019- Psychological Measurement, 1(3), 385–401. https://doi.org/
0078-0 10.1177/014662167700100306
Jacobson, N. C., Weingarden, H., & Wilhelm, S. (2019b). Using Ramraj, S., Uzir, N., Sunil, R., & Banerjee, S. (2016).
digital phenotyping to accurately detect depression Experimenting XGBoost algorithm for prediction and classi­
severity. Journal of Nervous & Mental Disease, 207(10), fication of different datasets. International Journal of Control
893–896. https://doi.org/10.1097/NMD.0000000000001042 Theory & Applications, 9(40), 651–662.
Jordan, D. G., Collins, A. C., Dunaway, M. G., Kilgore, J., & Santor, D. A., Zuroff, D. C., Ramsay, J. O., Cervantes, P., &
Winer, E. S. (2021). Negative affect interference and fear of Palacios, J. (1995). Examining scale discriminability in the
happiness are independently associated with depressive BDI and CES-D as a function of depressive severity.
Psychological Assessment, 7(2), 131–139. https://doi.org/10.
symptoms. Journal of Clinical Psychology, 77(3), 646–660.
1037/1040-3590.7.2.131
https://doi.org/10.1002/jclp.23066
Seligman, M. E. P., Steen, T. A., Park, N., & Peterson, C. (2005).
Koster, E. H. W., De Lissnyder, E., Derakshan, N., & De Raedt, R.
Positive psychology progress: Empirical validation of
(2011). Understanding depressive rumination from
interventions. American Psychologist, 60(5), 410–421.
a cognitive science perspective: The impaired disengage­
https://doi.org/10.1037/0003-066X.60.5.410
ment hypothesis. Clinical Psychology Review, 31(1),
Shafer, A. B. (2006). Meta-analysis of the factor structures of
138–145. https://doi.org/10.1016/j.cpr.2010.08.005
four depression questionnaires: beck, CES-D, Hamilton, and
Lekkas, D., Price, G., McFadden, J., & Jacobson, N. C. (2021). The
Zung. Journal of Clinical Psychology, 62(1), 123–146. https://
application of machine learning to online mindfulness inter­
doi.org/10.1002/jclp.20213
vention data: A primer and empirical example in compliance
Shahriyari, L. (2019). Effect of normalization methods on the
assessment. Mindfulness, 12(10), 2519–2534. https://doi.org/
performance of supervised learning algorithms applied to
10.1007/s12671-021-01723-4 HTSeq-FPKM-UQ data sets: 7SK RNA expression as
Lundberg, S. M., & Lee, S. I. (2017). A unified approach to a predictor of survival in patients with colon
interpreting model predictions. Advances in Neural adenocarcinoma. Briefings in Bioinformatics, 20(3), 985–994.
Information Processing Systems, 30, 4765–4774. https://doi.org/10.1093/bib/bbx153
Lyubomirsky, S., King, L., & Diener, E. (2005). The benefits of Sharma, A., & Verbeke, W. J. M. I. (2020). Improving diagnosis of
frequent positive affect: Does happiness lead to success? depression with XGBOOST machine learning model and
Psychological Bulletin, 131(6), 803–855. https://doi.org/10. a large biomarkers Dutch dataset (n = 11,081). Frontiers in
1037/0033-2909.131.6.803 Big Data, 3, 15. https://doi.org/10.3389/fdata.2020.00015
Meinlschmidt, G., Tegethoff, M., Belardi, A., Stalujanis, E., Oh, M., Sheldon, K. M., & Lyubomirsky, S. (2021). Revisiting the sustain­
Jung, E. K., Kim, H.-C., Yoo, S. S., & Lee, J. H. (2020). able happiness model and pie chart: Can happiness be
Personalized prediction of smartphone-based psychothera­ successfully pursued? The Journal of Positive Psychology, 16
peutic micro-intervention success using machine learning. (2), 145–154. https://doi.org/10.1080/17439760.2019.
Journal of Affective Disorders, 264, 430–437. https://doi.org/ 1689421
10.1016/j.jad.2019.11.071 Sin, N. L., & Lyubomirsky, S. (2009). Enhancing well-being and
Myers, D. G., & Diener, E. (2018). The scientific pursuit of alleviating depressive symptoms with positive psychology
happiness. Perspectives on Psychological Science, 13(2), interventions: A practice-friendly meta-analysis. Journal of
218–225. https://doi.org/10.1177/1745691618765171 Clinical Psychology, 65(5), 467–487. https://doi.org/10.1002/
Park, N., Park, M., & Peterson, C. (2010). When is the search for jclp.20593
meaning related to life satisfaction? Applied Psychology: Stickgold, R., & Walker, M. P. (2013). Sleep-dependent memory
Health and Well-Being, 2(1), 1–13. https://doi.org/10.1111/j. triage: Evolving generalization through selective processing.
1758-0854.2009.01024.x Nature Neuroscience, 16(2), 139–145. https://doi.org/10.
Pearson, R., Pisner, D., Meyer, B., Shumake, J., & Beevers, C. G. 1038/nn.3303
(2019). A machine learning ensemble to predict treatment Teachman, B. A. (2014). No appointment necessary: Treating
outcomes following an Internet intervention for depression. mental illness outside the therapist’s office. Perspectives on
 THE JOURNAL OF POSITIVE PSYCHOLOGY 11

Psychological Science, 9(1), 85–87. https://doi.org/10.1177/ G. I. Webb (Eds.), Encyclopedia of machine learning (pp. 600–601).
1745691613512659 Springer US.
Vabalas, A., Gowen, E., Poliakoff, E., Casson, A. J., & Hernandez- Wilhelm, S., Weingarden, H., Ladis, I., Braddick, V., Shin, J.,
Lemus, E. (2019). Machine learning algorithm validation with & Jacobson, N. C. (2019). Cognitive-behavioral therapy
a limited sample size. PLOS ONE, 14(11), e0224365. https:// in the digital age: Presidential address. Behavior
doi.org/10.1371/journal.pone.0224365 Therapy, 51(1), 1–14. https://doi.org/10.1016/j.beth.
Van Rossum, G., & Drake, F. L. (2009). Python 3 reference manual. 2019.08.001
CreateSpace. Winer, E. S., & Salem, T. (2016). Reward devaluation: Dot-probe
Varma, S., & Simon, R. (2006). Bias in error estimation when using meta-analytic evidence of avoidance of positive information
cross-validation for model selection. BMC Bioinformatics, 7(1), in depressed persons. Psychological Bulletin, 142(1), 18–78.
1–8. https://doi.org/10.1186/1471-2105-7-91 https://doi.org/10.1037/bul0000022
Walsh, L. C., Boehm, J. K., & Lyubomirsky, S. (2018). Does happiness Wojnarowski, C., Firth, N., Finegan, M., & Delgadillo, J. (2019).
promote career success? Revisiting the evidence. Journal of Career Predictors of depression relapse and recurrence after cogni­
Assessment, 26(2), 199–219. https://doi.org/10.1177/ tive behavioural therapy: A systematic review and
1069072717751441 meta-analysis. Behavioural and Cognitive Psychotherapy, 47
Wantland, D. J., Portillo, C. J., Holzemer, W. L., Slaughter, R., & (5), 514–529. https://doi.org/10.1017/S1352465819000080
McGhee, E. M. (2004). The effectiveness of web-based vs. Woodworth, R. J., O’Brien-Malone, A., Diamond, M. R., & Schüz, B.
non-web-based interventions: A meta-analysis of behavioral (2017). Web-based positive psychology interventions:
change outcomes. Journal of Medical Internet Research, 6(4), A reexamination of effectiveness. Journal of Clinical
e40. https://doi.org/10.2196/jmir.6.4.e40 Psychology, 73(3), 218–232. https://doi.org/10.1002/jclp.22328
Webb, G. I., Sammut, C., Perlich, C., Horváth, T., Wrobel, S., Korb, K. B., Woodworth, R. J., O’Brien-Malone, A., Diamond, M. R., &
Noble, W. S., Leslie, C., Lagoudakis, M. G., Quadrianto, N., Schüz, B. (2018). Data from, ‘web-based positive psychol­
Buntine, W. L., Quadrianto, N., Buntine, W. L., Getoor, L., ogy interventions: A reexamination of effectiveness.
Namata, G., Getoor, L., Han, X. J., Ting, J.-A., Vijayakumar, S. . . . Journal of Open Psychology Data, 6, 1. https://doi.org/10.
Raedt, L. D. (2011). Leave-one-out cross-validation. In C. Sammut & 5334/jopd.35