Internet Interventions 34 (2023) 100668

Contents lists available at ScienceDirect

Internet Interventions
journal homepage: www.elsevier.com/locate/invent

A chatbot-delivered intervention for optimizing social media use and

reducing perceived isolation among rural-living LGBTQ+ youth:
Development, acceptability, usability, satisfaction, and utility
César G. Escobar-Viera a, b, c, *, Giovanna Porta b, d, Robert W.S. Coulter e, Jamie Martina d,
Jeremy Goldbach f, Bruce L. Rollman b, c, g
a
Department of Psychiatry, School of Medicine, University of Pittsburgh, United States
b
The Enhancing Triage and Utilization for Depression and Emergent Suicidality (ETUDES) Center, School of Medicine, University of Pittsburgh, United States
c
Center for Behavioral Health, Media, and Technology, School of Medicine, University of Pittsburgh, United States
d
Western Psychiatric Hospital, University of Pittsburgh Medical Center, United States
e
Department of Behavioral and Community Health Sciences, School of Public Health, University of Pittsburgh, United States
f
Brown School of Social Work, Washington University in St. Louis, United States
g
Division of General Internal Medicine, School of Medicine, University of Pittsburgh, United States

A R T I C L E I N F O A B S T R A C T

Keywords: Background: Lesbian, gay, bisexual, transgender, and queer/questioning (LGBTQ+) youth are at higher risk of
Rural isolation and depression than their heterosexual peers. Having access to tailored mental health resources is a
Social isolation documented concern for rural living LGBTQ+ youth. Social media provides access to connections to a broader
LGBT youth
and like-minded community of peers, but it also is a vehicle for negative interactions. We developed REALbot, an
Social media, chatbot
automated, social media–based educational intervention to improve social media efficacy, reduce perceived
isolation, and bolster connections for rural living LGBTQ+ youth. This report presents data on the acceptability,
feasibility, and utility of REALbot among its target audience of rural living LGBTQ+ youth.
Methods: We conducted a week-long exploratory study with a single non-comparison group of 20 rural-living
LGBTQ+ youth aged 14–19 recruited from social media to test our Facebook- and Instagram-delivered chat­
bot. We assessed pre- and post-test scores of social media self-efficacy, social isolation (4-item Patient-Reported
Outcomes Measurement System – PROMIS), and depressive symptoms (Patient Health Questionnaire, Adolescent
Version – PHQ-A). At post-test, we assessed acceptability (User Experience Questionnaire – UEQ–S), usability
(Chatbot Usability Questionnaire –CUQ and Post-Study Satisfaction and Usability Questionnaire –PSSUQ), and
satisfaction with the chatbot (Client Satisfaction Questionnaire – CSQ), along with two open-ended questions on
‘likes’ and ‘dislikes’ about the intervention. We compared pre- and post-test scores with standard univariate
statistics. Means and standard deviations were calculated for usability, acceptability, and satisfaction. To analyze
the responses to post-test open-end questions, we used a content analysis approach.
Results: Acceptability of REALbot was high with UEQ-S 5.3 out of 7 (SD = 1.1) and received high usability scores
with CUQ and PSSUQ (mean score (M) = 78.0, SD = 14.5 and M = 86.9, SD = 25.2, respectively), as well as high
user satisfaction with CSQ (M = 24.9, SD = 5.4). Themes related to user ‘likes’ and ‘dislikes’ were organized in
two main categories: usability and content provided. Participants were engaged with the chatbot, sending an
average of 49.3 messages (SD = 43.6, median = 30). Pre-/post- changes in scores of perceived isolation,
depressive symptoms and social media self-efficacy were not significant (p's > 0.08).
Conclusion: REALbot deployment was found to be feasible and acceptable, with good usability and user satis­
faction scores. Participants reported changes from pre- to post-test in most outcomes of interest and effect sizes
were small to medium. Additional development and a formal evaluation of feasibility and engagement with
behavioral targets is warranted.

* Corresponding author at: Department of Psychiatry, School of Medicine, University of Pittsburgh, Room 505 Bellefield Towers, 3811 O'Hara Street, Pittsburgh, PA
15213, United States.
E-mail address: [email protected] (C.G. Escobar-Viera).

https://doi.org/10.1016/j.invent.2023.100668
Received 14 April 2023; Received in revised form 21 June 2023; Accepted 7 September 2023
Available online 9 September 2023
2214-7829/© 2023 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-
nc-nd/4.0/).
C.G. Escobar-Viera et al. Internet Interventions 34 (2023) 100668

1. Introduction and reduce perceived isolation among rural living LGBTQ+ youth. Fig. 1
provides an overview of the study flow. Between November 2021 and
Perceived isolation is an unsatisfactory quantity or quality of social February 2022, we recruited participants via social media ads placed on
relations with others at interpersonal, group, or community levels Instagram and Facebook using the software platforms' ad creation
(Zavaleta et al., 2014), and it is a risk factor for depression, while feature and made them viewable only in rural zip codes within the
community connectedness via family and school are protective (DiFul­ United States as classified by the Health Resources Service Administra­
vio, 2011; Ge et al., 2017; Holt-Lunstad, 2020; Meyer, 2007; Paceley, tion (https://www.hrsa.gov/rural-health/about-us/what-is-rural/data-
2016; Paceley et al., 2017). Lesbian, gay, bisexual, transgender, and files). Youth who were interested tapped on a link in the ads that
queer/questioning (LGBTQ+) youth are at 2–3 times higher risk of redirected them to a study website and eligibility screening survey.
reporting perceived isolation and depression than their heterosexual Youth were eligible to participate if they were 14–19 years of age,
peers (King et al., 2008; Marshal et al., 2011; Medley et al., 2017), with identified as LGBTQ+, lived in a rural area, had access to Facebook
LGBTQ+ youth living in rural areas at still higher risk than those living Messenger web-based or mobile app, and screened positive (score of 16
in urban areas (Cain et al., 2017; Horvath et al., 2014; Lyons et al., or more out of 20) for social isolation on the 4-item PROMIS Social
2015). Indeed, rural communities that value familiarity and sameness Isolation Scale (Patient-Reported Outcomes Measurement Information
may lack diversity and social support for LGBTQ+ youth living there, System, 2015). To protect the privacy of youth under 18 years of age
thus increasing risk for depression and perceived isolation (DiFulvio, from being accidentally outed to their parents/guardians, we obtained a
2011; Paceley, 2016). waiver of parental consent. Eligible youth were then shown informed
Reducing isolation and increasing access to LGBTQ-specific mental assent/consent forms and those who consented were asked to provide
health resources are needs well documented among rural living their contact information.
LGBTQ+ youth (Steinke et al., 2017). To fill these needs, rural living
LGBTQ+ youth often turn to social media to meet others going through 2.2. Ethics approval
similar experiences, connect to a community, or seek information and
social support perceived as unavailable in rural areas (Paceley et al., All recruitment and study procedures were approved by the Internal
2019, 2022). Unfortunately, social media can also be a vehicle for Review Board at the University of Pittsburgh (STUDY19070379).
rejection, discrimination, and other negative experiences, potentially
increasing perceived isolation and depression risk among these youth 2.3. Program development
(Kim et al., 2018; Webb et al., 2021).
Social media self-efficacy refers to a person's perceived ability to REALbot is a rule chatbot deployed on the Facebook Messenger app
reach desired outcomes in their social media interactions (i.e., skills in and on a social media profile on Instagram to deliver educational con­
curating a positive and supportive online environment) (Hocevar et al., tent. To inform development, we conducted an intensive and iterative
2014). Education to promote social media self-efficacy may improve formative research process to learn about youth preferences toward
social media interactions (Haraldstad et al., 2019) and reduce perceived intervention content and favored technology delivery modality. In the
isolation among youth. Social media experiences of rural living LGBTQ+ spring of 2020 (in coincidence with the initial stages of the COVID-19
youth could benefit from educational interventions that reduce the risk lockdown in the U.S.), we conducted a series of online interviews with
of negative interactions and mental health outcomes (Paceley et al., rural living LGTBQ+ adolescents aged 14–19 years to inquire what they
2019, 2022). Despite the limitations of the empirical evidence (i.e., would like to be included in an intervention focused on improving social
focus on screen time, reliance on self-report and lack of data-intensive media use and reducing isolation. Our findings have been published
longitudinal studies focused on impact of social media interactions on elsewhere (Escobar-Viera et al., 2022) and included elements like seeing
mental well-being), initial recommendations to improve social media a positive representation of LGBTQ+ people and learning from people
experiences have been developed with a focus on several personal be­ with shared experiences. Participants also wanted to learn about social
haviors that could help users improve their social media interactions media platform features to help them select different audiences to
(Primack et al., 2018). Furthermore, digital health interventions (e.g., connect with and make their social media experience safer. Addition­
delivered via web, social media, mobile apps, conversational agents) ally, we found that youth preferred using LGBTQ-specific groups on
could be a suitable conduit to deliver this educational content, given existing social media platforms where they already have accounts, but
their wide acceptability to LGBTQ+ persons (Gilbey et al., 2020). Un­ their personal information was not as publicly accessible (Karim et al.,
fortunately, said interventions are few (Escobar-Viera et al., 2021), and 2022).
to the best of our knowledge, just one has been developed with sub­ Based on prior research (Primack et al., 2018) and our own formative
stantial input from rural living LGBTQ+ youth (Fish et al., 2021). work (Escobar-Viera et al., 2022; Karim et al., 2022), we decided to
Additionally, the COVID-19 pandemic has highlighted difficulties for develop content related to four areas of social media interactions. These
mental health services to keep up with an ever-increasing demand and included (1) avoiding negative content and interactions, (2) keeping a
for LGBTQ+ youth to maintain mental well-being while isolated with balance between engaging more passively (i.e., scrolling) and more
unsupportive families (Fish et al., 2020). actively (i.e., commenting on other people's content), (3) connecting
To help address this gap, we now report the development and with actual allies or people with a potential of becoming in-person
deployment of an automated, social media–based educational inter­ friends, and (4) limiting time, frequency of checks, and number of so­
vention called REALbot. Specifically, we evaluated the REALbot's cial media platforms to those that are more personally significant to the
acceptability, feasibility, and utility for increasing social media self- individual and bring them more enjoyment. We chose to focus on these
efficacy and reducing perceived isolation among rural living LGBTQ+ because the state of the evidence on social media use and mental health
youth. specifically pointed at reducing the likelihood of negative interactions,
which might explain depressive symptoms among LGBTQ+ people who
2. Methods are social media users (Escobar-Viera et al., 2020).
To deliver the program content, we decided to have REALbot
2.1. Study design and recruitment deployed on Facebook Messenger app and a social media profile on
Instagram. We chose this combination because our formative work
We conducted a one-week exploratory pilot study using a single indicated that rural living LGBTQ+ youth preferred a combination of
group, pretest-posttest design to evaluate REALbot, a chatbot intended delivery modalities for accessing intervention (Escobar-Viera et al.,
to deliver an educational program to increase social media self-efficacy 2022). Chatbots are convenient artificial intelligence–operated

2
C.G. Escobar-Viera et al. Internet Interventions 34 (2023) 100668

conversational agents; they can be deployed within a social media site, Finally, when users entered their zip code, REALbot provided a list of
do not require user download, and do not take up space in a user's de­ resources including contact information of community based organiza­
vice. Chatbots provide real-time personalized health promotion, pre­ tions for LGBTQ+ rural youth at their local and state level. Fig. 4 pro­
vention, and screening (Aggarwal et al., 2023; Laranjo et al., 2018). In vides examples of infographics included in REALbot.
addition to providing ongoing monitoring and fast access to information
and support tools, chatbots have demonstrated acceptability for mental 2.5. Assessment measures
health interventions, especially for hard-to-reach populations such as
rural living LGBTQ+ people (He et al., 2022; Vaidyam et al., 2019). We Demographic characteristics were assessed at baseline, including date
chose Instagram because it is quite popular app among youth (Vogels of birth, gender identity (“do you consider yourself transgender” with
et al., 2022) and Facebook Messenger because it provides an easy path to options ‘yes’, ‘no’, and ‘don't know/not sure’), gender assigned at birth
deploy chatbots and because, given that it serves users of both Instagram (“what gender were you assigned at birth” with options ‘female’, ‘male’,
and Facebook, participants did not need to create new accounts or leave and ‘not sure/I don't know’), sexual orientation (“which of the following
the environment of the social media apps they use to interact with the best describes you” with options ‘gay or lesbian’, ‘bisexual’, ‘not sure’,
chatbot. ‘straight’), race and ethnicity (“which of the following describes your
race/ethnicity (please check all that apply)” with the options ‘Hispanic
2.4. REALbot: development and content or Latino’, ‘White or Caucasian’, ‘Black/African-American’, ‘Asian/
Asian American’, ‘Native Hawaiian/Other Pacific Islander’, ‘Middle
For developing and designing the content, we conducted two human Eastern/North African’, ‘other’), education status (7th, 8th, 9th, 10th,
centered design (HCD) (Lyon et al., 2019, 2020) sessions with 20 youth 11th, 12th, and college), employment status (‘employed for wages’, ‘a
of lived experience in the winter of 2021. Fig. 2 provides an overview of student’, ‘other’), relationship status (‘single’, ‘member of an unmarried
REALbot's infrastructure and functions. HCD sessions were conducted couple’, ‘in a polyamorous relationship with more than one person’),
online with one participant at a time. Along with two research assistants, person(s) currently living with (‘parent/guardian’, ‘by myself’, ‘signifi­
we developed a series of infographics, animated short videos, and short cant other’, ‘friends’, ‘acquaintances’, ‘other’), current state of resi­
text stories with alternate endings to cover each of the four main topics dence, and zip code.
(i.e., avoiding negative content and interactions, keeping a balanced Acceptability, usability, and satisfaction were assessed at the end of the
engagement, connecting with actual allies, and limiting use), as well as 7-day pilot, with the post-test survey. These included (1) Usability, with
design and color options. HCD sessions were then used to share text, the Chatbot Usability Questionnaire –CUQ (Holmes et al., 2019; Larbi
design, and color palette ideas with participants and get their feedback, et al., 2022) and the Post-Study Satisfaction and Usability Questionnaire
as well as suggestions for other topics that should be covered. –PSSUQ (Lewis, 1992). CUQ comprises 20 statements (e.g., “The chatbot
Next, we used a private hosting service to create the code for all explained its scope and purpose well”) rated on a 5-point Likert scale: (1)
interactions that would occur between the chatbot and the user. Chat Strongly disagree to (5) Strongly agree. PSSUQ consists of 16 statements
interactions followed a classic unidirectional “tree” structure and text (e.g., “It was easy to learn to use this system”) with choices ranging from
for these was stored in a collaborative text document platform (i.e., (1) Strongly disagree to (7) Strongly agree. (2) Acceptability, with short
Google Docs). Fig. 3 provides screenshots of interactions between the version of the User Experience Questionnaire –UEQ-S (Laugwitz et al.,
user and the chatbot. Once REALbot was deployed on Facebook 2008; Schrepp et al., 2017). Items comprise eight domains (obstructive/
Messenger, it included four educational modules called Reneging supportive, complicated/easy, inefficient/efficient, confusing/clear,
negativity, Engaging with balance, Connecting with real allies, and boring/exciting, not interesting/interesting, conventional/inventive,
Limiting use. Each module covered one of the target social media be­ usual/leading edge), each rated on a 1 to 7 Likert scale, with higher
haviors and interactions we selected, and it comprised a set of info­ scores indicating better acceptability. We also included two open ended
graphics, an animated video, and a story with alternate endings questions that asked, “Tell us the things you did like about REALbot” and
depending on the user's choice. Content for each of these included “Tell us what you did not like about REALbot and would like to see
providing a definition of the topic the module would cover, examples of improved.” (3) Satisfaction, with the Client Satisfaction Questionnaire
negative and positive interactions, practical suggestions to avoid nega­ –CSQ-8 (Larsen et al., 1979), which included eight items (e.g., “I am
tive interactions and increase likelihood of positive ones, and testi­ satisfied with the amount of help I received through the chatbot”) rated
monies and suggestions from other rural living LGBTQ+ youth. from (1) Strongly disagree to (4) Strongly agree. For all scales, higher
Infographics, videos, and stories with alternate endings were hosted in scores indicated better outcomes. Finally, ‘frequency’ of use had a range
Amazon Web Services, YouTube, and a private server, respectively. of 0 to 7 for each single calendar day of use, and ‘time’ of use was

Fig. 1. Study Participatns Flowchart

3
C.G. Escobar-Viera et al. Internet Interventions 34 (2023) 100668

Fig. 2. Infrastructure of the 'REALbot' chatbot

calculated by subtracting the time of the first message sent by the user in interaction with the chatbot on Facebook Messenger. During day seven,
a single day from the time of the last message sent by the user. we sent the link to the post-test survey to each participant; we sent a
Social media self-efficacy and perceived isolation were assessed at both single reminder 24 h later to those participants who did not fill the post-
pre- and post-test surveys and included: (1) social media self-efficacy test survey. Participants were asked to provide ongoing assent/consent
with two items asking about perceived overall social media skills and before completing both the pre- and post-test questionnaires. Partici­
perceived ability to find content on social media (Hocevar et al., 2014), pants who completed the pre- and post-surveys were compensated for
(2) perceived isolation with the 8-item PROMIS Social Isolation Scale their time.
(Patient-Reported Outcomes Measurement Information System, 2015),
and (3) depressive symptoms using the eight-item Patient Health 2.7. Data analysis
Questionnaire adapted for adolescents (PHQ-A), a self-reported measure
of depressive symptoms (Johnson et al., 2002; Kroenke et al., 2009). We compared pre- and post-test scores of social isolation, self-
reported depression, and social media skills with standard univariate
2.6. Study procedures and participants statistics (matched-pairs t-test or Wilcoxon signed-rank test when data
failed the Shapiro-Wilk normality test). Tests of significance were two-
A study staff contacted assenting/consenting youth using their tailed with α = 0.05. Given the exploratory nature of the analysis, we
preferred method (i.e., text message, phone call, or email) to describe did not adjust for multiple tests. We report effect sizes (Cohen's d) in
the study, asked them to use REALbot at will over a period of seven days addition to statistical significance. Means and standard deviations were
and fill two surveys, one before using the chatbot and one after. Then, calculated for usability, acceptability, and satisfaction with REALbot
we sent (1) link to a pre-test survey and (2) instructions for initiating an after the intervention. Pearson's correlation coefficient was calculated

4
C.G. Escobar-Viera et al. Internet Interventions 34 (2023) 100668

Fig. 3. Examples of interactions between the user and the 'REALbot' chatbot

for the relationship between time spent on REALbot and change in and after triangulating with the other co-authors, we identified and
scores pre/post intervention. described emerging themes.
To analyze the responses to post-test open-end questions, we used a
content analysis approach (Hsieh and Shannon, 2005). We categorized 3. Results
feedback (both likes and dislikes) according to whether it related to
chatbot usability or content provided. Usability pertained to comments 3.1. Participant characteristics
related to user's interface, perceptions on the responsiveness of the
system, and efficiency of the functions. Content provided comprised As shown in Table 1, 20 adolescents ages 14–20 years old who lived
aspects on whether the chatbot offered relevant and interesting content across the United States participated in the study. Half identified as
related to social media and perceived isolation. Two co-authors coded an transgender and 35 % as cisgender gay/lesbian. They were mostly white
initial set of five responses, compared their initial codes, and resolved (75 %), lived with a parent or guardian (90 %), and were still in high
disagreements. Inter-rater reliability was excellent (k = 0.80) (Landis school (80 %).
and Koch, 1977). After an agreement was reached, they coded inde­
pendently the remaining responses. Next, we reviewed coded excerpts

5
C.G. Escobar-Viera et al. Internet Interventions 34 (2023) 100668

Fig. 4.. Examples of infographics that users were able to download from the 'REALbot' chatbot

3.2. Acceptability and usability of REALbot example quotes.

The chatbot received high user satisfaction with CSQ (M = 24.9, SD 3.3.1. User ‘likes’
= 5.4). As can be seen in Table 2, across all domains, participants rated Related to usability, two main themes emerged. (1) Pleasant chatbot
the acceptability of REALbot 5.3 out of 7 (SD = 1.1) using the UEQ-S. design and interactions [8/19]. In general, participants thought that
Scores on each domain ranged from an average of 4.6 (SD = 1.6) to REALbot had an attractive design for teens. For example, one participant
6.0 (SD = 1.3). While only 25 % of participants described the chatbot as (15 years old, male, AMAB, bisexual) liked that the chatbot was spe­
exciting or leading edge (score of 6 or 7 out of 7), 40 % to 75 % of cifically “designed for rural queer teens like me, and even gives you
participants gave higher scores (6 or 7) to the other domains. Eight state-specific resources.” Youth thought the chatbot was respectful
participants (42 %) interacted with REALbot for two or more days. They during chat interactions; another participant (15 years old, male, AMAB,
spent an average of 35.6 min on the chatbot (standard deviation [SD] = bisexual) particularly liked “how the chatbot asked me for my name
59.2, median = 15), range 1.4 min to 4 h and 21 min. They received an instead of reading it off my Facebook because I have to use my dead
average of 62.6 messages from REALbot (SD = 57.5, median = 39, range name on Facebook, so it was nice having the bot ask me what my name
8 to 202) and they sent an average of 49.3 messages to the chatbot (SD = was and continue to use my chosen name throughout the interactions”.
43.6, median = 30, range 6 to 162). REALbot received high usability A third participant (18 years old, transgender, AMAB, gay or lesbian)
scores on both the CUQ (mean score (M) = 78.0, SD = 14.5) and PSSUQ mentioned that the chatbot was welcoming during the chat but also
(M = 86.9, SD = 25.2). quick to respond with very little lag time. (2) Ease of use and interesting
features [11/19]. Participants concurred that it was easy to access and
start using the chatbot. One participant (19 years old, transgender,
3.3. Qualitative user feedback
AFAB, bisexual) liked that REALbot did not require installing any
additional app and it was easy to start interacting with, and they “also
A total of eight themes emerged. Four were related to usability and
like[d] that once you chose an option to get information about, it spaced
four themes were about content provided by the chatbot. Below, we
out each text or image a bit, instead of sending everything at once.”
provide details of each theme, their frequency (in brackets), and

6
C.G. Escobar-Viera et al. Internet Interventions 34 (2023) 100668

Table 1 remember past information like name/age” (17 years old, female, AFAB,
Participants demographic characteristics. not sure of sexual orientation). (6) Limited number of platforms where
Category Description Participants (n = 20) chatbot is available [5/19]. Several youths mentioned that the bot should
be available more widely, in several platforms, to boost opportunities of
Age Mean years (SD) 16.6 (1.5)
Sex at birth Male 7 (35 %) interactions with the user. “You could continue to have this on Facebook
Female 13 (65 %) messenger but could also have it on Instagram for example” (14 years
Gender identity Transgender 10 (50 %) old, transgender, AFAB, bisexual).
Non-Transgender 8 (40 %) Related to the content provided, two themes emerged. (7) Insufficient
Don't know/not sure 2 (10 %)
Sexual orientation Bisexual 11 (55 %)
content on social media interactions and ways of delivering said content [6/
Gay/lesbian 7 (35 %) 19]. While participants thought the content related to being mindful of
Not Sure 2 (10 %) negativity on social media and focusing on connecting with real allies
Race/ethnicity White 15 (75 %) was important, some youth disliked the lack of more practical, directive
Black/African American 2 (10 %)
instructions on “how much time to spend on social media or how to stay
Asian/Asian American 1 (5 %)
Hispanic/Latino 1 (5 %) away from it” (17 years old, transgender, AFAB, bisexual) and others
Middle Eastern/North African 1 (5 %) wanted more in-depth discussion and conversation about the topic.
Educational level Jr. High School (7th–8th grade) 2 (10 %) Participants also wanted content to be delivered using voiceover for the
High School (9th–12th grade) 16 (80 %) infographics and the text that appeared on videos. One participant
Some College/Tech School 2 (10 %)
Employment Student 16 (84.2 %)
mentioned that voiceover will help the user concentrate more on the
Unemployed 2 (10.5 %) content. (8) Lack of content on other topics important to LGBTQ+ youth [9/
Employed for wages 1 (5.3 %) 19]. Some participants felt bored with the focus on only social media
Living situation With Parent/Guardian 18 (90 %) interactions at the expense of other topics that are important to rural
With Friends 1 (5 %)
living LGBTQ+ youth, such as how to develop healthy relationships with
Other 1 (5 %)
Relationship status Single 16 (80 %) friends and romantic partners, as well as how to cope with discrimina­
Member of unmarried couple 3 (15 %) tion, and “advice on how to come out and how to deal with homophobic
In polyamorous relationship 1 (5 %) family members and friends”.

Regarding content provided, themes included (3) Interesting range to 3.4. Pre – post evaluation of the chatbot
topics covered [12/19]. Youth thought that the chatbot offered content in
a variety of topics and these were interesting and important. For Pre- and post-intervention scores for social isolation, self-reported
example, one participant (16 years old, male, AMAB, gay) liked that depression and social media skills are displayed in Table 3. Partici­
“REALbot did provide very good advice on how to deal with problems pants reported non-significant changes in scores of perceived isolation
surrounding sexuality and components of social life” and another (p = 0.44, d = − 0.36), depressive symptoms (p = 0.08, d = − 0.36), and
participant (15 years old, not sure if transgender, AMAB, bisexual) social media self-efficacy (overall, p = 0.17, d = 0.18; ability to find
thought “it was very informative and helpful when it came to advice content, p = 0.002, d = 0.77). Small non-significant correlations were
looking for resources in my area and advice about social interactions.” found between social isolation (r = − 0.27, p = 0.27), self-reported
(4) Comprehensiveness of information provided [7/19]. Participants depression (r = − 0.33, p = 0.19), overall social media skills (r = 0.23,
mentioned that REALbot offered a good number of references and p = 0.38) and time spent on REALbot. While not statistically significant,
sources of information, such as links to videos; one participant said that change in scores were in the expected direction (i.e., the more engaged
they “liked the way that the chatbot explained things, it was easy to with REALbot the better social media skills users reported, and the less
comprehend, and having videos and pictures in it will surely help others social isolation and depression). The correlation between ability to find
with engagement.”
Table 3
3.3.2. User ‘dislikes’ Pre/post intervention scores.
The main challenges related to usability centered around (5) Chatbot Outcomes Pre Post Test df P d
felt robotic and not smart enough [12/19]. Youth coalesced around the
Social Isolation M(SD) 30.2 28.8 z= – 0.44 − 0.36
idea that even though REALbot was friendly during the interactions, it (7.5) (9.3) 0.77
still seemed robotic and limited in its ability to remember and use in­ Patient Health 14.3 11.6 t= 18 0.08 − 0.36
formation that was already entered by the user and making it part of a Questionnaire M(SD) (5.3) (7.1) 1.88
cohesive conversation throughout successive interactions. One partici­ Social Media Self-
efficacy M(SD)
pant (15 years old, transgender, AFAB, bisexual) thought it was tedious Overall skills 5.6 6.2 t = 16 0.17 0.18
to have to re-enter their information for every interaction. Another (2.2) (2.4) − 1.44
participant expressed frustration because “you have to completely Ability to find content 5.3 6.9 t = 16 0.002 0.77
restart the conversation each day you talk with the bot— it does not on social media (2.3) (1.7) − 3.77

Table 2
Post-intervention assessment of the acceptability of REALbot.
REALbot is 1 2 3 4 5 6 7 M (SD)

Obstructive/supportive 1 (5 %) 0 (0 %) 0 (0 %) 2 (10 %) 5 (25 %) 8 (40 %) 4 (20 %) 5.5 (1.4)

Complicated/easy 1 (5 %) 0 (0 %) 1 (5 %) 2 (10 %) 1 (5 %) 6 (30 %) 9 (45 %) 5.8 (1.6)
Inefficient/efficient 2 (10 %) 0 (0 %) 0 (0 %) 3 (15 %) 3 (15 %) 3 (15 %) 9 (45 %) 5.5 (1.9)
Confusing/clear 0 (0 %) 0 (0 %) 2 (10 %) 1 (5 %) 2 (10 %) 6 (30 %) 9 (45 %) 6.0 (1.3)
Boring/exciting 0 (0 %) 0 (0 %) 2 (10 %) 4 (20 %) 7 (35 %) 4 (20 %) 1 (5 %) 4.6 (1.4)
Not interesting/interesting 0 (0 %) 0 (0 %) 3 (15 %) 2 (10 %) 6 (30 %) 4 (20 %) 4 (40 %) 5.1 (1.5)
Conventional/inventive 2 (10 %) 0 (0 %) 0 (0 %) 3 (15 %) 5 (25 %) 4 (20 %) 6 (30 %) 5.3 (1.8)
usual/leading edge 2 (10 %) 0 (0 %) 2 (10 %) 3 (15 %) 8 (40 %) 4(20 %) 1 (5 %) 4.6 (1.6)

7
C.G. Escobar-Viera et al. Internet Interventions 34 (2023) 100668

content on social media and time was negligible (r = 0.02, p = 0.94). media self-efficacy (Hocevar et al., 2014). This finding aligns with
youth's positive qualitative feedback related to the chatbot's ability to
4. Discussion provide links to location-based state and local mental health resources
available to LGBTQ+ teens. At the same time, the effect on youth's
4.1. Summary of evidence overall skills to manage their social media interactions was small. This
also correlates with qualitative feedback in that youth wanted more in-
Rural living LGBTQ+ youth are at compound risk for perceived depth content related social media use, including specific suggestions on
isolation, depression, and other mental health concerns (Holt-Lunstad, how to limit their time on social media. More research with REALbot
2020; Monteith et al., 2021). This study detailed the development and and other, similar interventions comparing different content or delivery
evaluation of a social media–based, chatbot-delivered educational modalities (including delivery via peer support or counseling) is
intervention for optimizing social media experiences and reducing necessary.
perceived isolation among rural living LGBTQ+ youth. Our findings
suggest that REALbot deployment is feasible, the chatbot is acceptable 4.2. Limitations
among youth, and participants who interacted with it reported higher
scores at follow-up (compared to baseline) in social media self-efficacy. This is study had several limitations to consider. First, given the
Importantly, qualitative participants' feedback provides a roadmap for study design, we did not have a comparison group, thus limiting our
necessary improvements related to both usability and desired content ability to make any inference about the cause of the changes found after
that LGBTQ+ teens consider important for their lives and mental well- using REALbot. Second, given that our one-week study was conducted at
being. a time when the COVID-19 pandemic, though receding, was still
In terms of engagement, participants received an average of 62 ongoing, we cannot rule out that some of the changes our participants
messages from REALbot and sent an average of 49 messages to the showed were due to contextual changes along the course of the outbreak
chatbot. Recent studies of digital mental health interventions delivered (e.g., schools re-opening). Additionally, the small to medium score
via chatbots reported lower user response to the chatbot, from an changes that were detected could be due to the short duration of the
average of 17 responses of 46 days of interaction among youth with study, or the positive evaluation of the attention given to this socially
depression (Dosovitsky et al., 2020) to 116 messages over an 8-week isolated group. However, most schools had already re-opened at the
long study with a chatbot for anxiety and depression among college time we conducted our one-week exploratory study (November 2021 –
students (Klos et al., 2021). While our study was only 7-day long and we February 2022). Third, our measure of social media self-efficacy
had no comparison group, the level of participant interaction with included only two items (i.e., overall skills and ability to find content
REALbot was encouraging and it echoes findings from qualitative and information on social media). While overall social media skills
research that rural living LGBTQ+ youth have an urgent need for might relate to perceived ability to curate a safe online environment,
LGBTQ+ specific resources (Steinke et al., 2017) for reducing perceived and ability to find content on social media might relate to finding re­
isolation (Paceley et al., 2019). sources and other forms of support, we did not directly assess these
Forty-two percent of participants engaged with REALbot two or more outcomes and future research should include a more detailed exami­
days during our one-week exploratory study. However, given the limited nation of the different aspects of social media self-efficacy. Fourth, we
content that our chatbot covered for this study, it is likely that the did not assess changes in youth loneliness. While the constructs of
average 35 min of actual engagement was enough for participants to go loneliness and perceived isolation overlap to some extent, we cannot
through the entirety of the available topics. This resonates with teens' rule that some cognitive aspects (that were not assessed with our mea­
feedback about the chatbot feeling robotic, with limited content and sure of perceived isolation) changed over the course of the study,
ability to engage in conversations. These findings are in between similar therefore impacting the measured outcomes. Finally, our small sample
studies that found both shorter (Anmella et al., 2023) and longer (Luk size and the lack of more objective outcome measures (such as assess­
et al., 2022) user engagement with other chatbot-delivered in­ ment of number of positive and negative social media interactions) limit
terventions. Despite the relative low engagement, REALbot received our ability to fully examine REALbot's efficacy.
good acceptability, usability (with two different scales), and satisfaction Our study suggests more research is needed to understand the po­
scores. Interestingly, the lower scores in two of the acceptability sub­ tential benefits of interventions such as REALbot. For rural youth, one
scales (i.e., boring/exciting, usual/leading edge) were also mentioned in potential advantage of REALbot is that its entire content was delivered
youth's qualitative feedback. These findings highlight the importance of online; given the increasing constraint for providing effective in-person
keeping an adequate amount of intervention content to boost engage­ preventive mental health services, automated interventions delivered
ment with digital health interventions. Moreover, our participants' via existing online platforms, such as social media, offer a potential
qualitative feedback asking for content related to how to deal with implementation path that warrants further research. This is even more
discriminating family members and friends is a clear reminder that important given that while LGBTQ+ people are highly acceptant of so­
managing these difficult interactions is an important factor for feeling cial media–based interventions (Gilbey et al., 2020), there are few of
lonely and isolated for rural living LGBTQ+ youth (DiFulvio, 2011). these interventions focused on youth mental health (Escobar-Viera et al.,
Although not the larger purpose of the study, REALbot did appear to 2021).
have potential for increasing social media self-efficacy and for reducing
perceived isolation and depressive symptoms from pre- to post-test 5. Conclusions
among LGBTQ+ teens. Changes in the ability to find content on social
media were statistically significant, and the effect size of all pre/post REALbot deployment on a social media platform was feasible; users
scores ranged from small (social isolation, depressive symptoms, and found it acceptable, usable, and were generally satisfied with the con­
social media overall skills) to medium (ability to find content on social tent. Nevertheless, users enumerated a series of dislikes and targets for
media). Given the sample size in our study, non-significant results are improvement, both in terms of interactions with the chatbot and the
somewhat not surprising; effect sizes, however, are independent of content provided within it. Study participants reported changes from
sample size (Sullivan and Feinn, 2012). The effect sizes of our inter­ pre- to post-test in all the outcomes of interest (social media self-efficacy,
vention on perceived isolation and depressive symptoms are in line with perceived isolation, and depressive symptoms), and effect sizes were
those of other interventions for loneliness and isolation among LGBTQ+ small to medium. Additional development and a formal evaluation of
people (Smith et al., 2016, 2017). The chatbot had considerable effect feasibility and engagement with behavioral targets are warranted.
on users' ability to find content online, which is one aspect of social Further research is also needed to incorporate new topics and new

8
C.G. Escobar-Viera et al. Internet Interventions 34 (2023) 100668

intervention targets responsive to rural living LGBTQ+ youth who are in E40719. https://www.jmir.org/2022/11/e40719, 24(11), e40719. https://doi.
org/10.2196/40719.
high need of interventions to support their mental well-being.
Hocevar, K.P., Flanagin, A.J., Metzger, M.J., 2014. Social media self-efficacy and
information evaluation online. Comput. Hum. Behav. 39, 254–262. https://doi.org/
Funding 10.1016/j.chb.2014.07.020.
Holmes, S., Moorhead, A., Bond, R., Zheng, H., Coates, V., Mctear, M., 2019. Usability
testing of a healthcare chatbot: can we use conventional methods to assess
We gratefully acknowledge funding from the National Institute of conversational user interfaces?. In: Proceedings of the 31st European Conference on
Minority Health and Health Disparities (NIH grant number R00 - Cognitive Ergonomics, pp. 207–214. https://doi.org/10.1145/3335082.3335094.
Holt-Lunstad, J., 2020. Social isolation and health - health policy brief. Health Aff.
MD012813), the Enhancing Triage and Utilization for Depression and
https://doi.org/10.1377/hpb20200622.253235.
Emergent Suicidality (ETUDES) Center (NIH grant number P50 - Horvath, K.J., Iantaffi, A., Swinburne-Romine, R., Bockting, W., 2014. A comparison of
MH115838), and the Center for Behavioral Health, Media, and Tech­ mental health, substance use, and sexual risk behaviors between rural and non-rural
nology at the University of Pittsburgh for the completion of this transgender persons. J. Homosex. 61 (8), 1117–1130. https://doi.org/10.1080/
00918369.2014.872502.
research. Funding sources played no role in the study design; collection, Hsieh, H.-F., Shannon, S.E., 2005. Three approaches to qualitative content analysis. Qual.
analysis, and interpretation of data; writing of the report; or decision to Health Res. 15 (9), 1277–1288. https://doi.org/10.1177/1049732305276687.
submit the article for publication. Johnson, J.G., Harris, E.S., Spitzer, R.L., Williams, J.B.W., 2002. The patient health
questionnaire for adolescents: validation of an instrument for the assessment of
mental disorders among adolescent primary care patients. J. Adolesc. Health 30 (3),
Declaration of competing interest 196–204. https://doi.org/10.1016/S1054-139X(01)00333-0.
Karim, S., Choukas-Bradley, S., Radovic, A., Roberts, S.R., Maheux, A.J., Escobar-
Viera, C.G., 2022. Support over social media among socially isolated sexual and
The authors declare that they have no known competing financial gender minority youth in rural U.S. during the COVID-19 pandemic: opportunities
interests or personal relationships that could have appeared to influence for intervention research. Int. J. Environ. Res. Public Health 19 (23), 15611. https://
doi.org/10.3390/ijerph192315611.
the work reported in this paper. Kim, S., Boyle, M.H., Georgiades, K., 2018. Cyberbullying victimization and its
association with health across the life course: a Canadian population study. Can. J.
References Public Health 108 (5–6), e468–e474. https://doi.org/10.17269/cjph.108.6175. Jan
22.
King, M., Semlyen, J., Tai, S.S., Killaspy, H., Osborn, D., Popelyuk, D., Nazareth, I., 2008.
Aggarwal, A., Tam, C.C., Wu, D., Li, X., Qiao, S., 2023. Artificial intelligence–based
A systematic review of mental disorder, suicide, and deliberate self harm in lesbian,
chatbots for promoting health behavioral changes: systematic review. J. Med.
gay and bisexual people. In: BMC Psychiatry, vol. 8. BioMed Central, p. 70. https://
Internet Res. 25 (1), e40789 https://doi.org/10.2196/40789.
doi.org/10.1186/1471-244X-8-70.
Anmella, G., Sanabra, M., Primé-Tous, M., Segú, X., Cavero, M., Morilla, I., Grande, I.,
Klos, M.C., Escoredo, M., Joerin, A., Lemos, V.N., Rauws, M., Bunge, E.L., 2021. Artificial
Ruiz, V., Mas, A., Martín-Villalba, I., Caballo, A., Esteva, J.-P., Rodríguez-Rey, A.,
intelligence–based Chatbot for anxiety and depression in university students: pilot
Piazza, F., Valdesoiro, F.J., Rodriguez-Torrella, C., Espinosa, M., Virgili, G.,
randomized controlled trial. JMIR Form Res. 5 (8), e20678 https://doi.org/10.2196/
Sorroche, C., Hidalgo-Mazzei, D., 2023. Vickybot, a Chatbot for anxiety-depressive
20678.
symptoms and work-related burnout in primary care and health care professionals:
Kroenke, K., Strine, T.W., Spitzer, R.L., Williams, J.B.W., Berry, J.T., Mokdad, A.H.,
development, feasibility, and potential effectiveness studies. J. Med. Internet Res. 25
2009. The PHQ-8 as a measure of current depression in the general population.
(1), e43293 https://doi.org/10.2196/43293.
J. Affect. Disord. 114 (1–3), 163–173. https://doi.org/10.1016/j.jad.2008.06.026.
Cain, D.N., Mirzayi, C., Rendina, H.J., Ventuneac, A., Grov, C., Parsons, J.T., 2017.
Landis, J.R., Koch, G.G., 1977. The measurement of observer agreement for categorical
Mediating effects of social support and internalized homonegativity on the
data. Biometrics 33 (1), 159–174. https://doi.org/10.2307/2529310.
association between population density and mental health among gay and bisexual
Laranjo, L., Dunn, A.G., Tong, H.L., Kocaballi, A.B., Chen, J., Bashir, R., Surian, D.,
men. LGBT Health 4 (5), 352–359. https://doi.org/10.1089/lgbt.2017.0002.
Gallego, B., Magrabi, F., Lau, A.Y.S., Coiera, E., 2018. Conversational agents in
DiFulvio, G.T., 2011. Sexual minority youth, social connection and resilience: from
healthcare: a systematic review. J. Am. Med. Inform. Assoc. 25 (9), 1248–1258.
personal struggle to collective identity. Soc. Sci. Med. 72 (10), 1611–1617. https://
https://doi.org/10.1093/jamia/ocy072.
doi.org/10.1016/j.socscimed.2011.02.045.
Larbi, D., Denecke, K., Gabarron, E., 2022. Usability testing of a social media Chatbot for
Dosovitsky, G., Pineda, B.S., Jacobson, N.C., Chang, C., Bunge, E.L., 2020. Artificial
increasing physical activity behavior. J. Person. Med. 12 (5), 828. https://doi.org/
intelligence Chatbot for depression: descriptive study of usage. JMIR Form Res. 4
10.3390/JPM12050828.
(11), e17065.
Larsen, D.L., Attkisson, C.C., Hargreaves, W.A., Nguyen, T.D., 1979. Assessment of
Escobar-Viera, C.G., Shensa, A., Sidani, J., Primack, B., Marshal, M.P., 2020. Association
client/patient satisfaction: development of a general scale. Eval. Program Plann. 2
between LGB sexual orientation and depression mediated by negative social media
(3), 197–207. https://doi.org/10.1016/0149-7189(79)90094-6.
experiences: national survey study of US young adults. JMIR Mental Health 7 (12),
Laugwitz, B., Held, T., Schrepp, M., 2008. Construction and evaluation of a user
e23520. https://doi.org/10.2196/23520.
experience questionnaire. Lect. Notes Comput. Sci 5298, 63–76. https://doi.org/
Escobar-Viera, C.G., Melcher, E.M., Miller, R.S., Whitfield, D.L., Jacobson-López, D.,
10.1007/978-3-540-89350-9_6/COVER.
Gordon, J.D., Ballard, A.J., Rollman, B.L., Pagoto, S., 2021. A systematic review of
Lewis, J.R., 1992. Psychometric evaluation of the post-study system usability
the engagement with social media–delivered interventions for improving health
questionnaire: the PSSUQ. Proc. Hum. Factors Ergon. Soc. Ann. Meet. 36 (16),
outcomes among sexual and gender minorities. Internet Interv. 25 https://doi.org/
1259–1260. https://doi.org/10.1177/154193129203601617.
10.1016/J.INVENT.2021.100428.
Luk, T.T., Lui, J.H.T., Wang, M.P., 2022. Efficacy, usability, and acceptability of a
Escobar-Viera, C.G., Choukas-Bradley, S., Sidani, J., Maheux, A.J., Roberts, S.R.,
Chatbot for promoting COVID-19 vaccination in unvaccinated or booster-hesitant
Rollman, B.L., 2022. Examining social media experiences and attitudes toward
young adults: pre-post pilot study. J. Med. Internet Res. 24 (10), e39063 https://doi.
technology-based interventions for reducing social isolation among LGBTQ youth
org/10.2196/39063.
living in rural United States: an online qualitative study. Front. Digit. Health 4.
Lyon, A.R., Munson, S.A., Renn, B.N., Atkins, D.C., Pullmann, M.D., Friedman, E.,
https://doi.org/10.3389/fdgth.2022.900695.
Areán, P.A., 2019. Use of human-centered design to improve implementation of
Fish, J.N., McInroy, L.B., Paceley, M.S., Williams, N.D., Henderson, S., Levine, D.S.,
evidence-based psychotherapies in low-resource communities: protocol for studies
Edsall, R.N., 2020. “I’m kinda stuck at home with unsupportive parents right now”:
applying a framework to assess usability. JMIR Res. Protoc. 8 (10) https://doi.org/
LGBTQ Youths’ experiences with COVID-19 and the importance of online support.
10.2196/14990.
J. Adolesc. Health 67 (3), 450–452. https://doi.org/10.1016/j.
Lyon, A.R., Brewer, S.K., Areán, P.A., 2020. Leveraging human-centered design to
jadohealth.2020.06.002.
implement modern psychological science: return on an early investment. Am.
Fish, J.N., Williams, N.D., McInroy, L.B., Paceley, M.S., Edsall, R.N., Devadas, J.,
Psychol. 75 (8), 1067–1079. https://doi.org/10.1037/amp0000652.
Henderson, S.B., Levine, D.S., 2021. Q chat space: assessing the feasibility and
Lyons, A., Hosking, W., Rozbroj, T., 2015. Rural-urban differences in mental health,
acceptability of an internet-based support program for LGBTQ youth. Prev. Sci. 1, 1.
resilience, stigma, and social support among young Australian gay men. J. Rural.
https://doi.org/10.1007/S11121-021-01291-Y.
Health 31 (1), 89–97. https://doi.org/10.1111/jrh.12089.
Ge, L., Yap, C.W., Ong, R., Heng, B.H., 2017. Social isolation, loneliness and their
Marshal, M.P., Dietz, L.J., Friedman, M.S., Stall, R., Smith, H.A., McGinley, J., Thoma, B.
relationships with depressive symptoms: a population-based study. PLoS One 12 (8),
C., Murray, P.J., D’Augelli, A.R., Brent, D.A., 2011. Suicidality and depression
e0182145. https://doi.org/10.1371/journal.pone.0182145.
disparities between sexual minority and heterosexual youth: a meta-analytic review.
Gilbey, D., Morgan, H., Lin, A., Perry, Y., 2020. Effectiveness, acceptability, and
J. Adolesc. Health 49 (2), 115–123. https://doi.org/10.1016/j.
feasibility of digital health interventions for LGBTIQ+ young people: systematic
jadohealth.2011.02.005.
review. J. Med. Internet Res. 22 (12), e20158 https://doi.org/10.2196/20158.
Medley, G., Lipari, R., Bose, J., Cribb, D., Kroutil, L., McHenry, G., 2017. Sexual
Haraldstad, K., Kvarme, L.G., Christophersen, K.A., Helseth, S., 2019. Associations
orientation and estimates of adult substance use and mental health: results from the
between self-efficacy, bullying and health-related quality of life in a school sample of
2015 national survey on drug use and health. NSDUH Data Review. https://www.sa
adolescents: a cross-sectional study. BMC Public Health 19 (1), 1–9. https://doi.org/
mhsa.gov/data/sites/default/files/NSDUH-SexualOrientation-2015/NSDUH-S
10.1186/s12889-019-7115-4.
exualOrientation-2015/NSDUH-SexualOrientation-2015.htm.
He, Y., Yang, L., Zhu, X., Wu, B., Zhang, S., Qian, C., Tian, T., 2022. Mental health
Meyer, I.H., 2007. Prejudice and discrimination as social stressors. In: Meyer, I.H.,
Chatbot for young adults with depressive symptoms during the COVID-19 pandemic:
Northridge, M. (Eds.), The Health of Sexual Minorities. Springer, pp. 242–267.
single-blind, three-arm randomized controlled trial. J. Med. Internet Res. 24 (11),

9
C.G. Escobar-Viera et al. Internet Interventions 34 (2023) 100668

Monteith, L.L., Holliday, R., Brown, T.L., Brenner, L.A., Mohatt, N.V., 2021. Preventing Smith, N.G., Hart, T.A., Moody, C., Willis, A.C., Andersen, M.F., Blais, M., Adam, B.,
suicide in rural communities during the COVID-19 pandemic. J. Rural. Health 37 (1), 2016. Project PRIDE: a cognitive-behavioral group intervention to reduce HIV risk
179–184. https://doi.org/10.1111/JRH.12448. behaviors among HIV-negative young gay and bisexual men. Cogn. Behav. Pract. 23
Paceley, M.S., 2016. Gender and sexual minority youth in nonmetropolitan communities: (3), 398–411. https://doi.org/10.1016/j.cbpra.2015.08.006.
individual- and community-level needs for support. Fam. Soc. 97 (2), 77–85. https:// Smith, N.G., Hart, T.A., Kidwai, A., Vernon, J.R.G., Blais, M., Adam, B., 2017. Results of a
doi.org/10.1606/1044-3894.2016.97.11. pilot study to ameliorate psychological and behavioral outcomes of minority stress
Paceley, M.S., Hwu, A., Arizpe, H.D., 2017. Nonmetropolitan sexual and gender minority among young gay and bisexual men. Behav. Ther. 48 (5), 664–677. https://doi.org/
youths’ friendships: perceptions of social support among SGM and non-SGM peers, 10.1016/j.beth.2017.03.005.
29 (4), 399–414. https://doi.org/10.1080/10538720.2017.1365674. Steinke, J., Root-Bowman, M., Estabrook, S., Levine, D., Kantor, L., 2017. Meeting the
Paceley, M.S., Fish, J.N., Conrad, A., Schuetz, N., 2019. Diverse community contexts and needs of sexual and gender minority youth: formative research on potential digital
community resources for sexual and gender minority youth: a mixed methods study. health interventions. J. Adolesc. Health 60 (5), 541–548. https://www.sciencedirect.
J. Community Appl. Soc. Psychol. 29 (6), 445–460. https://doi.org/10.1002/ com/science/article/pii/S1054139X1630876X.
casp.2417. Sullivan, G.M., Feinn, R., 2012. Using effect size—or why the P value is not enough.
Paceley, M.S., Goffnett, J., Sanders, L., Gadd-Nelson, J., 2022. “Sometimes you get J. Grad. Med. Educ. 4 (3), 279–282. https://doi.org/10.4300/JGME-D-12-00156.1.
married on Facebook”: the use of social media among nonmetropolitan sexual and Vaidyam, A.N., Wisniewski, H., Halamka, J.D., Kashavan, M.S., Torous, J.B., 2019.
gender minority youth. J. Homosex. 69 (1), 41–60. https://doi.org/10.1080/ Chatbots and conversational agents in mental health: a review of the psychiatric
00918369.2020.1813508. landscape. Can. J. Psychiatr. https://doi.org/10.1177/0706743719828977,
Patient-Reported Outcomes Measurement Information System, 2015. Social Isolation: a 070674371982897.
brief guide to the PROMIS Social Isolation instruments. https://www.assessmentcent Vogels, E.A., Gelles-Watnick, R., Massarat, N., 2022. Teens, Social Media and Technology
er.net/documents/PROMISSocialIsolationScoringManual.pdf. 2022. https://www.pewresearch.org/internet/2022/08/10/teens-social-media
Primack, B.A., Shensa, A.S., Sidani, J.E., Bowman, N.D., Knight, J.M., Karim, S.A., -and-technology-2022/.
Bisbey, M.A., Colditz, J.B., Woods, M.S., Escobar-Viera, C.G., 2018. Reducing risk for Webb, L., Clary, L.K., Johnson, R.M., Mendelson, T., 2021. Electronic and school bullying
mental health conditions associated with social media use: encouraging “REAL” victimization by race/ethnicity and sexual minority status in a nationally
communication. In: Van Hook, J., McHale, S.M., King, V. (Eds.), Families and representative adolescent sample. J. Adolesc. Health 68 (2), 378–384. https://doi.
Technology. Springer, pp. 155–176. org/10.1016/j.jadohealth.2020.05.042. Feb.
Schrepp, M., Thomaschewski, J., Hinderks, A., 2017. Design and evaluation of a short Zavaleta, D., Kim, S., Mills China, 2014. Social Isolation: a conceptual and measurement
version of the user experience questionnaire (UEQ-S). Int. J. Interact. Multimedia proposal | OPHI (No. 67; OPHI Working Papers). https://ophi.org.uk/social-isolat
Artif. Intell. 4 (Regular Issue), 103. https://doi.org/10.9781/IJIMAI.2017.09.001. ion-a-conceptual-and-measurement-proposal/.

10