Computers in Human Behavior 55 (2016) 477e485

Contents lists available at ScienceDirect

Computers in Human Behavior

journal homepage: www.elsevier.com/locate/comphumbeh

Full length article

Augmented reality-based video-modeling storybook of nonverbal

facial cues for children with autism spectrum disorder to improve
their perceptions and judgments of facial expressions and emotions
Chien-Hsu Chen a, *, I-Jui Lee a, **, Ling-Yi Lin b
a
Department of Industrial Design, College of Planning and Design, National Cheng Kung University, No.1 University Rd., East District, Tainan, Taiwan
b
Department of Occupational Therapy, College of Medicine, National Cheng Kung University, No.1 University Rd., Tainan, Taiwan

a r t i c l e i n f o a b s t r a c t

Article history: Autism spectrum disorders (ASD) are characterized by a reduced ability to understand the emotions of
Received 23 July 2014 other people. Increasing evidence indicates that children with ASD might not recognize or understand
Received in revised form crucial nonverbal behaviors, which likely causes them to ignore nonverbal gestures and social cues, like
20 July 2015
facial expressions, that usually aid social interaction. We used an augmented reality (AR)-based video
Accepted 21 September 2015
Available online 5 November 2015
modeling (VM) storybook (ARVMS) to strengthen and attract the attention of children with ASD to
nonverbal social cues because they have difficulty adjusting and switching their attentional focus. In this
research, AR has multiple functions: it extends the social features of the story, but it also restricts
Keywords:
Augmented reality (AR)
attention to the most important parts of the videos.
Visual indicator Evidence-based research shows that AR attracts the attention of children with ASD. However, few studies
Visual supports have combined AR with VM to train children with ASD to mimic facial expressions and emotions to improve
Video modeling their social skills. In addition, we used markerless natural tracking to teach the children to recognize patterns
Nonverbal social cues as they focused on the stable visual image printed in the storybook and then extended their attention to an
animation of the story. After the three-phase (baseline, intervention, and maintenance) test data had been
collected, the results showed that ARVMS intervention provided an augmented visual indicator which had
effectively attracted and maintained the attention of children with ASD to nonverbal social cues and helped
them better understand the facial expressions and emotions of the storybook characters.
© 2015 Elsevier Ltd. All rights reserved.

1. Introduction Allison, Baron-Cohen, Chakrabarti, & Hoekstra, 2013); it requires

using both joint attention and nonverbal social skills (Constantino
Autism spectrum disorders (ASD) are characterized by repetitive et al., 2003). Although some people with high-functioning autism
and stereotyped behavior, and by impairments in verbal and (HFA) are relatively adept at social communication involving
nonverbal communication and in social behavior (Boelte & complex emotions, they find nonverbal communication a barrier
Hallmayer, 2013). In particular, social reciprocity defects are one (Elder, Caterino, Chao, Shacknai, & De Simone, 2006).
of the core deficits in social interaction for people with ASD In addition, among the most characteristic early symptoms of
(American Psychiatric Association, 2000; White, Keonig, & Scahill, ASD are atypical eye contact and atypical joint attention, which
2007). Social reciprocity depends upon the ability to empathize profoundly impair the development of social communication skills
with (attribute mental states to) others, to be aware of emotional (Senju & Johnson, 2009). Children with ASD typically have behav-
and interpersonal cues, and to respond appropriately (Sucksmith, ioral difficulties that suggest problems with visual attention. It is
unclear whether this attention deficit causes the other symptoms
of ASD or is a consequence of the disorder (Koldewyn, Weigelt,
* Corresponding author. Department of Industrial Design, National Cheng Kung Kanwisher, & Jiang, 2013). Most studies which aim to identify the
University, No.1 University Rd., East District, Tainan, Taiwan. Tel.: þ886 6 eye-tracking locus of young children with ASD report an emerging
2757575x54324. consensus that detailed characterizations at the level of eye
** Corresponding author. movements in response to fixating and tracking visual stimuli are
E-mail addresses: [email protected] (C.-H. Chen), iammimosa@gmail.
com (I.-J. Lee), [email protected] (L.-Y. Lin).
important (Falck-Ytter, Bolte, & Gredeback, 2013). Other research

http://dx.doi.org/10.1016/j.chb.2015.09.033
0747-5632/© 2015 Elsevier Ltd. All rights reserved.
478 C.-H. Chen et al. / Computers in Human Behavior 55 (2016) 477e485

(Durham University News, 2013) says that children with ASD might 2. Related work
be missing crucial nonverbal indicating behaviors, which likely
causes them to not recognize or understand nonverbal gestures and Of all the assistive technologies used to support people with
social cues like facial expressions and relevant gestures that usually autism, the most thoroughly researched is VM, an evidence-based
aid social interaction. Missing these cues generally has a negative instructional strategy and a medium-tech assistive technology
effect on their social interaction skills and the flow of their (Odom, Collet-Klingenberg, Rogers, & Hatton, 2010) and a form of
communication (Mundy, Sigman, Ungerer, & Sherman, 1986), observational learning in which desired behaviors are acquired by
because people with ASD cannot judge other people's expressions watching a videotape demonstration and then imitating of the
and emotional states, or understand the intentions and internal target behavior of the model (Charlop-Christy, Le, & Freeman, 2000).
activities of others (Krasny, Williams, Provencal, & Ozonoff, 2003). However, VM may have some weak points for training children with
Neither can they respond with appropriate gestures, postures, or ASD in social skills. Evidence shows that computer technology-
proximity (Ryan & Ni Charragain, 2010)da defect in what re- based interventions (CBIs) like VM used on the iPod Touch, iPad,
searchers have called Theory of Mind ability: the skill to view things and other tablet computers are therapeutically effective for teaching
from other people's perspective and to understand the mental functional, social, and behavioral skills to children with ASD (Ayres
states of others (Smith, 2006)di.e., the ability to empathize (Baron- & Langone, 2005; Bellini & Akullian, 2007). The flexibility and
Cohen & Belmonte, 2005; Baron-Cohen, Leslie, and Frith (1985). portability of modern devices make it much easier for them to learn
Therefore, children with ASD, who normally pay more attention to and develop new skills. For example, Cihak, Fahrenkrog, Ayres, and
inanimate objects than to nonverbal social cues, need to be taught Smith (2010) evaluated the efficacy of VM delivered through a
specific visual indicating behaviors involved in social interactions, handheld device (video iPod) and of using the system of least
and must learn to pay attention to the nonverbal social cues of the prompts to assist elementary school students to transition between
people they meet and talk with in order to understand social locations and activities within the campus. Four students with
reciprocity (Martins & Harris, 2006; McPartland, Webb, Keehn, & autism learned to manipulate a handheld device to watch video
Dawson, 2011). That is why we believe it necessary to teach chil- models. An ABAB withdrawal design was used to assess a functional
dren with ASD to pay attention to some pivotal social signals and relationship between VM and the performance of independent
ignore others. transitions by the student. The A-B-A-B design represents an
The motivation of this research is derived from a practical attempt to measure a baseline (the first A), a treatment measure-
problem that helps improve the social interaction of children with ment (the first B), the withdrawal of treatment (the second A), and
ASD. Autism remains a medical mystery. No single approach is the re-introduction of treatment (the second B). In other words, the
effective for alleviating its core symptoms. Children with ASD find it A-B-A-B withdrawal design involves two parts: (1) gathering
difficult to focus on specific social cues. They tend to pay attention baseline information, using a treatment, and then measuring its
to insignificant objects. Other studies, without considering the effects; and (2) measuring a return to baseline or what happens
entire context, used only static or dynamic images as the learning when the treatment is removed, reusing the treatment, and then
materials to train children with ASD. However, using static or measuring the change. After the intervention began, all participants
fragmented images is too limited and not ecologically valid. Dy- began transitioning more independently, but their percentage levels
namic videos are advantageous, but children with ASD have trouble fell when the intervention was withdrawn, which showed that VM
focusing their attention on dynamic videos. Because of this, chil- methods with a video iPod were useful for intervention.
dren with ASD cannot always be attracted by the media. Therefore, Moreover, Young and Posselt (2012), using the Transporters DVD
we used augmented reality (AR) technology to attract their atten- as a VM learning tool to attract the attention of children with ASD,
tion. In addition, to decrease their visual stress and loading, we focused on the expressions of animated human faces on toy trains,
chose suitable content that was less than 45 s long, focused on buses, and other vehicles (transporters) that are characters in a
specific social signals, and combined the AR technology with a video story which illustrates, names, and describes emotions in
video modeling (VM) strategy. Although other researchers have some common social situations. They reported that their partici-
already provided evidence that AR is useful for training typically pants' ability to judge the emotions of others had significantly
developing children (Chen, Wu, & Zhung, 2006; Chen & Su, 2011), improved. This showed that modeling is a key way for humans to
using AR is a novel approach for training children with ASD to make learn behavior (Mason & Ganz, 2011). In related VM applications for
judgments about the facial expressions and emotions of others. teaching children with ASD how to identify nonverbal social reci-
Moreover, AR learning activities have been proposed in many procity cues, Blum-Dimaya, Reeve, Reeve, and Hoch (2010) used
studies which demonstrate that an AR system not only provides various facial expressions in photos and videos to develop the
students the basics and is flexible and innovative (Chang, Wu, & communication skills of people with ASD, which enabled them to
Hsu, 2013; Pan, Cheok, Yang, Zhu, & Shi, 2006), but that it also focus on the specific visual representations and facial cues from
positively increases the motivation to learn (DiSerio, Ib
~ ez, &
an which the facial emotions of others can be determined. Moreover,
Kloos, 2013). AR attracts the attention of children with ASD and Axe and Evans (2012) used VM to train 3 five-year-olds with
encourages them to maintain their focus on nonverbal social cues. pervasive developmental disorder not otherwise specified (PDD-
Other experiments (Escobedo, Tentori, Quintana, Favela, & Garcia- NOS), an ASD, to respond to 8 facial expressions. The training ma-
Rosas, 2014) support that claim. However, few studies have used terial was a video of an adult modeling a response to each facial
AR as a training tool for autistic children's social skills. Because of expression. Two of the children correctly responded to all facial
the developmental deficit in children with ASD that prevents them, expressions after viewing the video models once or twice. This
in social situations, from easily and correctly judging and under- suggested that VM was effective for helping the children observe
standing the emotions of other people, we used AR to strengthen and imitate simple feature points such as facial expression.
their attention on achieving this goal, and enabling them to un- Tetreault and Lerman (2010) showed that VM in a multiple baseline
derstand the perceptions and judgments of facial expressions and across contexts design was useful for teaching 3 children diagnosed
emotions. We hypothesized that adding AR to VM with a storybook with an ASD to initiate and maintain a conversation with a
(ARVMS) to augment virtual visual hints that indicate and amplify conversant. The children were taught to make eye contact and to
the nonverbal social cues in videos will improve VM and encourage initiate a conversation without being prompted. The treatment
children with ASD to focus on specific parts of the videos. included both presenting a target video and reinforcing the target
 Download English Version:

https://daneshyari.com/en/article/350194

Download Persian Version:

https://daneshyari.com/article/350194

Daneshyari.com