Energy 221 (2021) 119775

Contents lists available at ScienceDirect

Energy
journal homepage: www.elsevier.com/locate/energy

AI-assistance for predictive maintenance of renewable energy systems

Won Shin a, 1, Jeongyun Han b, 1, Wonjong Rhee b, *
a
ONYX InSight Analytics Ltd, Nottingham, NG7 2TU, United Kingdom
b
Seoul National University, Department of Transdisciplinary Studies, 1 Gwanak-ro, Gwanak-gu, Seoul, 08826, Republic of Korea

a r t i c l e i n f o a b s t r a c t

Article history: Although promising results of high-performance AI algorithms have been reported in recent predictive
Received 4 March 2020 maintenance researches, most of the existing studies merely deal with AI-only solutions and do not
Received in revised form consider the interaction between humans and AI. In this study, we explicitly focus on the benefits of
30 December 2020
interactions where a human inspector is assisted by AI solutions. A case study is conducted for predictive
Accepted 3 January 2021
Available online 6 January 2021
maintenance of wind farms, where endoscopic images were used for bearing fault detection. The
experiment consisted of 54 technical inspectors and 2301 images collected over 138 wind turbines, and
each inspector was shown images and asked to identify bearing faults in the absence and presence of AI-
Keywords:
AI-Assistance
assistance. The results showed that AI-assistance had a statistically significant impact on improving the
Predictive maintenance technical inspector’s specificity and time efficiency. The level of improvement was dependent on the
Image-based diagnostics level of expertise, where the generalist group showed greater improvements in specificity and time
Wind energy efficiency (24.6% and 25.3%, respectively) when compared with the specialist group (4.7% and 6.4%,
Bearing fault respectively). Both groups responded positively on the reuse intention and usefulness of AI-assistance,
and the change in cognitive load was not statistically significant.
© 2021 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license
(http://creativecommons.org/licenses/by/4.0/).

1. Introduction such activities for in-service equipment are called predictive

maintenance.
In the renewable energy industry, operation and maintenance For predictive maintenance, the industry has actively embraced
(O&M) costs are an essential part of commercial success and play a new technologies that utilize sensors and imaging devices [3e8]. In
crucial role in important measures such as levelized cost of energy this study, we focus on image-based inspection, where visual images
(LCOE) [1,2]. To minimize O&M cost, it is critical to identify faults in collected from various devices such as endoscopes and thermal
the early stages. Once a fault progresses to a more severe stage, the imaging cameras are utilized [3,4]. With the latest imaging devices,
cost for repair can become exponentially larger and the time until inspectors can examine critical mechanical parts that were previ-
full repair can become extensively longer. Compared with other ously difficult to access (e.g., internal components of a gearbox).
machinery industries, the impact can be excessive in the renewable Therefore, image-based inspection is becoming a core part of
energy industry. For instance, consider an offshore wind farm. If the routine maintenance. As an example, a leading wind turbine
damage to a major component is found late, manufacturing and manufacturer has included endoscopic inspection of a generator in
transportation of a massive component and scheduling of a crane its scheduled service list [6].
vessel for lifting will be required. Because the logistics can be The basic mode of image-based inspection is the human-only
complicated, the wind turbine might need to be kept off for several approach where the human inspectors in the field determine if a
months, thus resulting in a large loss of revenue in addition to the fault is present. With the recent advances in deep learning tech-
repair cost. Therefore, it is important to predict a failure in advance niques, numerous studies have analyzed the potential of the AI-only
such that catastrophic equipment damage can be prevented and approach and demonstrated promising results. For example, a
time-consuming repair scheduling can be done in time. Generally, previous study [9] applied convolution neural networks (CNNs) for
fault detection of photovoltaic cell defects and achieved 93.02%
accuracy. Another study [10] employed region-based CNN (ReCNN)
* Corresponding author. to automate the training of object detection in a real-time manner,
E-mail addresses: [email protected] (W. Shin), [email protected] while one study [11] used ReCNN to detect five types of structural
(J. Han), [email protected] (W. Rhee).
1 surface damages on bridges and achieved a mean average precision
These authors contributed equally.

https://doi.org/10.1016/j.energy.2021.119775
0360-5442/© 2021 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
W. Shin, J. Han and W. Rhee Energy 221 (2021) 119775

(mAP) of 87.8%. A recent survey indicated that predictive mainte- were evaluated by two-way ANOVA based on two fac-
nance is expected to be one of the first fields where AI-based torsdexistence of AI-assistance and level of task proficiency. The
technologies will be successfully put into practice [12]. evaluation was conducted on both performance and perception of
Despite the impressive performance and increasing interest, the human inspectors; performance was evaluated based on the results
AI-only approach has its own limitations. First, a sufficiently large- of the inspection tasks in terms of specificity, sensitivity, and time
scale dataset is required for the training of an AI-only solution, and efficiency, and perceptions were evaluated using questionnaires in
it takes time and effort to collect the labeled dataset [13]. terms of cognitive load, intention to reuse, and usefulness. Even
Furthermore, data collection might be extremely difficult, if not though the experiment was a classification task, area under the
impossible, for some failure modes that occur rarely. For such rare receiver operating characteristics (AUROC) were not considered
cases, AI-only solutions will be helpless, whereas we can expect because they could not be calculated for human involved experi-
experienced inspectors to be additionally cautious toward ments. Based on this comprehensive investigation, we discuss the
addressing such situations. Second, the current AI-only solutions expected impacts of AI-assisted solutions when it is deployed to the
can typically detect five or fewer failure modes where a real in-field industrial fields.
inspection often requires handling of a large number of failure
modes. This issue is pronounced for in-situ inspections where all 2. Background
the inspections have to be performed during normal operation. For
instance, the ISO 15243 standard specifies 16 classes of failure In human-only inspection scenarios, where human inspectors
modes for bearing fault diagnosis [14]. make decisions manually, several issues have to be considered.
Because AI-only solutions are unlikely to be able to perform fully First, the inspector’s performance is highly dependent on their
automated fault diagnosis on their own, employing the AI-assis- expertise level. In the experiment by Smith-Bindman R et al. [24],
tance approach has garnered attention in cases where human in- the rate of false-positive examinations by less experienced physi-
spectors are assisted by AI algorithms [13,15]. In this integrated cians was about 50% higher than that of more experienced physi-
approach, AI provides diagnostic suggestions to human inspectors cians in screening mammography. This issue is particularly
to assist them in making a final decision. Such assistance can be problematic for the renewable energy industry, which lacks spe-
expected to lead to better predictive maintenance by overcoming cialists who are well trained and experienced. Second, it is difficult
several limitations of human inspection, such as fatigue and vari- to ensure consistency of diagnostic decisions among multiple in-
ability among inspectors [16,17]. In this aided decision-making spectors. In the experiment by Gulshan et al. [25], seven US certi-
process, measuring the performance of AI-assistance systems in fied ophthalmologists were asked to grade the severity levels of
isolation is not sufficient to understand its full effect. Thus, it is diabetic retinopathy based on photographs of the retinal fundus.
crucial to investigate the comprehensive effect to actual users. In the The results confirmed that the consistency among ophthalmolo-
machinery field, however, previous studies have mainly focused on gists was very poor. Out of 683 cases, only 20% cases showed
the isolated performance of AI-only solutions (e.g., Refs. [18e20]). complete agreement among the ophthalmologists. Lastly, human
Especially, in the field of renewable energy, the concept of AI- inspectors who analyze a large number of images for a long time
assistance has not been investigated sufficiently, and it is the can make more mistakes and work inefficiently because of mental
main subject of this study. fatigue and cognitive overload [17]. With the recent advance of
When investigating the comprehensive effect of AI, task profi- imaging devices, high-quality image data allow human inspectors
ciency of a human inspector should be considered. We can expect to perform fault diagnosis on a wider range of critical parts of
generalists with relatively low task proficiency to benefit from AI machines that were inaccessible before [5]. However, there is a
suggestions, but its influence to the specialists with professional definite limit for human inspectors to handle this growing size of
inspection skills might or might not be positive [21]. In addition, the image data adequately.
effects of AI-assistance need to be investigated not only in terms of To address the limitations of human-only inspection, re-
performance but also in terms of perception. Perceptions such as searchers have applied machine vision approaches to develop AI-
perceived cognitive load, usefulness, and intention to reuse are based solutions for image-based fault diagnosis. CNN is the most
crucial human factors to be considered when a new technology has common type of deep learning algorithm proven to be effective for
to be introduced [22,23]. Even when a new technology has many image-based diagnosis in the medical field [25], and it is certainly
proven technical benefits for improving the performance of users, applicable to the image-based machine fault diagnosis as well. This
its adoption is often unsuccessful when there is a lack of acceptance algorithm automatically extracts important features from an image
because of the users’ poor perception of it. Therefore, investigating which are required for fault diagnosis. Multiple failure modes can
the perception on AI-assistance is an essential research topic for its be trained with a sufficient amount of training data, and the AI
effective application. model can determine whether there is a fault on a previously un-
In this regard, this study aims to investigate the effect of AI- seen image. For example, Miaoyiquan et al. [26] used an AI model
assistance on both performance and perception of human in- based on CNN to detect three types of faults in power cables and
spectors by considering their task proficiency. For this purpose, the achieved 81%e87% accuracy. Cha et al. [11] used the CNN model to
research questions were as follows: detect structural surface damages on bridges and achieved a mean
average precision of 87.8%. By employing AI-based solutions,

How does AI-assistance affect the performance and perception technical inspection can be performed in a fully automated manner
of human inspectors? based on the image dataset with high performance.

How do the effects differ between the generalist group and Although the performance of AI-based models has significantly
specialist group? improved, the AI-only approach still has several shortcomings. The
first is regarding the datasets for building these AI models. The
To address these questions, we developed an AI model based on models require a sufficient amount of data to learn important
a deep learning algorithm, specifically convolutional neural patterns for fault inspection, which means that they perform poorly
network (CNN), utilizing the labeled endoscope image dataset. The when the dataset is not available. Therefore, Brynjolfsson and
AI model assisted human inspectors’ in the tasks of detecting Mitchell [13] suggested “large data sets exist or can be created
bearing faults for treatment in this study. The effects of AI models containing input-output pair” as one of the seven criteria for
2
W. Shin, J. Han and W. Rhee Energy 221 (2021) 119775

assessing whether AI can automate a particular task. However, affect human’s life and thus ethics and regulation immediately
because a large amount of cost and time are required to collect and become a crucial part. This aspect can potentially slow down the
maintain such large datasets, it is not easy to develop and imple- process of adopting AI technology. For instance, CAD has been
ment these AI-based solutions to the real field in a short period of developed and proven to be effective since 1990s, but its uptake in
time. In particular, in some cases of fault modes that occur very the clinical practice has been limited [39]. In contrast, adopting AI
rarely, collecting sufficient data with high quality labels to build AI in the energy field is primarily decided based on the economic
models is challenging. benefits, and not so much on ethics or regulation. Therefore, the
The next shortcoming is regarding the number of fault types adoption can be much faster with an immediate impact. Second,
that can be detected by AI models. AI-based solutions developed in energy field has both generalists and specialists while medical field
previous studies have been focused on a limited number of fault has only specialists. In the medical field, stringent regulations are in
types, while there are more types of faults required in the real place and only specialists such as licensed radiologists are qualified
technical inspection; the ISO 15243 standard classifies the types of to work on diagnosis. In the energy field, however, both skilled
bearing faults into 16 categories [14]. For example, the AI model in specialists and less skilled generalists can work on diagnosis. In fact,
Ref. [11] was trained to detect merely five types of fault modes, and more and more diagnostic tasks are being handled by generalists as
[26] used three types of fault modes to train its AI model. This predictive maintenance becomes a common and repeated practice.
situation indicates that there are still fault modes that the AI model The benefit of assistance system is known to be dependent on the
cannot handle, which makes it difficult for the AI-only approach to proficiency level [21,36], and a thorough investigation is required to
perform a fully automated inspection. understand the effects of AI assistance to the specialists and gen-
Therefore, employing the solution as an AI assistant to human eralists in the energy field.
users can make a meaningful contribution to fault diagnosis. In this
approach, the AI model assists human inspectors in making a final 3. Dataset and AI model
decision on technical diagnostic investigations by providing addi-
tional suggestions or information [15]. In this aided decision- To investigate the effects of AI-assistance on human inspectors,
making process, measuring the AI models’ performance in isola- we focused on bearing fault diagnostics of wind turbines. Wind
tion is not sufficient to completely understand its effect. Instead, it turbines are designed to convert wind’s kinetic energy into electric
is crucial to investigate the effects on human inspectors in a energy through rotating components and an electrical generator.
comprehensive manner. However, to the best of our knowledge, no Naturally, mechanical bearings are used to constrain the motions
published work in the field of predictive maintenance has evalu- and to reduce friction between the rotating parts. In this section, we
ated the effects of AI approaches on human inspectors. Instead, explain our endoscope image dataset and how we have constructed
recent studies mainly focused on the performance improvement of the AI model for the AI-assisted system.
AI-only approach (e.g., Refs. [18e20]). To complement and address
the shortcomings in both AI-only and human-only approaches, it is 3.1. Dataset of endoscope bearing images
necessary to comprehensively investigate the effects of the AI-
assistance approach in which the two approaches are integrated. A total of 3073 endoscope images of rolling-element bearing
In the medical field, the integrated approach has become well were collected from 138 wind turbine gearboxes and main bear-
established where image data are actively used for diagnostic tasks. ings. All data collections were performed using endoscopy as a part
In this field, the development of a stand-alone expert system that of in-situ inspection of wind turbines in operation. Because of the
provides definitive diagnosis was attempted before the 1990s [27]. nature of endoscopy, the position and orientation of the bearings in
Recently, the role of the system has been shifted from decision- the images varied. The inspected wind turbines consisted of various
maker to task-assistance which provides objective and reliable models manufactured by different manufacturers, and their oper-
decision support for human physicians. This process is called ating age ranged from just deployed to over ten years. The endo-
computer-assisted diagnostics (CAD) [27,28]. CAD has been prac- scopes used for inspection were also of various models, and the
tically applied in the clinic since 1998 [29], and many studies have resulting images different in brightness, resolution, and image
shown that CAD significantly improves radiologists’ task perfor- quality. The rolling element bearing and three examples of endo-
mance by overcoming the shortcomings of the human-only fault scopic images in normal and abnormal conditions are shown in
diagnosis [30e35]. In addition, researchers in the medical field Fig. 1.
have examined the system’s effect considering the radiologist’s Among the collected images, some were not appropriate for this
proficiency level for diagnosis tasks because radiologists may have study. Therefore, the inappropriate images were removed from the
different rooms for performance improvement being affected by dataset based on the following rules: (1) if the rolling bearing el-
the assistance system [21,36]. The effect of incorrect detection by ements took up less than 20% of the image, (2) if the image was out
CAD has been investigated at more specific levels, such as false of focus, and (3) if the surface of the bearing was not visible enough
positives and false negatives because the system cannot provide due to excessive oil or grease. After the filtering procedure, 772
100% correct answers [37,38]. Also, the effects of the system on user images were removed, and the remaining 2301 images were used
perception have to be considered. Performance gains using a new for this study.
technological tool are often obstructed by the users’ unwillingness
to accept and use the available system. Studies on the technology 3.2. Ground truth labels and data division
acceptance model revealed that both the system’s diagnostic per-
formance and the user’s perception of the system are important Ground truth labeling was required to build the AI model and
factors driving the user’s willingness to accept the system [40]. conduct the experiments in this study. Ideally, longitudinal obser-
Therefore, it is necessary to measure the user’s perception to vational records of the wind turbines or independent and more
comprehensively assess the effect on users. accurate results based on vibration test or disassembly inspection
While many essential lessons have been learned in the medical would be used for creating the ground truth labels. In our study,
field, there are fundamental differences between medical field and however, such information was unavailable. To address the issue,
energy field. First, energy field can be more receptive to the AI we referred to the rigorous labeling process developed in the
technology. In the medical field, diagnosis accuracy can directly medical field [25] based on a majority voting over a group of expert
3
W. Shin, J. Han and W. Rhee Energy 221 (2021) 119775

Fig. 1. Image dataset example: Rolling element bearing (far left); an endoscope image of the rolling element in the normal condition (left); and two endoscopic images in abnormal
conditions (right) are shown. For the images in abnormal conditions, the fault areas are marked with a yellow circle and a yellow box. The markings are only for illustration
purposes, and they are not available in the dataset. (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

examiners. Following the process, images were labeled by three the inspector’s perceived cognitive load, reuse intention, and use-
expert inspectors, and majority voting was applied to determine fulness (details of the metrics will be further explained in
the ground truth labels. Before starting the labeling process, the subsection 4.3). The data measured by these metrics were analyzed
inspectors were provided with a technical standard including the to investigate the overall effect of the AI-assisted approach.
criteria for abnormality. After the labeling process, 1415 images
were labeled as normal, and 886 images were labeled as abnormal. 4.1. Participants
The labeled dataset was divided into training and test datasets.
Among the 2301 bearing images, 2101 images were randomly A total of 54 engineers participated in this study. All participants
selected and assigned to the training dataset and the remaining 210 were field engineers in the wind energy industry, but not all par-
images were assigned to the test dataset. The training dataset was ticipants had specialty areas directly relevant to bearing fault in-
used to build a classification model based on a deep learning al- spection. Depending on their specialty areas, they were divided
gorithm, CNN (further details are provided in Section 3.3). The test into generalist and specialist groups, as shown in Table 1. The
dataset was used to evaluate the classification model’s generalist group consisted of 34 engineers and technicians with a
performance. basic understanding of machinery and bearing fault symptoms.
However, they had limited experience in the bearing fault diagnosis
3.3. AI model area because they were mainly engaged in the O&M of wind farms
and typically performed various maintenance tasks of wind turbine
For the AI-assistance experiment, a high-performance deep equipment. The specialist group consisted of 20 engineers with
learning model based on CNN was developed for bearing fault more than three years of experience in bearing fault diagnosis. They
detection. The input image was resized to 150  150  3 pixels were either inspection engineers or engineering consultants, and
(height  width  RGB channel). Our CNN model had four con- their routine duties included bearing fault diagnosis.
volutional layers and two fully connected layers. Batch normaliza-
tion and dropout layers were adopted to improve model 4.2. Procedure and inspection task
performance and a standard softmax layer was used to classify
whether each input image was normal or abnormal. For training, The experiment procedure of this study is shown in Fig. 2. For
2101 training images were used where data augmentations (rota- the experiment, half of the participants in each generalist and
tion by 90 , 180 , and 270 and left-right flip) were applied to the specialist group were randomly selected and assigned to teams 1
training dataset to enhance the generalization performance. To and 2. In the instruction step, all participants were first informed
handle the imbalance between the number of normal images and about their right to withdraw from the experiment at any time
the number of abnormal images, data augmentation was applied without any consequences. Then, they were explained about the
more extensively to the abnormal samples. The AI model showed procedure and the inspection task of classifying endoscopic images
high performance when evaluated over the test dataset; the spec- into normal and abnormal. In addition, the inspectors were famil-
ificity was 0.91 and the sensitivity was 0.89. iarized with the two different user interfaces for human-only
setting (Fig. 3-a) and AI-assistance setting (Fig. 3-b) before start-
4. Experimental settings ing the experiment.
The participants were asked to perform the inspection task over
We designed an experiment using the AI model to measure the two rounds throughout the procedure. For the inspection task, a
effects of AI-assistance on the human inspector. The experiment total of 34 problems were developed based on 34 images that were
was designed to have two main factors. The first factor was the randomly selected from the test dataset. The problems were evenly
existence of AI-assistance in the inspection taskdunassisted or split into two different problem setsdset A and Bdand used in the
assisted. The second factor was human inspectors’ level of task two rounds of inspection tasks across the different teams. This
proficiencydgeneralist or specialist. Throughout the experimental experimental design was for canceling out the selection bias among
procedure, the two groups of inspectors were asked to perform the problem sets and participant teams. The ratio of normal to
bearing fault identification tasks with and without AI-assistance. abnormal in set A and B was 9 to 8 and 8 to 9, respectively.
The effects of AI-assistance were analyzed in terms of both per- In the first round, the human-only setting was applied so that
formance and perception. For performance, we focused on speci- participants performed the inspection task without AI-assistance
ficity, sensitivity, and time efficiency. For perception, we focused on (Fig. 3-a). In the second round, otherwise, an AI-assisted setting
4
W. Shin, J. Han and W. Rhee Energy 221 (2021) 119775

Table 1
Participant demography.

Demography Generalist (n ¼ 34) Specialist (n ¼ 20)

Gender e Number of males (% of males) 34 (100%) 19 (95%)

Average work experience in related areas 6.38 years 8.15 years
Major Mechanical engineering 33 17
Electrical, electronic engineering 14 11
Others 7 6
Specialty area Inspection and failure analysis 18
Bearing specialist 2
Operation and maintenance 31
Gearbox design and analysis 3
Nationality Korea 25 5
US 7
Japan 5
UK 4
India 3
Others 4 1

Fig. 2. Experimental procedure.

was applied, so the participants were provided with the probability to a questionnaire designed to assess the cognitive load of the task.
of each class estimated by the AI model (Fig. 3-b). In this round, the Particularly, at the end of the second round, surveys for assessing
participant was also informed about the AI model’s performance, so participants’ intention to reuse and usefulness of the AI-assistance
that they could establish their trust level on the provided proba- were conducted. In this survey, the participants could voluntarily
bility. Between the two rounds of inspection tasks, a break time was leave their own feedback regarding AI-assistance as well as the
provided to prevent side effects due to fatigue or stress. overall experiment.
At the end of each round, the participant was asked to respond

5
W. Shin, J. Han and W. Rhee Energy 221 (2021) 119775

Fig. 3. User interface for the inspection task.

4.3. Performance metrics Table 2

Performance metrics.

The participants’ performances on the inspection task were Metric Equation

assessed using three metrics: specificity, sensitivity, and time effi- Specificity True Negative
ciency (Table 2). Specificity and sensitivity are the most commonly True Negative þ False Positive
used statistical measures for diagnostic systems [41]. Specificity Sensitivity True Positive
measures the proportion of actual normality (negatives) that are True Positive þ False Negative
Time efficiency Number of images in the task
correctly identified as normal. It is also called the true negative rate.
Time to complete the task ðminutesÞ
High specificity means that actual normality is rarely mistaken as
abnormal, so false positives are few. Sensitivity measures the pro-
portion of actual abnormality (positives) that is correctly identified
using questionnaires adapted from Refs. [44e46] by considering
as abnormal. It is also called the true positive rate or recall. High
the context of this study. Participants responded to the question-
sensitivity means that actual abnormality is rarely overlooked, and
naires after each round without knowing the scores of the in-
hence, false negatives are few. The terms “positive” and “negative”
spection task. All items in the questionnaires were assessed using a
refer to the presence or absence of an abnormality condition, which
5-point Likert scale (strongly disagree ¼ 1; strongly agree ¼ 5), and
is the condition of interest. Apart from the two metrics for diag-
example items are provided in Table 3 with their reliabilities
nostic systems, time efficiency of the inspector was also analyzed
(Cronbach’s alpha).
because time is an essential constraint for industrial tasks [16]
because it is directly related to the cost. Time efficiency was defined
as the average number of images that were analyzed per minute. 5. Analysis results

4.4. Perception metrics To investigate the effects of an AI-assisted approach on the in-
spector’s performance and perception, a descriptive analysis was
The participants’ perceptions of the inspection task and AI- first conducted on all the measurements of the metrics. Then, a
assistance were assessed in terms of cognitive load, intention to two-way mixed ANOVA was conducted to investigate the impact of
reuse, and usefulness. Cognitive load refers to the effort used in the the two main factors, i.e., the existence of AI-assistance and task
working memory to perform the task [42]. If decision support proficiency of human inspectors, on the three performance mea-
through AI-assistance reduces the user’s effort to complete the fault surements and cognitive load. When significant effects were found,
diagnosis task, the cognitive load metric is expected to be reduced. post-hoc analysis was subsequently conducted using Bonferroni’s
Intention to reuse is the user’s behavioral intention to use the multiple comparison tests to investigate the two factors’ effects in
system again in a similar situation in the future [43]. Usefulness can detail. For measurement of intention to reuse and usefulness,
be defined as the degree to which a person believes that using the which were measured once at the end of the second round, a t-test
system would enhance his or her job performance [44]. A good fit was performed to investigate the significant differences between
between task and technology positively influences the perceived the two inspector groups. Missing values were excluded from the
intention to reuse and usefulness. All perceptions were measured analysis.
6
W. Shin, J. Han and W. Rhee Energy 221 (2021) 119775

Table 3
Perception metrics.

Metric Number of items Example Cronbach’s alpha

Cognitive load 4 This task required much mental and perceptual activity to complete 0.75e0.76
Intention to reuse 3 If I had a software that provides AI-assistance, I will perform bearing inspection by using it 0.77
Usefulness 4 Using AI-assistance would enhance the effectiveness of my job 0.95

5.1. Effects on performance (Table 5eb). There was also no significant interaction between
the two factors.
5.1.1. Specificity
The results of the analysis regarding the inspectors’ specificity
5.1.3. Time efficiency
are summarized in Table 4. In both unassisted and assisted settings,
The results of analysis regarding inspectors’ time efficiency are
the specialist group showed a higher specificity than the generalist
summarized in Table 6. In the absence of AI-assistance, the
group. The specificity for both groups was improved by AI-
specialist group showed higher time efficiency than the generalist
assistance; in particular, this improvement was greater in the
group, but this difference was reversed by AI-assistance provision
generalist group than the specialist group (Table 4ea). Generalists’
(Table 6ea); the generalist group’s time efficiency was improved by
specificity was improved by 24.6% (from M ¼ 0.65 to M ¼ 0.81),
25.3% (from M ¼ 5.88 to M ¼ 7.38), whereas the generalist group’s
whereas the specialists’ specificity was improved by 4.7% (from
efficiency was improved by 6.4% (from M ¼ 6.28 to M ¼ 6.68). In
M ¼ 0.85 to M ¼ 0.89). Two-way mixed ANOVA (Table 4eb) yielded
time efficiency, a significant main effect was only found in the ex-
a significant effect both for the level of task proficiency
istence of AI-assistance factor (F(1, 47) ¼ 5.52, p < 0.05, Table 6eb).
(F(1,52) ¼ 12.94, p < 0.001) and for the existence of AI-assistance
In the following post-hoc analysis, no significant difference was
(F(1,52) ¼ 9.30, p < 0.01); however, the interaction between the
found.
two factors was not significant (F(1,52) ¼ 3.40, p < 0.01). In the
results of post-hoc analysis (Table 4ec), specificity improvement of
the generalist group due to AI-assistance was found to be signifi- 5.2. Effects on perception
cant (difference ¼ 0.15, p < 0.01). Under the human-only setting, a
significant difference was observed in the specificity between the 5.2.1. Cognitive load
two groups (difference ¼ 0.20, p < 0.01), but this difference was not The results of the analysis of the inspectors’ cognitive load are
significant when AI-assistance was provided (difference ¼ 0.09, summarized in Table 7. The descriptive analysis shows that the
p ¼ 0.55). As expected, assisted specialists significantly out- overall cognitive load of specialists was lower than that of gener-
performed the unassisted generalists (difference ¼ 0.24, p < 0.001). alists (Table 7ea). This was not surprising because specialists are
more familiar with the task and use less effort than generalists.
Interestingly, however, changes in cognitive load because of AI-
5.1.2. Sensitivity assistance provision occurred in opposite directions in the two
The analysis results regarding inspectors’ sensitivity are sum- groups; the generalist group’s cognitive load was reduced by 7.0%
marized in Table 5. In sensitivity measurements, the differences (from M ¼ 5.88 to M ¼ 7.38), whereas the specialists’ cognitive load
between the two groups were very small, and the improvement in was improved by 2.4% (from 3.29 to 3.37). In ANOVA, a significant
sensitivity because of the AI-assistance was also very slight main effect was found only in the level of task proficiency
(Table 5ea). ANOVA yielded no significant main effects both for the (F(1,46) ¼ 4.34, p < 0.01, Table 7eb). In the post-hoc analysis, all the
level of task proficiency and the existence of AI-assistance differences were not statistically significant, but there was a

Table 4
Analysis results of specificity.

Note. **p < 0.01, ***p < 0.001 (2-tailed).

7
W. Shin, J. Han and W. Rhee Energy 221 (2021) 119775

Table 5
Analysis results of sensitivity.

Table 6
Analysis results of time efficiency.

Note. *p < 0.05 (2-tailed).

marginal difference between generalists and specialists under the 6.1. Advantages of AI-assistance
human-only setting (difference ¼ 0.68, p ¼ 0.08).
Our experiment was limited to the task of bearing diagnostics of
wind turbines, and caution is required for interpreting the impli-
cations to the general image-based diagnostics. With this in mind,
5.2.2. Intention to reuse and usefulness
we addressed the advantages that might be widely applicable to AI-
Table 8 presents the results of the analysis regarding inspectors’
assisted diagnostics.
intention to reuse AI-assistance and its’ usefulness. Both groups
showed positive perceptions regarding both metrics. In terms of
intention to reuse, a positive willingness was found in the gener-
6.1.1. Overall effects of AI-assistance
alist group (M ¼ 3.95, SD ¼ 0.89) and the specialist group (M ¼ 3.93, Regarding the performance metrics, AI-assistance improved all
SD ¼ 0.64). In terms of usefulness, the specialist group showed a
three metrics for both generalists and specialists. While only some
slightly higher response (M ¼ 3.97, SD ¼ 0.46) than the generalist
of the performance improvements were statistically significant, it is
group (M ¼ 3.79, SD ¼ 0.89). The differences between the two
still promising that the effects were positive for specialists and even
groups regarding the metrics were not significant.
for sensitivity. The specialists were technical experts with years of
experience in bearing fault diagnostics. For sensitivity, AI alone
performed worse than the generalists, but it was able to enhance
6. Discussion the human’s performance for both generalists and specialists.
Regarding the perception metrics, the effects of AI-assistance on
The effects of AI-assistance on human inspectors were assessed cognitive load were small (7.0% reduction for generalists and 2.4%
for performing bearing fault diagnostics. The results show the po- increase for specialists) and statistically insignificant. Therefore, the
tential for a positive influence, and a few important related topics experimental results indicate that AI-assistance can be adopted
are discussed in this section. without a significant downside on cognitive load. In addition, the
8
W. Shin, J. Han and W. Rhee Energy 221 (2021) 119775

Table 7
Analysis results of cognitive load.

Note. *p < 0.05 (2-tailed).

Table 8 specialist’s time might be better utilized by working on image la-

Analysis results of intention to reuse and usefulness. beling (creating ground truth labels for AI model training) or by
Level of task proficiency t supporting a team of generalists. Similar results were obtained for
Existence of AI-assistance tasks in other industries, including legal services [49] and medical
Generalist Specialist
M (SD) M (SD) diagnostics [50].
Intention to reuse 3.81 (0.89) 3.97 (0.46) 0.71
Usefulness 3.95 (0.88) 3.93 (0.64) 0.11

6.1.3. Time efficiency

High sensitivity (i.e. recall) and low specificity are common
characteristics of fault diagnostics because the penalty for a false
metrics for intention to reuse and usefulness were positive for both negative is far larger than the penalty for a false positive [51].
groups (3.79e3.97 out of 5). According to the technology accep- Because all inspectors are concerned about missing a fault, even
tance model [46,47], intention to reuse and usefulness are known generalists show a high sensitivity performance, as shown in
as the major factors driving an individual’s willingness to accept a Table 6-(a). In the absence of AI-assistance, the generalists achieved
new technology tool in an organizational setting, and it is encour- as high a sensitivity performance as that of the specialists (0.93 vs.
aging that all of the three metrics were neutral or positive. 0.93) at the cost of low specificity performance (0.65 vs. 0.85). The
AI model’s detection threshold, however, was chosen neutrally and
therefore the AI model achieved a sensitivity of 0.89 and a speci-
6.1.2. Impacts on generalists ficity of 0.91. The high specificity performance compared to the
The influence of AI-assistance on specialists was limited, but its human inspectors is an important factor to be considered, and its
impact on generalists was relatively large. Consequently, the dif- influence on time efficiency might be important.
ference in performance between the generalist and specialist When AI-assistance was provided, the time efficiency was
groups became statistically insignificant for all three performance improved for both generalists and specialists where the improve-
metrics under AI-assistance. In particular, generalists even out- ment was larger for generalists than for specialists (25.3% vs. 6.4%).
performed specialists in terms of time efficiency. Similar results The improvement in time efficiency indicates that the inspectors
have been reported in other disciplines. For instance, a similar were able to make faster decisions with AI-assistance without
improvement was observed in the field of clinical radiology where sacrificing sensitivity and specificity. Therefore, a reasonable
the performance of the generalists was improved with Computer- conjecture is that AI-assistance was helpful in making quick de-
aided Detection [21]. cisions for obvious cases while it was not harmful for the borderline
If AI-assistance can enable generalists to perform as well as cases. The effect might have been even stronger for obviously
specialists, the implication can be significant. Typically, it takes negative cases because of the AI model’s superior specificity per-
years for a generalist to become a specialist, and any task that re- formance. This explanation was also supported by a participant’s
quires a specialist is constrained by the cost and availability of the feedback. The participant mentioned that “The AI assistance helped
specialist. With AI-assistance, the constraints can be removed or me weed out obviously normal and abnormal cases quickly”.
mitigated without sacrificing the quality of diagnostics, and skilled Therefore, time saving was more likely from obvious cases than
specialists can be released from routine diagnostic works and the from subtle borderline cases and probably more from relieving
resources can be invested in new areas [48]. As an example, a inspectors from the worry of obtaining false negatives.
9
W. Shin, J. Han and W. Rhee Energy 221 (2021) 119775

6.2. Toward frequent and periodic predictive maintenance specialists is available and the collected dataset results in a high-
performance AI model for a given diagnostics task, there is a
Many fields are actively adopting image-based predictive good chance that AI-assistance can be used for accelerated
maintenance. For instance, structure monitoring based on drone deployment of predictive maintenance.
camera inspection is another growing field. Similar to renewable
energy and medical fields, the goal is to perform periodic in- 6.3. Limitations of this work
spections such that a possible problem in buildings, bridges, con-
struction sites, or telecommunication sites can be detected early. In 6.3.1. Cognitive load
all such fields, the recent revolution in image classification through There are a few things that can be addressed regarding cognitive
deep learning algorithms is leading the trend of pursuing frequent load. For real image-based diagnostics, the inspectors are required
and periodic predictive maintenance. A few issues related to the to process a large number of images over many hours. Because
role of AI-assistance in predictive maintenance are discussed mental fatigue can lead to frequent mistakes and degraded effi-
below. ciency, it is important to reduce the cognitive load or at least pre-
One of the basic problems that is encountered with predictive vent it from increasing. In our experiment, each of set A and set B
maintenance is the increased workload. Inspection needs to contained only 17 images. For a real work environment with a
become a periodic and routine process, where image data collec- much larger number of images, it is unclear if the relative influence
tion is the first part of work and reading and diagnosis of the images of AI-assistance on cognitive load will be positive or negative. It
is the second part of the work. In the early phase of adopting pre- might become positive if the mental fatigue of repeated image
dictive maintenance, the inspectors often need to use image de- screening is reduced. In contrast, it might become negative because
vices to collect the image data manually. While this part cannot of information overload [22] or the initial lack of trust on AI-
benefit from AI-assistance, it can be automated in some of the fields assistance [52]. A further study is required to address this issue.
if the manufacturers start to include built-in image-based moni-
toring systems as equipment. The reading and diagnosis part, 6.3.2. Human performance, AI performance, and their relative
however, is unlikely to be fully automated in the near future relationship
because AI-only is limited in the sense that it can recognize only the In our experiment, the AI model’s performance was comparable
faults that it has learned well from the training data. When there to a human’s performance. For a given diagnostic task, however, the
are new types of problems that can occur, humans are required to situation can be different and there are a few important factors to
be kept in the loop and detect possibly disastrous cases in their be considered. The first that needs to be considered is human
early stages. performance. If the diagnostic task is too difficult even for spe-
As long as the human’s role cannot be eliminated, AI- cialists, there might be no incentive to pursue predictive mainte-
assistance’s utility might only be in the form on time efficiency. nance. Note that AI models of today are typically trained with
As in our experiment, even a direct and simple integration of AI- human generated labels, and therefore, AI models are unlikely to
assistance might be sufficient to improve the human inspector’s perform well, either. The second factor that needs to be considered
time efficiency. Additionally, more sophisticated approaches might is AI performance. If the diagnostics task is simple and sufficiently
helpful. For instance, a careful trade-off between the AI model’s easy that an AI model can diagnose all types of faults with almost
specificity and sensitivity might result in an additional improve- perfect accuracy, then using AI-only would be sufficient for pre-
ment depending on the characteristics of human decisions. In our dictive maintenance. For such a simple and easy task, however, the
case, human-only was biased toward a higher sensitivity at the cost chances are that the situation is already serious and obvious that
of lower specificity. Another possible approach is to train multiple even simple rule-based or other hand-crafted algorithms might
AI models with different goals. A version of AI models might be perform as well as any sophisticated AI models. In this case, auto-
designed and trained with the goal of confidently detecting the mation of diagnostics can be easy without using AI models or AI-
obvious cases while simply providing ‘unknown’ as the output for assistance.
the difficult cases. Another version of the AI model might be When a diagnostic task is workable enough for specialists and
designed and trained with the goal of confidently identifying only a yet difficult enough for generalists, AI-assistance can be a viable
particular fault type that commonly occurs. If high quality datasets solution. The usefulness of AI-assistance, however, is not an easy
are available, many other versions can be learned, resulting in a one to determine. Human performance depends on many elements
good chance of additional improvement. In this case, however, the such as training and development of domain knowledge. AI per-
effects on perception should be carefully studied for choosing the formance also depends on many elements such as image quality
ideal subset of AI models. In addition, some of the extreme versions and dataset enhancement. Furthermore, each task can have many
might be used in AI-only mode where their utility is for automat- details and can be complex, so that it is difficult to clearly define.
ically filtering out the trivial cases. When all these factors are considered together, a careful study
Apart from time efficiency, another important role of AI- might be necessary on a task-by-task basis to draw a reliable
assistance is supporting generalists such that they can perform as conclusion regarding the usefulness of AI-assistance. In this regard,
well as specialists. This issue has been already addressed, but it is our study can only serve as a case study, and the results should be
worthwhile to point out its explicit effect on predictive mainte- carefully interpreted when applying them to other tasks. When
nance. As visual inspection with imaging devices is being widely ultimate cost-saving through predictive maintenance is very large
adopted as a routine maintenance procedure, the demand for as in the renewable energy sector, repeating a study like ours can be
skilled inspectors is sharply increasing. However, years are required a very prudent and timely topic to work on.
for a generalist to become a specialist, and the availability of
qualified inspectors is a bottleneck for quickly adopting predictive 7. Conclusion
maintenance. This was one of the reasons why predictive mainte-
nance was only slowly adopted in the past decades. However, with Predictive maintenance is a crucial component for the successful
the recent deep learning solutions that can provide close-to-human operation of renewable energy systems, and adopting the latest AI
performance on image classification tasks, the problem can be technology is expected to be inevitable for scaling predictive
greatly alleviated through AI-assistance. If a minimum number of maintenance to sufficiently frequent and periodic inspections. In
10
W. Shin, J. Han and W. Rhee Energy 221 (2021) 119775

this work, we studied a representative case of bearing fault [15] James WH, Daugherty PR. Collaborative intelligence: humans and AI are
joining forces. Harv Bus Rev 2018;2018(July-August).
detection where endoscope images are collected for the inspection.
[16] Megaw ED. Factors affecting visual inspection accuracy. Appl Ergon
We considered the limitations of human-only and AI-only ap- 1979;10(1):27e32.
proaches and evaluated the benefits of AI-assistance. The results [17] Reiner BI, Krupinski E. The insidious problem of fatigue in medical imaging
show that all factors, specificity, sensitivity, and time efficiency, can practice. J Digit Imag 2012;25(1):3e6.
[18] Mostefa Khelil C, Amrouche B, Benyoucef A, Kara K, Chouder A. New intelli-
be improved with AI-assistance, for both the generalist group and gent fault diagnosis (IFD) approach for grid-connected photovoltaic systems.
specialist group. In fact, AI-assistance was helpful for inspectors Energy 2020;211:118591.
even when the performance of AI-only was worse than human [19] Brkovic A, Gajic D, Gligorijevic J, Savic-Gajic I, Georgieva O, Gennaro S. Early
fault detection and diagnosis in bearings for more efficient operation of
performance. The perception effects were also investigated, and rotating machinery. Energy 2017;136:63e71.
positive results were obtained. Possible future works include ex- [20] Bode G, Thul S, Baranski M, Müller D. Real-world application of machine-
periments under a regular operation mode and extended experi- learning-based fault detection trained with experimental data. Energy
2020;198:117323.
ments in the renewable energy fields outside of wind farm [21] Fisichella VA, Ja €derling F, Horvath S, Stotzer PO, Kilander A, Båth M,
operation. Hellstro€m M. Computer-aided detection (CAD) as a second reader using
perspective filet view at CT colonography: effect on performance of inexpe-
rienced readers. Clin Radiol 2009;64(10):972e82.
Author statement [22] M escu I, Sutton SG. The effects of decision aid structural restrictiveness on
ala
cognitive load, perceived usefulness, and reuse intentions. Int J Account Inf
Won Shin: Conceptualization, Methodology, Software, Valida- Syst 2015;(17):16e36.
[23] Dale L, Ronald L. Task-technology fit and individual performance. MIS Q
tion, Investigation, Resources, Writing e original draft, Writing e
1995;19(2):213e36.
review & editing, Project administration, Jeongyun Han: Method- [24] Smith-Bindman R, Chu P, Miglioretti D, Quale C, Rosenberg R, Cutter G,
ology, Validation, Formal analysis, Writing e original draft, Writing Geller B, Bacchetti P, Sickles EA, Kerlikowske K. Physician predictors of
mammographic accuracy. J Natl Cancer Inst 2005;97(5):358e67.
e review & editing, Visualization, Wonjong Rhee: Conceptualiza-
[25] Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A,
tion, Methodology, Resources, Writing e original draft, Writing e Venugopalan S, Widner K, Madams T, Cuadros J, Kim R, Raman R, Nelson P,
review & editing, Supervision Mega JL, Webster DR. Development and validation of a deep learning algo-
rithm for detection of diabetic retinopathy in retinal fundus photographs.
J Am Med Assoc 2016;316(22):2402.
Declaration of competing interest [26] Wang M, Tong W, Liu S. fault detection for power line based on convolution
neural network. In: Proceedings of the 2017 international conference on deep
The authors declare that they have no known competing learning technologies - ICDLT ’17; 2017. p. 95e101.
[27] Doi K. Computer-aided diagnosis in medical imaging: historical review, cur-
financial interests or personal relationships that could have rent status and future potential. Comput Med Imag Graph 2007;31(4e5):
appeared to influence the work reported in this paper. 198e211.
[28] Doi K. Current status and future potential of computer-aided diagnosis in
medical imaging. Br J Radiol 2005;78(1):s3e19.
Acknowledgments [29] van Ginneken B, Schaefer-Prokop CM, Prokop M. Computer-aided diagnosis:
how to move from the laboratory to the clinic. Radiology 2011;261(3):
This work was supported by the National Research Foundation 719e32.
[30] Brown MS, McNitt-Gray F, Pais R, Shah K, Qing P, Da I, Aberle R, Goldin G. CAD
of Korea, South Korea; (NRF) grant funded by the Korea government in clinical trials: current role and architectural requirements. Comput Med
(MSIT) (Grant No. NRF-2017R1E1A1A03070560 and Grant No. NRF- Imag Graph 2007;31(4e5):332e7.
2020R1A2C2007139). [31] Cooper JG, West RM, Clamp SE, Hassan TS. Does computer-aided clinical de-
cision support improve the management of acute abdominal pain? A sys-
tematic review. Emerg Med J 2011;28(7):553e7.
References [32] Freer TW, Ulissey MJ. Screening mammography with computer-aided detec-
tion: prospective study of 12,860 patients in a community breast center.
[1] Irena. The power to change: Solar and wind cost reduction potential to 2025. Radiology 2001;220(3):781e6.
2016. [33] Birdwell RL, Bandodkar P, Ikeda DM. Abbreviations: BI-RADS breast imaging
[2] Bosch J, Staffell I, Hawkes AD. Global levelised cost of electricity from offshore reporting and data system CAD computer-aided detection PPV positive pre-
wind. Energy 2019;189:116357. dictive value. Radiology 2005;236:451e7.
[3] ReliaPlant. Predictive Maintenance Explained, https://www.reliableplant.com/ [34] Helvie MA, Hadjiiski L, Makariou E, Chan HP, Petrick N, Sahiner B, Lo SC,
Read/12495/preventive-predictive-maintenance. Freedman M, Adler D, Bailey J, Blane C, Hoff D, Hunt K, Joynt L, Klein K,
[4] Life Cycle Engineering. Preventive and Predictive Maintenance, https://www. Paramagul C, Patterson SK, Roubidoux MA. Sensitivity of noncommercial
lce.com/pdfs/The-PMPdM-Program-124.pdf. computer-aided detection system for mammographic breast cancer detec-
[5] Addepalli S, Roy R, Axinte D, Mehnen J. In-situ’ inspection technologies: trends tion: pilot clinical trial. Radiology 2004;231(1):208e14.
in degradation assessment and associated technologies. Procedia CIRP [35] Ko JM, Nicholas MJ, Mendel JB, Slanetz PJ. Prospective assessment of
2017;59:35e40. computer-aided detection in interpretation of screening mammography. Am J
[6] GE Energy Power Conversion. External Report on maintenance Schedule Roentgenol 2006;187(6):1483e91.
Checklist for GEPC - RM Commissioning & services generators for wind tur- [36] Halligan S, Altman D, Mallett S, Taylor S, Burling D, Roddie M, Honeyfield L,
bine application. 2017. McQuillan J, Amin H, Dehmeshki J. Computed tomographic colonography:
[7] Hur S, Recalde-Camacho S, Leithead W. Detection and compensation of assessment of radiologist performance with and without computer-aided
anomalous conditions in a wind turbine. Energy 2018;124:74e86. detection. Gastroenterology 2006;131(6):1690e9.
[8] Zhou D, Zhang H, Weng S. A novel prognostic model of performance degra- [37] Mani A, Napel S, Paik DS, Jeffrey Jr RB, Yee J, Olcott EW, Prokesch R, Davila M,
dation trend for power machinery maintenance. Energy 2014;78:740e6. Schraedley-Desmond P, Beaulieu CF. Computed tomography colonography:
[9] Akram M, Li G, Jin Y, Chen X, Zhu C, Zhao X, Khaliq A, Faheem M, Ahmad A. feasibility of computer-aided polyp detection in a first reader paradigm.
CNN based automatic detection of photovoltaic cell defects in electrolumi- J Comput Assist Tomogr 2004;28(3):318e26.
nescence images. Energy 2019;189:116319. [38] Alberdi E, Povyakalo A, Strigini L, Ayton P. Effects of incorrect computer-aided
[10] Ren S, He K, Girshick R, Sun J, Faster R-CNN. Towards real-time object detection (CAD) output on human decision-making in mammography. Com-
detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell puter Assisted Radiology and Surgery 1004; 11(8):909-918.
2017;39(6):1137e49. [39] Eom Sb. The development of decision support systems research: a biblio-
[11] Cha YJ, Choi W, Suh G, Mahmoudkhani S, Büyüko €ztürk O. Autonomous metrical approach. Edwin Mellen Press; 2007.
structural visual inspection using region-based deep learning for detecting [40] Shibl R, Lawley M, Debuse J. Factors influencing decision support system
multiple damage types. Comput Civ Infrastruct Eng 2018;33(9):731e47. acceptance. Decis Support Syst 2013;54(2):953e61.
[12] Bughin J, Hazan E. Artificial intelligence e the next digital frontier. 2017. [41] Gonçalves VM, Delamaro ME, Nunes FDLDS. A systematic review on the
[13] Brynjolfsson E, Mitchell T. What can machine learning do? Workforce im- evaluation and characteristics of computer-aided diagnosis systems. Rev Bras
plications: profound change is coming, but roles for humans remain. Science Eng Biome dica 2014;30(4):355e83.
2017;358:6370. American Association for the Advancement of Science. [42] Sweller J. Cognitive load during problem solving: effects on learning. Cognit
2017;1530e1534. Sci 1988;12(2):257e85.
[14] ISO. ISO 15243:2004(en): rolling bearings d damage and failures d terms, [43] Venkatesh V. Determinants of perceived ease of use: integrating control,
characteristics and causes. 2004. intrinsic motivation, and emotion into the technology acceptance model. Inf

11
W. Shin, J. Han and W. Rhee Energy 221 (2021) 119775

Syst Res 2000;11(4):342e65. [48] Benke K, Benke G, Benke K, Benke G. Artificial intelligence and big data in
[44] Venkatesh V, Smith RH, Morris MG, Davis GB, Davis FD, Walton SM. User public health. Int J Environ Res Publ Health 2018;15(12):2796.
acceptance of information technology: toward A unified view. User Accept IT [49] Reid M. A call to arms: why and how lawyers and law schools should embrace
MIS Q 2003;27(3):425e78. artificial intelligence, vol. 50. Univ Toledo Law Rev; 2018.
[45] Hart SG, Staveland LE. Development of NASA-TLX (task load index): results of [50] Jha S, Topol EJ. Adapting to artificial intelligence: radiologists and pathologists
empirical and theoretical research. Adv Psychol 1998;52(C):139e83. as information specialists. J Am Med Assoc 2016;316(22):2353.
[46] Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of [51] Knop K, Olejarz E, Ulewicz R. Evaluating and improving the Effectiveness of
information technology. MIS Q 1989;13(3):319. visual inspection of Products from the Automotive industry. Cham: Springer;
[47] Ortega Egea JM, Roma n Gonzalez MV. Explaining physicians’ acceptance of 2019. p. 231e43.
EHCR systems: an extension of TAM with trust and risk factors. Comput Hum [52] Jianlong ZS, Chen F. Effects of uncertainty and cognitive load on user trust in
Behav 2011;27(1):319e32. predictive decision making. Int Fed Inf Process 2017;10516(October):23e39.

12