electronics

Article
Development of Evaluation Criteria for Robotic Process
Automation (RPA) Solution Selection
Seung-Hee Kim

Department of IT Convergence Software Engineering, Korea University of Technology & Education (KOREATECH),
1600, Chungjeol-ro, Dongnam-gu, Cheonan-si 31253, Chungcheongnam-do, Republic of Korea;
[email protected]

Abstract: When introducing a robotic process automation (RPA) solution for business automation,
selecting an RPA solution that is suitable for the automation target and goals is extremely difficult for
customers. One reason for this difficulty is that standardised evaluation items and indicators that can
support the evaluation of RPA have not been defined. The broad extension of RPA is still in its infancy
and only a few studies have been conducted on this subject. In this study, an evaluation breakdown
structure for RPA selection was developed by deriving evaluation items from prior studies related to
RPA selection and a feasibility study was conducted. Consequently, a questionnaire was administered
three times, and the coefficients of variation, content validity, consensus, and convergence of factors
and criteria were measured from the survey results. All of these measurement results are reflected
in the final suitability value that was calculated to verify the stability of the evaluation system and
evaluation criteria indicators. This study is the first to develop an RPA solution selection evaluation
standard and the proposed evaluation breakdown structure provides useful evaluation criteria and a
checklist for successful RPA application and introduction.

Keywords: robot process automation (RPA); RPA selection; RPA evaluation criteria

Citation: Kim, S.-H. Development of

1. Introduction
Evaluation Criteria for Robotic Robotic process automation (RPA) is a business-process-based software solution that
Process Automation (RPA) Solution automates and processes simple and repetitive tasks using software robots [1–5]. Specifi-
Selection. Electronics 2023, 12, 986. cally, it processes organisational structured data and provides rule-based outputs [6,7]. The
https://doi.org/10.3390/ term was coined in the early 2000s by the Blue Prism company, which introduced software
electronics12040986 robots based on screen scraping technology [8]. RPA automates human behaviour and
Academic Editors: Sanjay Misra,
tasks, whereas artificial intelligence (AI) automates analysis and decision making by imitat-
Robertas Damaševičius and ing intelligence and reasoning. Both technologies can enable various services individually
Bharti Suri or in combination [9]. RPA has facilitated innovation in the productivity improvement
of many industries [10]. It originated in the field of information systems as a disruptive
Received: 18 January 2023 innovation that, among other automation solutions, has had a profound effect on job de-
Revised: 12 February 2023
scriptions and work itself [11]. Since then, RPA adoption has grown because RPA does not
Accepted: 15 February 2023
require dedicated software development and is a low-cost solution that requires a small
Published: 16 February 2023
workforce and minimal implementation time to automate operations. It can automate the
functions of existing software, promote communication between IT and other departments,
and can more easily recognise coding-related capabilities and knowledge compared to
Copyright: © 2023 by the author.
other development methods. In particular, software integration is made possible based
Licensee MDPI, Basel, Switzerland. on various tasks in existing environments, which facilitates low complexity with high
This article is an open access article efficiency and productivity [12], and inevitable process improvement [7,13] in Industry 4.0.
distributed under the terms and Therefore, automation using AI technology has recently been widely introduced across
conditions of the Creative Commons many industrial sectors. RPA has been applied to more than 20 diverse business areas,
Attribution (CC BY) license (https:// including internal organisational operations, functional improvements, risk management
creativecommons.org/licenses/by/ audits, data analysis, and reporting [2,3], and many enterprises have successfully integrated
4.0/). RPA [2,6,14–18].

Electronics 2023, 12, 986. https://doi.org/10.3390/electronics12040986 https://www.mdpi.com/journal/electronics

Electronics 2023, 12, 986 2 of 22

According to Gartner [19], survey result scores for RPA product and service levels
based on a five-point scale in 2020 were as follows: WorkFusion, 4.34; Microsoft, 4.20;
Automation Anywhere, 4.18; Pegasystems, 4.16; UiPath, 4.15; Kofax, 4.12; ServiceeTrace,
4.11; NICE, 4.05; and Edge, 4.05.
According to McKinsey, the adoption rate for RPA in 2020 was 22% [20], which
exceeds that for artificial-intelligence-based computer vision (18%) and deep learning (16%)
solutions. Furthermore, the average annual growth from 2021 to 2024 was predicted to
be in the double digits. According to Gartner [19], intelligent process automation linked
to AI will further expand its market size and adoption, with 90% of global conglomerates
being expected to introduce RPA by 2022. Furthermore, Samsung SDS, LG CNS, POSCO
ICT, Grid One, Symation, Inzisoft, and EDENTNS have released RPA solutions and are
competing with UiPath and Automation Anywhere.
However, organisations that wish to implement RPA solutions have a wide range of
services to choose from and often have difficulty in selecting appropriate RPA solutions for
their characteristics. This is because there are no standards or guidelines for the evaluation
criteria used for selecting solutions. To address this issue, this study aimed to develop
evaluation criteria for RPA solution selection. Specifically, this study was designed to
evaluate both strategy and technology, and this is the first paper to propose an evaluation
index for RPA solution selection. The evaluation criteria derived in this study can be used
as a checklist for the introduction of RPA. Additionally, this paper is expected to serve as
both a theoretical and practical reference when revising national laws and systems related
to software projects.
The remainder of this paper is organised as follows. Section 2 presents the preliminary
research and related research associated with this study. Section 3 presents the detailed
research procedure and methodology adopted in this study. Section 4 describes a detailed
development process and evaluation criteria for RPA solution selection. Finally, Section 5
summarises the main conclusions of this study.

2. Preliminary Research
2.1. Screen Scraping
In the term RPA, ‘robotic’ does not refer to a physical robot, but a ‘computer process’,
in the sense that it replaces human cognitive work [14]. This implies that perception and
behaviour are connected intelligently. Therefore, when introducing RPA, it is necessary
to distinguish between the role of RPA in existing stereotyped processes and the role of
employees.
Figure 1 compares pre- and post-RPA business processing for an ‘Order Details Pro-
cessing Task’. Prior to applying RPA, an employee periodically logs in directly to the
system. After confirming and verifying orders, the employee applies prices and discount
rates that meet specific conditions, applies any additional discounts, and then charges the
post-delivery price. However, after applying RPA, the employee only needs to perform
the role of verifying order information based on contract terms and the other tasks are
completed entirely by the RPA software.
Electronics 2023,
Electronics 12,12,
2023, 986x FOR PEER REVIEW 33 ofof2224

Figure1.1.Differences
Figure Differencesbetween
betweenmanual
manualand
androbotic
roboticprocesses.
processes.Adaptation
Adaptationbased
basedononRefs.
Ref.[14,21].
[14,21].

The
Thetechnology
technologythat thatenables
enablesthis
thisprocess
processisiscalled
calledscreen
screenscraping
scraping(also
(alsoknown
knownasasweb web
scraping
scrapingororweb webharvesting).
harvesting).ThisThisterm
termrefers
referstotoa atechnique
techniqueused usedtotocapture
captureand anddecode
decode
text
textand
andbitmap
bitmapdata dataon on aa computer
computer screen that is
screen that is used
used primarily
primarilyin inweb
webenvironments
environmentsto
toextract
extractand
andconvert
convertstructured
structureddatadata from
from output
output datadata into
into a human-readable form
a human-readable form[8].
[8].
Screen
Screen scraping allows users to specify the outline of a box around icons and labels[22],
scraping allows users to specify the outline of a box around icons and labels [22],
which
whichthen
thenallows
allows robots
robots to to
identify and
identify andclick areas
click thatthat
areas areare
notnot
accessible through
accessible throughexisting
exist-
limited pixel-based coded screen scraping [1]. A screen scraper communicates
ing limited pixel-based coded screen scraping [1]. A screen scraper communicates with with the
system as if it was an ordinary user, explores the user screen of the system,
the system as if it was an ordinary user, explores the user screen of the system, and reads and reads
information
information[23]. Additionally, aa screen
[23]. Additionally, screenscraper
scrapercan canserve
serve asas a component
a component of aoflarger
a larger
pro-
program outside the information system [23]. A single-time scaler retrieves
gram outside the information system [23]. A single-time scaler retrieves all information all information
from
froman anold
oldinformation
informationsystemsystemandandstores
storesititinina anew
newdatabase,
database,butbutuses
usesa acontinuous
continuous
scaler
scaler to keep the existing system active and retrieve information on thesystem
to keep the existing system active and retrieve information on the systemscreen
screen
when requested. Based on this principle, RPA identifies the patterns through which users
when requested. Based on this principle, RPA identifies the patterns through which users
perform tasks on existing legacy system screens. Developing automated lists of tasks from
perform tasks on existing legacy system screens. Developing automated lists of tasks from
extracted patterns allows an RPA robot to repeat tasks directly in a graphical user interface
extracted patterns allows an RPA robot to repeat tasks directly in a graphical user interface
(GUI) automatically.
(GUI) automatically.
2.2. Comparative Studies on RPA Solutions
2.2. Comparative Studies on RPA Solutions
Kim [2] divided RPA solution functions into robots, robot managers, and script-editing
Kim
tools, as [2] divided
shown in FigureRPA solution
2, and functions
compared into
various robots,
RPA robot Ribeiro
solutions. managers,
et al.and script-edit-
[4] compared
ing tools, as shown in Figure 2, and compared various RPA solutions. Ribeiro
RPA intelligence functions between six RPA solutions with high market shares by dividing et al. [4]
compared
them RPA intelligence
into AI-related functions between
goals, technologies, six RPA solutions
and algorithms, as shownwith high market
in Figure 3. shares
by dividing them into AI-related goals, technologies, and algorithms, as shown
The results for each major RPA vendor presented in The Forrester (Cambridge, Mas- in Figure
3.
sachusetts) Wave Evaluation [24] were divided into solution functions and strategies. Scores
are provided on a scale of weak (0) to strong (5). These data represent an evaluation of
the top vendors in the RPA market and do not describe the entire vendor landscape. Each
vendor’s position on the vertical axis of the graphic indicates the strength of its current
offering [24]. The key criteria for these solutions include task and process discovery, port-
folio analysis, bot design and development, deployment and management, security and
governance, scaling experience, and architecture [24]. Placement on the horizontal axis
indicates the strength of vendor strategies [24]. This represents the product vision and
innovation roadmap, delivery and support models, financial performance, and partner
ecosystem [24]. The functional analysis results are presented in Figure 4.
 Electronics 2023, 12, 986 4 of 22
Electronics 2023, 12, x FOR PEER REVIEW 4 of 24
Electronics 2023, 12, x FOR PEER REVIEW 4 of 24

Figure
Figure2.2.
Figure Comparison
Comparisonofofof
2.Comparison RPARPA
RPA solution
solution
solution functions.
functions.
functions. Adaptation
Adaptation
Adaptation based
basedon
based onRef.
Ref.on Ref. [2].
[2].
[2].

Electronics 2023, 12, x FOR PEER REVIEW 5 of 24

Figure3.3.Comparison
Figure Comparisonof of technologies
technologies and and
goalsgoals associated
associated with
with AI. AI. Adaptation
Adaptation based onbased on Ref. [4].
Ref. [4].
Figure 3. Comparison of technologies and goals associated with AI. Adaptation based on Ref. [4].

The results for each major RPA vendor presented in The Forrester (Cambridge, Mas
The results for each major RPA vendor presented in The Forrester (Cambridge, Mas-
sachusetts) Wave Evaluation
sachusetts) Wave Evaluation[24] [24] were
were divided
divided intointo solution
solution functions
functions and strategies
and strategies.
Scores are provided on a scale of weak (0) to strong (5). These data represent
Scores are provided on a scale of weak (0) to strong (5). These data represent an evaluation an evaluation
ofofthe
the top vendorsininthe
top vendors theRPARPA market
market andanddo do
not not describe
describe the entire
the entire vendor vendor landscape
landscape.
Each vendor’s position on the vertical axis of the graphic indicates the strength
Each vendor’s position on the vertical axis of the graphic indicates the strength of its cur- of its cur
rentoffering
rent offering [24].
[24].The
Thekey
keycriteria forfor
criteria these solutions
these include
solutions task and
include taskprocess discovery,
and process discovery
portfolio analysis,
portfolio analysis, bot
botdesign
designand anddevelopment,
development, deployment
deployment and management,
and management,securitysecurity
and governance,
and governance, scaling
scalingexperience,
experience, andand
architecture [24]. [24].
architecture Placement on theon
Placement horizontal
the horizonta
axis indicates the strength of vendor strategies [24]. This represents the product and
axis indicates the strength of vendor strategies [24]. This represents the product vision vision and
innovation roadmap, delivery and support models, financial performance, and partner
innovation roadmap, delivery and support models, financial performance, and partner
ecosystem [24]. The functional analysis results are presented in Figure 4.
ecosystem [24]. The functional analysis results are presented in Figure 4.

Figure 4. RPA scorecard (Q1 2021). Adaptation based on Ref. [24].

Figure 4. RPA scorecard (Q1 2021). Adaptation based on Ref. [24].

2.3. Studies on RPA Solution Evaluation Elements

The US Federal RPA Community of Practice [25] has proposed evaluation elements
for each department in terms of technical capabilities, process management, and opera-
tions, as shown in Table 1.

Table 1. Evaluation factors by dimensional classification of RPA product functions according to the
US Federal RPA Community of Practice. Adaptation based on Ref. [25].
Electronics 2023, 12, 986 5 of 22

2.3. Studies on RPA Solution Evaluation Elements

The US Federal RPA Community of Practice [25] has proposed evaluation elements for
each department in terms of technical capabilities, process management, and operations, as
shown in Table 1.

Table 1. Evaluation factors by dimensional classification of RPA product functions according to the
US Federal RPA Community of Practice. Adaptation based on Ref. [25].

Dimensional Classification Evaluation Factors

Workflow management
Process recording and reproduction
Process Self-study capability
Usability
Process engineering and evaluation
Visual creation tool
Instruction library
Full/partial automation capability
Automation
Component sharing
Test/debug control method
Usability
Centralised deployment, management, and
scheduling
Licensing structure
Management and Operation Scalability, availability, and performance
management
Exception management
Dashboard capability
Business and operational analysis capability

Korea’s Software Policy & Research Institute (SPRI) [2] divides evaluation elements
into technology, process operational impact, and risk management categories according
to the automation and maturity of vendors. Specifically, elements can be divided into the
introduction stage, technical architecture, technological policy, process strategy consistency,
and operational management. The details of these items are provided in Table 2.

Table 2. Key considerations for RPA evaluation according to Korea’s SPRI. Adaptation based
on Ref. [8].

Classification Major Consideration Items

Cost Model deployment, licensing, and operating costs
Introduction Usability System interaction and integration
stage Technical aspect OS/hardware requirements, and RPA deployment and operational capabilities
Vendor support Support capabilities, education, customer service, contracts, etc.
Vendor Pre-evaluated market awareness, existing performance in the same field, customer cases,
experience terms and conditions, and considerations
Product function Applies mutatis mutandis [25]
Technical
architecture Legal compliance, account and personal identification management, risk/security
Security
establishment assessment, authentication, data encryption/protection, process tracking
Review hardware/software requirements and virtual server design, multi-tenancy,
Architecture on-premise/cloud, permissions, availability/disaster recovery capabilities, network
capacity/performance management capabilities
Electronics 2023, 12, 986 6 of 22

Table 2. Cont.

Classification Major Consideration Items

Develop risk management strategies, authority management strategies, and code
Security policy
control strategies after evaluating the risks involved in the scope of implementation
Account/personal
Technological Service/network and system/
identification
policy application-level access management required
management
Understand system/application lines, capabilities/functionality, and user reviews, and
Privacy
interact with related systems to determine whether current security policies or use cases
protection
violate privacy policies depending on the type of data the system handles
Technological Consistency of RPA program usage and objectives
compatibility Appropriateness of RPA service distribution/operation model
[25] RPA program technical policy/architecture consistency

Process Strategic Corporate and institutional missions and objectives

compatibility Leadership priorities, strategies, and initiatives
[25] Calculations corresponding departmental and corporate objectives
Effect [25] Compliance/audit functions

Operation Operational management, change management, automation interruption response, code

Operation
and sharing, automation scheduling, etc.
management Standardised asset
Licence management, technical policy updating, RPA lifecycle management
management

For processes, only review items that are applicable to the RPA solution selection eval-
uation criteria were extracted from [8], reconstructed, and included in the listed categories.
Major RPA vendors attended the RPA Introduction Guide Seminar [26] sponsored
by the Korea Electronic Newspaper and announced evaluation criteria for RPA solution
selection. Here, Chan Sik Bong from KPMG proposed the selection of a solution with
sufficient references to prioritise enterprise-level introduction and stable construction
when introducing an RPA. Sean Lee, who is the managing director of Automation Any-
where, explained the derivation and verification of non-functional requirements, including
automation functional requirements, architectural requirements, and development con-
venience/operability/maintenance/security requirements. Gye Kwan Kim, who is the
CEO of Grid One, opined that ‘Korea’s IT environment should include not only company
businesses, similar cases, business areas, and investment efficiency (return on investment,
ROI), but also the ability to perform tasks in non-standardised GUI environments such as
ActiveX and Flash’. Myung Su Jo, who is the managing director of Deloitte, announced
that the prime considerations for RPA solution selection should be application capabilities,
technical compatibility, manufacturing capabilities, and pricing.

2.4. Business Structural Optimisation Studies on Improving RPA Operational Efficiency

Algorithms that minimise the number of robots [27] when introducing RPA and algo-
rithms that optimise area clustering and storage location allocation in process automation
cloud systems are important considerations for operational efficiency. Strategically, new
capabilities in terms of the average cost of automation, total investment cost, quality control,
optimisation of management, and control of automation productivity have been presented
as important factors for consideration [25]. Following in-depth experimentation on the
RPA tools available in the market, the authors of [28] developed a classification framework
for product categorisation and a methodology for selecting target tasks for robotic process
automation using human interface records and process mining techniques [29]. Addition-
ally, the authors of [30] developed an application to automate data acquisition process
management and control by applying an RPA implementation workflow.
 presented as important factors for consideration [25]. Following in-depth experimentation
on the RPA tools available in the market, the authors of [28] developed a classification
framework for product categorisation and a methodology for selecting target tasks for
robotic process automation using human interface records and process mining techniques
Electronics 2023, 12, 986 [29]. Additionally, the authors of [30] developed an application to automate data7 of
acquisi-
22
tion process management and control by applying an RPA implementation workflow.

3. Research Procedure
3. Research Procedureand
andMethodology
Methodology
TheThe
sequence and
sequence andmethod
methodof
of implementation adopted
implementation adopted in in this
this study
study are are summarised
summarised
in Figure
in Figure 5. 5.

Figure 5. Overview of research process and methodology.

Figure 5. Overview of research process and methodology.

3.1. 3.1.
Research Procedure
Research Procedure
TheThe firststep
first stepin
in this
this study
studywas
was to structure collected
to structure evaluation
collected items. Therefore,
evaluation com-
items. Therefore,
prehensively structured items that should be evaluated when selecting RPA
comprehensively structured items that should be evaluated when selecting RPA solutions solutions were
collected from existing resources, including literature reviews, press releases, and seminar
were collected from existing resources, including literature reviews, press releases, and
videos. It was necessary to consider the efficiency and productivity of the introducing
seminar videos. It was necessary to consider the efficiency and productivity of the intro-
organisation and to be practical from both strategic and construction perspectives. I applied
ducing organisation and to
a solution-lifecycle-level be practical
approach from both
progressing fromstrategic and construction
user or organisational perspectives.
requirements to
I applied
actual construction, management, and control tasks. Consequently, the initial introduction, re-
a solution-lifecycle-level approach progressing from user or organisational
quirements toinfrastructure,
functional, actual construction, management,
and vendor andwere
support aspects control tasks. comprehensively.
evaluated Consequently, the in-
itial introduction,
The second functional, infrastructure,
step was to derive evaluation and vendor
criteria support
for RPA solutionaspects were
selection. evaluated
Therefore,
I developed
comprehensively. a draft RPA solution evaluation standard based on the detailed evaluation
department and evaluation items finalised in the structured evaluation item results. The
proposed RPA solution selection evaluation system consists of three layers: evaluation
department, evaluation item, and evaluation criteria. Each layer is based on similarity
and the group names of the evaluation department and evaluation item were defined by
referencing existing resources [2,4,8,24–26].
The third step was to verify the RPA solution evaluation criteria. Therefore, the Delphi
survey method was used to verify the evaluation criteria for the proposed RPA solution.
The validity of the questionnaire was evaluated using a seven-point Ricardo scale, and
the coefficient of variation (CV), content validity ratio (CVR), conformity assessment (CA),
and convergence degree (CGD) of the questionnaire results were calculated. If all validity
measurements were satisfactory, then it was deemed that the RPA solution-phase criteria
were satisfied.
(1) Stability measured using the CV
The CV measures the value of measurement data and uses measured values as the
basis for determining the agreement between panels [31,32]. It is the ratio of the standard
deviation to the mean (average), as defined in Equation (1) [33]. Based on the study by
Electronics 2023, 12, 986 8 of 22

Khorramshahgol and Moustakis [34], it was judged that a CV value below 0.5 is stable, a
value of 0.5 to 0.8 is relatively stable, and additional questionnaires are required for a CV
above 0.8 [34].
SD (Standard Deviation)
CV = (1)
Mean
(2) CVR
The CVR is defined as the total number of exports divided by the number of ‘im-
portant’ responses [31], as shown in Equation (2). The effective minimum value of the
CVR based on the number of experts was determined by Lawshe [35,36]. In this study,
there were 11 experts, so it was judged that a CVR value of 59 or more would satisfy the
relevant conditions.  
nr − N2
CVR = N
(2)
2
Here, nr refers to the number of panel members indicating an item as ‘essential’, and
N refers to the total number of panel members.
(3) Consensus degree (CSD) and convergence degree (CGD)
To determine whether a panel is looking for agreement, the results presented by
Delbecq et al. [37] were applied to measure the CSD and CGD, where the CSD was re-
quired to be at least 0.75 and the CGD was required to be less than 0.5, as defined in
Equations (3) and (4).
Q3 − Q1
 
CSD = 1 − (3)
Median
Q3 − Q1
 
CGD = (4)
2
Median = median value
Q1 = first quartile, 25% of the total
Q3 = third quartile, 75% of the total
(4) CA
CA applies Equation (5) to the CVR, CSD, and CDG values calculated using the
equations presented above. As shown in Equation (5), CV, CVR, CSD, and CGD are all
considered to be ‘conforming’ in the RPA solution selection evaluation criteria.

CA = (CV ( x ) ≤ 0.5) ∩ (CVR( x ) ≥ 0.99) ∩ (CSD ( x ) ≥ 0.75) ∩ (CGD ( x ) ≤ 0.5) (5)

The final step is defining the RPA solution evaluation criteria. The Delphi survey
method verifies the N-order evaluation criteria and determines the appropriate evaluation
criteria for RPA solutions. Therefore, anonymous experts were asked about the evaluation
criteria for RPA solutions after reflecting on the opinions of experts in the first round and
re-questioning the revised evaluation criteria.

3.2. Research Methodology

In this study, the Delphi methodology was employed to develop indicators for the
developed RPA breakdown structure and selection criteria with help from experts. The
Delphi method refers to a ‘set of procedures to guide experts’ views on the issues they want
to predict and summarise them into a comprehensive judgement’ [38]. This method can
be used in scenarios where relevant research is insufficient or new evaluation standards
are to be developed [17,31]. This method can also be used as a judgement, decision
support, or prediction tool because Delphi surveys often help in understanding problems,
opportunities, and solutions, as well as developing predictions [22].
In this study, the evaluation department, evaluation items, and evaluation standard
of the RPA solution derived from existing sources were evaluated based on a seven-point
Electronics 2023, 12, 986 9 of 22

Ricardo scale. Currently, there are no international standards or guidelines for selection
criteria for RPA. To the best of my knowledge, this is the first study to address this problem.
The Delphi methodology is suitable in that expert opinions are considered as much as
possible to derive meaningful items by collecting informed ideas. The stability of factors
and criteria were measured using CV, CVR, CSD, CGD, and CA, and then filtered.
To apply the Delphi survey method, RPA consulting and construction experts were
defined as people with at least three years of relevant work experience. Delbecq et al. [37]
suggested that between 10 and 15 people should be included for an appropriate number of
members of the Delphi method group. Therefore, this study used a panel of 11 experts.

4. Developing Evaluation Criteria for RPA Solution Selection

4.1. Structuring Selected Evaluation Items
A total of 87 major criteria were derived and structured selected evaluation items were
obtained from the collection of candidate items for RPA evaluation criteria that should
be considered by an organisation when selecting RPA solutions, as described in previous
studies. The details of the collected criteria are presented in Table 3.

Table 3. Numbers of key criteria for the evaluation of RPA selection collected through preliminary
research.

Ribeiro
Kang et al. U.S. Kim Lu et al. Etnews
Ref. et al.
[8] [25] [2] [24] [26]
[4]
Number of
20 17 9 13 10 18
key criteria

4.2. Deriving Evaluation Criteria for RPA Solution Selection

The selection criteria were divided into evaluation categories, evaluation items, and
evaluation criteria. The evaluation department converged the contents collected from
existing resources [2,4,8,24–26] as closely as possible, resulting in categories of ‘introduction
strategy’, ‘functionality’, ‘technical architecture’, and ‘operational management’. Regarding
the evaluation criteria, the collected considerations and evaluation criteria defined in
previous studies were rearranged according to their affinity. Next, similar standard names
were renamed to a single name with the same meaning and only one duplicate item was
deleted for the same standard.
Finally, evaluation items were defined as comprehensive representations of the con-
siderations contained in each group to form a final refined evaluation criteria group. In
this process, I attempted to maintain the framework of the evaluation criteria groups by
referring to the results of prior research [2,4,8,24–26].
Consequently, seven evaluation items for the introduction strategy evaluation depart-
ment were evaluated based on economic validity, supply maintenance, technical compati-
bility, real-time decision-making support, strategic compatibility, and process.
The functionality evaluation items map robot management and operation, analy-
sis/categorisation/prediction, automation, and process evaluation criteria. The technical
architecture evaluation department derives security and architecture evaluation items and
maps detailed evaluation criteria for each evaluation item. The operational management
evaluation department derives operational and standardised asset management evaluation
items and defines the key evaluation criteria for RPA operations. The final evaluation
criteria for RPA solution selection are presented in Table 4. For the convenience of the com-
position of questionnaires and the preparation and analysis of the results of questionnaires
according to the evaluation index, the numbers under column ‘No.’ refer to the evalua-
tion items and criteria. Additionally, for the same purpose, the evaluation department,
evaluation items, and evaluation criteria are represented by I, II, and III, respectively.
Electronics 2023, 12, 986 10 of 22

Table 4. Derived results for evaluation criteria for RPA solution selection.

I. Category No. II. Evaluation Items III. Evaluation Criteria

Economic Price [26], cost elements (deployment, licensing, and operating expenses) [8],
1 validity ROI [24], investment efficiency [26]
Reference customer case holder [8,26], solution provider capability [26], product
Supplier vision [24], dealer market awareness [8], existing results in the same field [8],
2 maintenance
support terms and conditions [8], operability [26], serviceability [26], product and service
support [24], support capabilities, education and customer service [8]
Hardware/software requirements [8], development convenience [26], technical
Technological elements (OS requirements and technology, and capabilities required for RPA
3
compatibility deployment and operation) [8], technical compatibility [26], performance [24],
system interaction and integration [8]
1.
Introduction strategy Personal information protection (consideration of personal information protection
policies according to system/application lines, capabilities/functionality and user
4 Security policies
reviews, and interactive system data types) [8], account/personal identification
management [8]

5 Real-time decision Classification [4], cognition [4], information extraction [4], optimisation [4]
making support
Portfolio [24], revolutionary roadmap [24], risk management strategies, corporate
Strategic
6 and institutional missions and objectives [8], and leadership priorities, strategies,
compatibility
and initiatives according to risk analysis evaluation [8]
Iterative and regular process identification/discovery [8], bot idea [24], delivery
7 Process model [24], company business characteristics and business areas [26], calculations
matching departmental and company-wide objectives [8]
Bot platform model, security [24], availability [25], and quality analysis of
Robot management business performance (provided a graph of business performance) [2],
8 management and analysis [24], dashboard capability [25], licensing structure [25],
and operations
robot management functions [2], business performance management [25],
exception management
Analysis/ Artificial neural network (ANN) [4], neuro-linguistic programming [4], decision
9 categorising/ tree [4], recommendation system [4], computer vision cognition [4], Text
predicting mining [4], statistical technique [4], fuzzy logic [4], fuzzy matching [4]
2. Excel and SAP (ERP Solution) API support [2], instruction library [25], security
Functionality enhancement site response [26], security character recognition [26], information
security [26], bot development [24], bot design and development [24], atypical
10 Automation
GUI infrastructure program automation (X-Internet, Active X, Flash) [2], task
performance ability in standardised GUI environment [26], Hangul character
recognition ability [26], Hangul character recognition [2]
Technology (RPA service distribution and operational model competency) [8],
technology (RPA program technology policy/architecture compatibility) [8],
11 Process usability [25], architectural requirements are easy to derive [26], application
functions [26], workflow [25], self-learning capabilities [25], process greening and
reproduction [25], process engineering and evaluation [25]
Compliance with legal systems such as personal information protection [8],
account and personal identification management [8], data
12 Security
3. encryption/protection [8], application security [8], risk/security evaluation [8],
Technical authentication [8], process traceability [8]
Architecture On-premise/cloud [8], virtualisation server design [8], availability/disaster
13 Architecture recovery capabilities [8], permissions [8], network capacity [8], multi-tenancy [8],
performance management capabilities review [8]
Change management [2], operation management [2], automation scheduling [8],
4. 14 Operation
automation interruption accident response [8]
Operational management
Standardised Code sharing method [8], technical policy update, RPA lifecycle management [8],
15
asset management licence management [8]

4.3. RPA Solution Evaluation Criteria Verification

To verify the RPA solution evaluation criteria, experts with more than three years
of experience in RPA construction and operation in Korea were invited. These experts
consisted of RPA service supply groups such as RPA solution vendors, builders, consultants,
and customer groups that introduce and operate RPA services. Detailed information
regarding the final participating experts is provided in Table 5.
Electronics 2023, 12, 986 11 of 22

Table 5. Information regarding experts who participated in this study.

RPA Construction (Supplier Group) RPA Operations (Client Group)

Experts
a b 1 c d 1 e f g
AO, AA,
1 20 5 - - -
BR
Finance, manufacturing,
2 5 BR 3 BR 10 3
MIS, IT
3 - - - AO 1 3 Public
MIS, manufacturing,
4 - - - AA 20 4
logistics, R&D
5 10 BR 3 - - -
AA, BP, UP,
6 10 5 - - - -
AO
FCM, HR, SCM, CRM,
7 7 AA, AW 4 AA, AW 250 3
MFG
8 10 AO 3 - - - -
9 20 UP 3 - - - -
10 15 AO 4 - - - -
Aviation, production,
11 13 UP 4 UP 20 3
pharmaceutical, retail
1 Solution: AO (Automate One), AA (Automation Anywhere), BR (Brity RPA), BP (Blue Prism), UP (UiPath),

AW (A. Works). a Number of constructions. b Construction experience solution. c Number of years of RPA
construction experience. d RPA Operation solution. e Number of RPA systems introduced. f Introduction and
operation period. g Field of introduction operations.

A total of three Delphi surveys were conducted to verify the criteria for RPA solution
selection. The questionnaire questions were repeated in the form of 34 items for the first
round and the evaluation indicators for 27 items were included in the second round. The
initial development of evaluation departments, evaluation items, and evaluation criteria
reflects the results of validity measurements and suggestions for evaluation indicators.
Validation was performed three times and the main contents of each step of verification
are summarised below.
First Verification Overview: The validity of the evaluation criteria listed in Table 4 was
verified for each I. evaluation category, II. evaluation item, and III. evaluation criterion.
After deriving indicators satisfying the conditions of CV ≤ 0.7, CVR ≥ 0.59, CSD ≥ 0.75,
and CGD ≤ 0.5, six indicators were identified as appropriate, as shown in Table 6. The
column headings a , b , c , and d in Table 6 indicate the conformity for each value, where
‘00 represents ‘suitable’ and ‘10 represents ‘unsuitable’.
When revised carefully, the evaluation categories as a whole and evaluation criteria
for ‘Architecture’ and ‘Technical Architecture’ were considered as security evaluation items.
These criteria are the most stable standards for evaluation.
Next, the opinions of the first expert evaluation were incorporated. The RPA solu-
tion selection evaluation benchmark index was improved by reflecting the ‘Proposal of
Evaluation Criteria’ of experts for each questionnaire item. As a result, the names asso-
ciated with the evaluation department were consolidated from ‘customer introduction
strategy’, ‘functionality’, ‘development and operability’, and ‘operation management’ into
the name of ‘operation management system’. Regarding the evaluation items, the real-time
decision support and strategy integrity evaluation items of the introduction strategy evalu-
ation department, analysis/classification/prediction, and process evaluation criteria of the
functional evaluation department were rearranged.
Evaluation criteria for AI technology collaboration and expansion of the functional
evaluation department were added, including real-time decision support for the introduction
strategy evaluation department and analysis/category/prediction. To include the revised
evaluation criteria, the names of the evaluation items were revised as deemed necessary.
Electronics 2023, 12, 986 12 of 22

Table 6. First conformity assessment results.

No. Mean SD CV CVR CSD CGD a b c d Selection

1 6.27 0.75 0.12 1.00 0.83 0.50 0 0 0 0 O
2 6.09 0.51 0.08 0.82 1.00 0.00 0 0 0 0 O
I.
3 5.45 0.78 0.14 0.64 0.80 0.50 0 0 0 0 O
4 5.82 1.03 0.18 0.64 0.83 0.50 0 0 0 0 O
1 5.73 1.66 0.29 0.64 0.75 0.75 0 0 0 1 X
2 5.18 1.75 0.34 0.45 0.40 1.50 0 1 1 1 X
3 4.91 1.50 0.31 0.45 0.75 0.75 0 1 0 1 X
4 4.45 1.62 0.36 0.09 0.60 1.00 0 1 1 1 X
5 2.64 1.30 0.49 −0.82 0.67 0.50 0 1 1 0 X
6 3.64 1.61 0.44 −0.45 0.63 0.75 0 1 1 1 X
7 4.18 1.80 0.43 −0.27 0.25 1.50 0 1 1 1 X
II. 8 4.73 1.54 0.33 0.27 0.60 1.00 0 1 1 1 X
9 4.09 1.93 0.47 0.27 0.50 1.25 0 1 1 1 X
10 4.82 1.90 0.39 0.45 0.60 1.00 0 1 1 1 X
11 3.73 1.48 0.40 −0.27 0.38 1.25 0 1 1 1 X
12 4.73 1.66 0.35 0.09 0.60 1.00 0 1 1 1 X
13 5.64 0.88 0.16 0.82 0.83 0.50 0 0 0 0 O
14 6.00 1.04 0.17 0.82 0.67 1.00 0 0 1 1 X
15 4.73 1.54 0.33 0.45 0.80 0.50 0 1 0 0 X
1 5.55 1.78 0.32 0.64 0.67 1.00 0 0 1 1 X
2 5.64 0.98 0.17 0.82 0.70 0.75 0 0 1 1 X
3 5.09 1.24 0.24 0.45 0.75 0.75 0 1 0 1 X
4 4.82 1.53 0.32 0.45 0.70 0.75 0 1 1 1 X
5 2.73 0.96 0.35 −1.00 0.83 0.25 0 1 0 0 X
6 4.55 1.44 0.32 −0.09 0.25 1.50 0 1 1 1 X
7 4.18 1.53 0.37 −0.27 0.38 1.25 0 1 1 1 X
III 8 5.09 0.79 0.16 0.45 0.70 0.75 0 1 1 1 X
9 3.45 1.88 0.54 −0.27 0.13 1.75 0 1 1 1 X
10 5.82 0.94 0.16 0.82 0.75 0.75 0 0 0 1 X
11 4.64 1.15 0.25 −0.09 0.50 1.00 0 1 1 1 X
12 5.00 1.60 0.32 0.64 0.80 0.50 0 0 0 0 O
13 5.00 1.54 0.31 0.45 0.70 0.75 0 1 1 1 X
14 5.91 1.16 0.20 0.64 0.67 1.00 0 0 1 1 X
15 4.64 1.49 0.32 0.09 0.70 0.75 0 1 1 1 X
a CV fitness (0: true, 1: false). b CVR fitness (0: true, 1: false). c CSD Fitness (0: true, 1: false). d CGD fitness
(0: true, 1: false).

Consequently, the introduction strategy evaluation department revised its evaluation

item names by adding ‘solution supplier capacity’ and changing ‘technical integrity’ to
‘technical policy integrity’, ‘security policy’ to ‘security policy conformity’, and ‘process’
to ‘methodology’.
In the functional evaluation department, the names of the evaluation items were
revised by adding ‘robot management and operability’ and changing ‘automation’ to
‘automation process development ease’. In the management evaluation department, ‘man-
agement’ was changed to ‘management policy’ and ‘standardised asset management’ was
changed to ‘information asset management policy’. The criteria for selecting RPA solutions
that reflect the results of the validity evaluation and expert questionnaires are summarised
in Table 7. In this table, ‘N10 represents drafts developed via literature research and ‘N20
represents revised drafts. ‘N30 represents deleted drafts and ‘N40 represents movement
between evaluation items. Many of the items marked with ‘O’ in the validity evaluation
were also corrected.
Electronics 2023, 12, 986 13 of 22

Table 7. Updated selection criteria based on the first round of Delphi evaluation opinions.

I. Categories No. II. Evaluation Items III. Evaluation Criteria

Economic Expenses (solution, introduction and construction, licence, operations) [8,26] N2 ,
1 validity investment value (ROI; [24], EVA, TCO, EVS, TEI, BSC, etc.) N1
Reference customer case holder [8,26] solution provider capability [26], product
Capabilities of vision [24], dealer market awareness [8], existing results in the same field [8],
2 solution
suppliers N2 terms and conditions [8], product and service support [24], support capabilities,
education and customer service [8], partner ecosystem [24]
Purpose of application of RPA introduction N4 , hardware/software
requirements [8], technical elements (OS/hardware requirements and technical
capabilities required for RPA deployment and operation) [8], technical
1. Technology compatibility [26], performance [24], system interaction and integration [8], RPA
Customer 3 policy program application consistency [8], scalability N1 , relatedness to other
deployment conformity N2 technologies N1 , portfolio [24] N4 , revolutionary roadmap [24] N4 , risk
strategy N2 management strategy based on risk analysis evaluation [8,26] N4 , mission and
objectives of companies and agencies [8] N4 , leadership priority and strategy,
initiative [8] N4 , licensing structure [25] N4
Personal information protection (considering personal information protection
Security policies based on system/application lines, capabilities, functions and user
4 policy reviews, interactive system data types) [8], account/personal identification
conformity
management [8] N3 , establishment of code control strategies [8]
Repeated and regular process identification/standardisation [8] (modify
benchmark names to reflect primary comments), bot ideas [24] N3 , business
7 Methodology N2 performance processing procedures [24] N2 , characteristics and operations of the
company [26] N3 , calculations consistent with departmental and enterprise unit
objectives [8], process engineering and evaluation [25] N4
Bot platform model and security [24], availability [25], business performance N3 ,
quality analysis (provided with quality transition graphs) [2], management and
analysis [24], dashboard capabilities [25], robot management functions
Robot (scheduling, load balancing, monitoring) [2], business and operational analysis
8 management
functions [25], business performance details [2] N3 , performance management
and operability N2
[25], exception management [25], centralised deployment management,
scheduling [25], operability [26] N4 , maintainability [26] N4 , self-study capability
[25] N4 , derive architectural requirements N3 , tenancy [8] N4
Excel and SAP (ERP solution) API support [2], command library [25], security
enhancement site response [26], security character recognition [2], security [26],
2.
Development bot development [24], bot design and development [24], atypical GUI-based
and program automation (X-Internet, Active X, Flash) [2], performance ability under
operability N2 standardised GUI environment [26], usability [25], visual creation tools [25],
Ease of full/partial automation capabilities [25], website automation essential security
development enhancements (HOMETAX, GOV24, Court, e-car) [2], convenient and intuitive
10
for automated creation (direct programming, flowchart, etc.) [2], component sharing [25],
processes N2 test/debug control methods [25], character recognition ability to handle local
language specificities without exception [2,26] N2 , OCR N1 , development
convenience [26] N4 , RPA program service distribution/operation model
conformity [8] N4 , application functions [26] N4 , workflow [25] N4 , process
recording and reproduction [25] N4
Collaboration Classification, cognition, information extraction, optimisation, ANN, natural
and expansion language processing (NLP), decision tree, recommendation system, computer
16 of AI technology vision cognition, text mining, statistical technique, fuzzy logic, fuzzy matching
[4] N2 N4 , process mining N1 , scalability [25] N4

Compliance with legal systems such as personal information protection, account

12 Security [8] and personal identification management, data encryption/protection,
application security, risk/security evaluation, authentication, process traceability
3.
Technical On-premise/cloud, virtualisation support using VM/content technology N2 ,
architecture availability/disaster recovery capabilities, permissions, network capacity,
13 Architecture [8] performance management capabilities review, RPA program technical
policy/architecture consistency N4 , availability of duplex configuration N1 ,
collaboration structure with customer’s internal system N1
Electronics 2023, 12, 986 14 of 22

Table 7. Cont.

I. Categories No. II. Evaluation Items III. Evaluation Criteria

Change management, operational management, automation scheduling,
Operation policy
4. 14 automation interruption accident response, bot management, and operational
[8] N2
Operation data visualisation policy N1
management
Asset
system N2 management Code sharing method, technical policy update, RPA lifecycle management,
15
for information licence management, standardised operating models N1
assets [8] N2
N1 :
Added to reflect opinions from the primary expert survey. N2 : Modified to reflect opinions from the primary
expert questionnaire. N3 : Deleted to reflect opinions from the primary expert survey. N4 : Modified to reflect
opinions from the primary expert questionnaire.

To the best of my knowledge, this is the first expert questionnaire on RPA solution
selection criteria and the conditions of CV ≤ 0.5 and CVR ≥ 0.99 were applied in con-
sideration of the large number of relocations in evaluation items and criteria for actively
incorporating expert opinions. To improve the accuracy of our study, the results in Table 6
were not used in their initial form and final amendments were applied to a more stringent
standard. To this end, the results of Survey I.1 were adopted in the first Delphi survey.
Additionally, the four evaluation items II.5, II.6, II.9, and II.11 were deleted, and the II.16
evaluation item was added. The evaluation criteria III.5, III.6, III.9, and III.11 included in the
four deleted evaluation items were deleted or rearranged for form other evaluation items.
Second Verification Overview: For the development of the secondary verification
questionnaire, the evaluation target index was selected based on whether CV ≤ 0.5 and
CVR < 0.99 were satisfied by the verification result criteria of the primary questionnaire.
Additionally, all indicators, deleted evaluation items, and deleted evaluation criteria were
excluded. When completing the second questionnaire, the experts could easily correct the
results by providing mean, standard deviation, stability, validity, consensus, convergence,
and final judgement values. A total of 27 questionnaire items were presented. Considering
that the evaluation department, evaluation items, and evaluation criteria were based on
the opinions collected in the first round, the stability index was CV ≤ 0.75 and validity was
determined according to CVR < 0.59 (p = 0.05) (Table 8).
The results of the second Delphi questionnaire were derived from 11 appropriately
fitted items. A detailed examination of each indicator identified ‘I.2. Development and
operability’ and ‘I.3. Technical architecture’ as appropriate categories for the evaluation de-
partment. Among the evaluation items, ‘II.2. Solution supplier capabilities’, ‘II.3. Technical
policy consistency’, ‘II.8. Robot management and operability’, and ‘II.13. The architecture’
were identified as appropriate items. Among the evaluation criteria, III.1, III.3, III.8, III.10,
and III.13 were identified as conforming.
Next, the opinions of the secondary expert evaluations were reflected. The RPA so-
lution selection evaluation standard index was refined again by reflecting the contents of
the ‘Evaluation Standard Opinion Proposal’ presented in the second questionnaire. No
additional opinions were expressed by the evaluation department according to the table
breakdown. Regarding the evaluation items, it was suggested that development and
evaluation methods are necessary for the evaluation items considered by the introduction
strategy evaluation departments of customers. Furthermore, ‘methodology’ was revised to
‘discovery and appropriateness evaluation of automation work objects’, and ‘automation
process development and evaluation methodology’ was added to the evaluation criteria.
Additionally, for the development and operability evaluation department, ‘methodology’
was revised to ‘automation process development and convenience’ because ‘ease of au-
tomation process development’ did not include evaluation criteria. The evaluation items
for the management system evaluation department and information asset management
policy are ambiguous, so no differences appeared. However, ‘operation policy’ empha-
sises that RPA falls under information service operation management policy. Therefore,
‘individual information service operation system’ and ‘information asset management
Electronics 2023, 12, 986 15 of 22

policy’ were changed to ‘company-wide information asset management system’. Other

changes, including changes to the evaluation criteria, are presented in Table 9. The contents
of this table are described in the form of ‘Evaluation Item Number: Evaluation Criteria
Elements’. For example, ‘4: Consistency with customer security architecture’ indicates that
item ‘II.4 Security policy consistency assessment’ has added an evaluation criterion called
‘Consistency with customer security architecture’.

Table 8. Secondary conformity assessment results.

No. Mean SD CV CVR CSD CGD a b c d Selection

2 6.18 0.72 0.12 1.00 0.83 0.50 0 0 0 0 O
I. 3 5.55 0.99 0.18 0.64 0.83 0.50 0 0 0 0 O
4 5.55 1.08 0.19 0.64 0.70 0.75 0 0 1 1 X
1 5.91 1.08 0.18 0.82 0.67 1.00 0 0 1 1 X
2 5.82 1.11 0.19 0.82 0.83 0.50 0 0 0 0 O
3 5.27 0.75 0.14 0.64 0.80 0.50 0 0 0 0 O
4 5.09 0.90 0.18 0.27 0.60 1.00 0 1 1 1 X
7 4.64 1.30 0.28 0.09 0.60 1.00 0 1 1 1 X
8 5.55 0.89 0.16 0.82 0.80 0.50 0 0 0 0 O
II.
10 5.27 1.21 0.23 0.45 0.70 0.75 0 1 1 1 X
16 5.91 1.08 0.18 0.64 0.75 0.75 0 0 0 1 X
12 5.55 1.08 0.19 0.45 0.75 0.75 0 1 0 1 X
13 5.45 0.89 0.16 0.64 0.83 0.50 0 0 0 0 O
14 5.64 0.98 0.17 0.82 0.70 0.75 0 0 1 1 X
15 5.09 1.16 0.23 0.27 0.60 1.00 0 1 1 1 X
1 6.36 0.77 0.12 1.00 0.86 0.50 0 0 0 0 O
2 6.00 0.85 0.14 1.00 0.67 1.00 0 0 1 1 X
3 5.36 0.77 0.14 0.64 0.83 0.50 0 0 0 0 O
4 5.00 0.95 0.19 0.27 0.70 0.75 0 1 1 1 X
7 4.55 1.30 0.29 −0.27 0.75 0.50 0 1 0 0 X
8 5.36 0.64 0.12 0.82 0.80 0.50 0 0 0 0 O
III
10 5.36 0.77 0.14 0.82 0.80 0.50 0 0 0 0 O
16 5.45 1.23 0.23 0.45 0.75 0.75 0 1 0 1 X
12 5.73 1.14 0.20 0.64 0.67 1.00 0 0 1 1 X
13 5.36 0.88 0.16 0.64 0.80 0.50 0 0 0 0 O
14 5.82 1.03 0.18 0.82 0.67 1.00 0 0 1 1 X
15 4.91 1.44 0.29 0.09 0.60 1.00 0 1 1 1 X
a CV fitness (0: true, 1: false). b CVR fitness (0: true, 1: false). c CSD Fitness (0: true, 1: false). d CGD fitness
(0: true, 1: false).

Table 9. Updated selection criteria following the second Delphi evaluation survey.

Results of Changes in Evaluation Criteria by Evaluation Item

Type
(Example: Evaluation Item Number (No.) of II: Evaluation Criteria Elements of III)
4: Consistency with customer security architecture
7: Automated process development and evaluation methodology
13: Review cloud architecture implementation standards (customer versus supplier)
Added
14: Customer feedback management, script management, logging/upgrading/migration policies
15: Company-wide common module standardisation and product repository (output/result storage)
management methods
4: Possibility of integration with information access authorisation, issue management, and code control systems
Modified 8: Robot operation status aggregation function
12: Application security (authentication, authorisation, encryption, logging, security testing, etc.)
Electronics 2023, 12, 986 16 of 22

Table 9. Cont.

Results of Changes in Evaluation Criteria by Evaluation Item

Type
(Example: Evaluation Item Number (No.) of II: Evaluation Criteria Elements of III)
2: Solution provider capabilities, terms, and conditions
3: RPA introduction objectives, technical conformity, consistency of RPA program application, risk management
strategy through risk analysis assessment, corporate and organisational mission and objectives, leadership
Deleted priorities and strategies, initiative
13: RPA program technology policy/architecture conformity
15: Technological policy update
16: Classification, cognition, information extraction, optimisation
14: Operability
Moved 16: Character recognition ability regardless of language specialties, OCR, scalability, relatedness to
other technologies

Third Verification Overview: To develop a questionnaire for the third round of verifi-
cation, only 16 indicators that were not selected in the results of the second round of verifi-
cation were included. The final adoption criteria were indicators satisfying Equation (5),
and the results are presented in Table 10.

Table 10. Third conformity assessment results.

No. Mean SD CV CVR CSD CGD a b c d Selection

I. 4 5.73 0.62 0.11 1.00 0.83 0.50 0 0 0 0 O
1 6.00 0.74 0.12 1.00 0.83 0.50 0 0 0 0 O
4 5.27 0.75 0.14 0.82 0.80 0.50 0 0 0 0 O
7 4.91 1.00 0.20 −0.09 0.60 1.00 0 1 1 1 X
10 5.55 0.89 0.16 1.00 0.80 0.50 0 0 0 0 O
II.
16 5.91 0.67 0.11 1.00 0.92 0.25 0 0 0 0 O
12 5.55 0.78 0.14 1.00 0.83 0.50 0 0 0 0 O
14 5.55 0.78 0.14 0.82 0.83 0.50 0 0 0 0 O
15 5.27 0.86 0.16 0.45 0.80 0.50 0 1 0 0 X
2 6.09 0.67 0.11 1.00 0.92 0.25 0 0 0 0 O
4 5.36 0.88 0.16 0.64 0.80 0.50 0 0 0 0 O
7 4.45 0.99 0.22 −0.09 0.75 0.50 0 1 0 0 X
III. 16 5.55 0.78 0.14 0.82 0.83 0.50 0 0 0 0 O
12 5.64 0.77 0.14 1.00 0.83 0.50 0 0 0 0 O
14 5.45 0.78 0.14 0.82 0.80 0.50 0 0 0 0 O
15 4.55 0.78 0.17 −0.27 0.75 0.50 0 1 0 0 X
a CV fitness (0: true, 1: false). b CVR fitness (0: true, 1: false). c CSD Fitness (0: true, 1: false). d CGD fitness
(0: true, 1: false).

Of the 16 included indicators, 12 were confirmed to be valid and four were found to be
inappropriate. The CVR of the evaluation items in II. 7 and II. 15 was each −0.09 and 0.45.
The CVR of the evaluation criteria in III. 7 was −0.09 and that of the evaluation criteria
in III.15 was −0.27. Despite failing to satisfy the CVR requirement, evaluation item II. 15
satisfied the requirements for stability, agreement, and convergence. Paradoxically, most
experts disproved that the associated evaluation criteria were inappropriate. In the end,
evaluation items II.7 and II.15, and evaluation criteria III.7 and III.15 were eliminated.
Next, the opinions of the third expert survey were incorporated. First, the evaluation
criteria for III.7 and III.15 reflect the opinions of experts from the first and third surveys,
and the evaluation criteria for automation policy development and convenience evaluation
are contained in II.10. Other calculations [8], process engineering, and evaluation [25],
which met the initial evaluation criteria derived from our literature search in terms of
departmental unit objectives and enterprise unit objectives, were deleted. The evaluation
criteria of III.15 were also deleted. III.15 defines the evaluation criteria corresponding to
the company-wide operational management system to be observed during the operation
Electronics 2023, 12, 986 17 of 22

of ‘III.14 individual automated processes’. Therefore, the III.15 evaluation criteria were
modified into expressions suitable for individual automated operating systems such as
code sharing [8], RPA lifecycle management [8], licence management [8], common module
standardisation, and product repository management.
The evaluation criterion name In II.12 was revised to ‘supplementary management’
because it was suggested that it should be revised to ‘security management’ to enhance
the ‘consistency of security policies’ and ‘differentiation of the introduction strategy’. The
evaluation criterion of ‘character recognition ability to handle the specificity of native
languages without exception [2,26], OCR’ was revised to ‘OCR (printed), OCR (written)’.
The results of the other detailed opinions are presented in Table 11.

Table 11. Updated selection criteria based on the third Delphi evaluation survey.

Results of Changes in Evaluation Criteria by Evaluation Item

Type
(Example: Evaluation Item Number (No.) of II: Evaluation Criteria Elements of III)
16: AI/ML level of optimisation for cognitive automation
10: Customer’s existing business performance procedures (manual, automation), steps/tools required to
Added automate from RPA suppliers
14: Operational model, operational product, operational standard policy, operational rules,
logging/dashboard management
12: Security management
Modified 16: OCR (printed), OCR (written)
14: Script code shape and change management, operational data policy
10: Process recording and reproduction [25]
16: ANN [4], NLP [4], decision tree [4], recommendation system [4], computer vision cognition [4], text
Deleted
mining [4], statistical technique [4], fuzzy logic [4], fuzzy matching [4],
14: Operability [26]
14: Code sharing [8], RPA lifecycle management [8], licence management [8], common module standardisation
Moved
and product repository (output and result storage) management, upgrading/migration

4.4. Final Validated RPA Solution Evaluation Criteria

The RPA solution evaluation criteria were divided into four evaluation departments
and 10 evaluation items, each of which contains several evaluation criteria. The customer
introduction strategy evaluation department introduced ‘economic feasibility’, ‘solution
supplier capabilities’, ‘technical policy consistency,’ and ‘security policy consistency’ evalua-
tion items. In the development and operability evaluation department, ‘robot management
and operability’, ‘automation process development and convenience’, ‘AI technology con-
nection and extension, ‘security management’, and ‘architecture evaluation’ were defined.
The final RPA solution selection evaluation criteria for each evaluation item are presented
in Table 12. In this table, the evaluation item numbers for the evaluation categories defined
in this study and the flow of results from previous surveys are presented to help readers
understand the development of the evaluation criteria.
Through the survey process, some of the criteria derived from our literature research
were deleted or revised and various additional opinions were incorporated based on expert
knowledge and experience.
In the evaluation of customer deployment strategies, cost items are divided into
solution prices, deployment costs, licensing costs, operating costs, etc. In the evaluation of
technical policy consistency, automated process development and evaluation methodology
were added as standards for evaluating the conformity of the security architecture of a
customer in relation to security policy conformity.
In the development and operability evaluation department, the ‘Business Operations
Analysis Function [25]’ was modified to be more intuitive and concrete in the form of
‘Robot Operational Status Summary Function [25]’. The development and convenience
evaluation items for automated processes were revised to provide a better understanding
of existing indicators, including customer performance procedures (manual, automation)
Electronics 2023, 12, 986 18 of 22

and RPA suppliers. In the case of AI technology connection and extension evaluation, AI
technology and application were included.

Table 12. Final validated criteria for evaluating RPA solutions.

I. Category No. II. Evaluation Items III. Evaluation Criteria

Expense (solution, introduction and construction, licence, operation
1 Economic validity
expenses) [8,26], investment value (ROI, EVA, TCO, EVS, TEI, BSC, etc.) [24]
Companies with reference client case [8,26], product vision [24], awareness of
Capabilities of manufacturer’s market [8], existing performance in the same field [8], product and
2
solution suppliers service support capabilities, education and customer service [8], partner
ecosystem [24]
1. Hardware/software requirement [8], technological elements (technology and
Customer ability for fulfilling OS/hardware requirements and RPA deployment and
deployment strategy Technology
3 operations) [8], performance [24], system interaction and integration [8],
policy conformity
portfolio [24], innovation roadmap [24], automation process development and
evaluation methodology
Personal information protection (system/application line, capability and user
review, information access and issue management strategies, interactive data
Security
4 types) [8], account/personal identification management (service/network and
policy conformity
system/application-level access management) [8], consistency with customer
security architecture
Bot platform model and security [24], availability [25], quality analysis (quality
transition graph provided) [2], management and analysis [24], dashboard
Robot capability [25], robot management functions (scheduling, load balancing,
8 management and monitoring) [2], robot operation status aggregation function [25], performance
operability management [25], exception management [25], centralised deployment
management, scheduling [25], maintainability [26], self-learning capability [25],
Multi-tenancy [8]
Excel and SAP (ERP solution) API support [2], command library [25], security
enhancement site response [26], security character recognition [2], security [26],
2. bot development [24], bot design and development [24], atypical GUI-based
Development and program automation (X-Internet, Active X, Flash) [2], performance ability under
operability standardised GUI environment [26], usability [25], visual creation tools [25],
Automation
process full/partial automation capabilities [25], website automation essential security
10 enhancements (HOMETAX, GOV24, Court, e-car) [2], Convenient and intuitive
development and
convenience creation (direct programming, flowchart, etc.) [2], component sharing [25],
test/debug control methods [25], development convenience [26], RPA program
service distribution/operation model conformity [8], application functions [26],
workflow [25], process recording and reproduction [25], customer’s existing
business performance procedures (manual, automation), steps/tools required to
automate from RPA suppliers
Collaboration and AI/ML optimisation level, process mining and scalability for cognitive
16 expansion of
AI technology automation [25], OCR (printed), OCR (written), relatedness to other technologies

Compliance with legal systems such as personal information protection [8],

Security account and personal identification management [8], data
12 management encryption/protection [8], application security [8], risk/security evaluation [8],
authentication [8], process traceability [8]
3.
Technical On-premise/cloud [8], virtualisation support using VM/container technology,
Architecture availability/disaster recovery capabilities [8], permission [8], network capacity [8],
13 Architecture performance management capabilities [8], dual configuration availability,
collaboration structure with customer internal systems, cloud architecture
deployment standards (customer versus supplier)
4. Script code shape and change management, operational management, automation
Operation Automation scheduling [8], automation interruption accident response [8], bot management
14 process operation
and systems and operational data policy, supplier-customer technical support system,
management systems operational model, operational product policy, operational standard

In the technical architecture evaluation section, ‘virtualisation support using VM/content

technology’ was modified and indicators such as ‘client versus supplier’ were added.
Finally, the operations management system evaluation department added indicators
such as ‘script code shape and change management’, ‘bot management and operation data
policy’, ‘supplier technical support system’, ‘operation model’, ‘operation calculation’, and
‘operation standardisation’.
Electronics 2023, 12, 986 19 of 22

4.5. Implications
RPA is limited to specific businesses and often accompanied by robot operations.
Although RPA it is a software tool, it has limitations in that it cannot directly apply the
technical evaluation criteria used in software construction. Therefore, organisations that
wish to introduce RPA must establish appropriate criteria for selecting solutions. This
minimises the time and effort required to modify and standardise subsequent maintenance
and operational processes to match solutions by standardising and selecting appropriate
applications for RPA. Even if there is no IT specialisation, it can easily be incorporated
and so-called shadow IT introduction may increase. Therefore, RPA management should
also be considered at the enterprise architecture standardisation and integration level in an
enterprise-wide resource management system.
One expert participating in our Delphi surveys suggested that organisational con-
sideration should be given to the evaluation criteria of RPA solutions for enterprise and
agency missions and objectives [8], leadership priorities, and strategies. Another vendor
expert stated that ‘RPA’s information management and business data management are
often independent of customer companies, and the involvement of suppliers is limited, so
it is necessary to manage data standards’. This company-wide issue is one of the evaluation
items associated with the operating management system evaluation department defined in
this study and is the main reason why this evaluation department and its corresponding
evaluation items are maintained.
AI technology collaboration and scalability evaluation items were established in terms
of development and operability, which is consistent with the current trend of selecting
RPA solutions starting with the introduction of AI. In particular, ‘robot management and
operation’ and ‘automation process development and operation’, which are not typically
considered in the software field, are emerging as unique elements compared to other
evaluation criteria.
Even if RPA is introduced based on these solution selection evaluation criteria, further
efforts as a company are essential to recognise and utilise RPA in an organisation in the
early stages of RPA development. Because RPA aims to automate repetitive business
processes that have been standardised by companies, it is necessary to change the structure
of an organisation to one that can further enhance and add value to existing human
resources. Furthermore, the efficiency of operations achieved through RPA should be
linked to the performance evaluation of individuals and their organisations, and the
results should be shared as best practices at the company level to induce the spread of
operational efficiency. Vendors and designers of RPA solutions should strive not only
to promote companies that wish to introduce RPA, but also to form active partnerships
that can promote the development of solutions that suit partners. Additionally, RPA
lacks a consistent vocabulary. Therefore, a vendor-independent conceptualisation of RPA
relationships between vocabularies is required [39].

4.6. Potential Threats to the Validity of this Study

The evaluation criteria used for selecting RPA solutions should be carefully selected
according to their level of importance and the characteristics of the target industry group.
Recently, several studies on various applications of RPA [40–46] have been published. It
has been emphasized that the aspect of modelling improvement through implementations,
adaptations, changes, and tracking that meet the needs of the target business environment
should be prioritised. This indicates that the problem of the productivity paradox [47],
where productivity decreases with increasing IT investment, including RPA solutions,
may appear.
The current practice for developing RPA is to observe how routines are executed
and then implement the executable RPA scripts that are necessary to automate routines
using software robots based on evaluations by skilled human experts [48]. However,
process optimization through intelligent automation that can interpret the UI logs of
routine executions and support changes in automation routines for intermediate inputs is
Electronics 2023, 12, 986 20 of 22

still in the research stage. Regardless, some studies [49,50] have indicated the emergence
of new trends in IT that will pave the way for developing new methods of achieving
sustainability that are very noteworthy for RPA adoption and selection. This implies that
when introducing RPA, one should not overlook the fact that RPA is continuing to develop
and evolve. Therefore, it must be clarified whether the goal of introducing RPA is simply
automation, or process integration, intelligent automation, and autonomous intelligent
work that enables decision making to minimize potential risks and threats when investing
in and constructing IT, including RPA.

5. Conclusions
In recent years, RPA has been rapidly adopted by commercial organisations to auto-
mate repetitive business processes [19,20]. However, with various RPA solutions available
on the market, it is difficult for companies to select RPA solutions that suit their business
characteristics and processes. No formal evaluation criteria for RPA solution selection have
been developed to alleviate this issue.
In this study, I developed evaluation indicators that can be used to select an optimal
RPA solution for a specific enterprise. Based on a literature review, evaluation indicators
were subdivided into evaluation departments, evaluation items, and evaluation criteria,
and organised hierarchically. Eleven experts rated the validity of the derived evaluation
indicators through three Delphi surveys. As a result, ten evaluation items were assigned to
four evaluation departments and the evaluation criteria to be considered for each item were
presented in detail. The customer deployment strategy evaluation department focuses on
items of ‘economic feasibility’, ‘solution supplier capabilities’, ‘technical policy consistency’,
and ‘security policy consistency evaluation’. The development and operability evaluation
department considers ‘robot management and operability’, ‘automation process develop-
ment and convenience’, ‘AI technology collaboration’, and ‘extension evaluation items’.
The technical architecture evaluation department considers ‘security management and
architecture evaluation items’, as well as ‘operation management’. The system evaluation
department considers individual automated process operating system evaluation items.
This study is of great significance for the development of evaluation indicators for RPA
solution selection. Additionally, the evaluation criteria for each evaluation item presented
in the developed evaluation index can be used as a checklist when applied in practice. This
should allow organisations that are introducing RPA and those who lack an understanding
of RPA to select RPA solutions that are optimised for enterprise and business characteristics.
Finally, the presented evaluation standard can provide a theoretical reference for revising
technical evaluation laws and regulations related to national software projects such as
Korea’s software technology evaluation standard.
Regardless, because this study did not consider the selection of RPA solutions for
a specific company, the feasibility of the derived RPA solution evaluation criteria must
be verified through additional studies. As part of a follow-up study, I intend to conduct
further research on the weight calculation for each indicator so that the work characteristics
of each company are reflected at the most optimal level.

Funding: This research received no external funding.

Data Availability Statement: The data supporting the reported results are available upon request.
Conflicts of Interest: The author declares no conflict of interest.

References
1. Madakam, S.; Holmukhe, R.M.; Kumar, J.D.K. The future digital work force: Robotic process automation (RPA). JISTEM-J. Inf.
Syst. Technol. Manag. 2019, 16, 1–17. [CrossRef]
2. Kim, K.B. A study of convergence technology in robotic process automation for task automation. J. Converg. Inf. Technol. (JCIT)
2019, 9, 8–13. [CrossRef]
3. Dossier: The Choice of Leading Companies RPA, How to Choose It Well, and Use It Well. IBM Korea. Available online:
https://www.ibm.com/downloads/cas/RRX5GWY1 (accessed on 6 April 2020).
Electronics 2023, 12, 986 21 of 22

4. Ribeiro, J.; Lima, R.; Eckhardt, T.; Paiva, S. Robotic process automation and artificial intelligence in industry 4.0-A literature
review. Procedia Comput. Sci. 2021, 181, 51–58. [CrossRef]
5. van der Aalst, W.M.P.; Bichler, M.; Heinzl, A. Robotic process automation. Bus. Inf. Syst. Eng. 2018, 60, 269–272. [CrossRef]
6. Aguirre, S.; Rodriguez, A. Automation of a business process using robotic process automation (RPA): A case study. Commun.
Comput. Inf. Sci. 2017, 742, 65–71. [CrossRef]
7. Pramod, D. Robotic process automation for industry: Adoption status, benefits, challenges and research agenda. Benchmarking
Int. J. 2021, 29, 1562–1586. [CrossRef]
8. Kang, S.H.; Lee, H.S.; Ryu, H.S. The Catalysts for Digital Transformation, Low·No-Code and RPA, Issue Report IS-117, Software
Policy & Research Institute. Available online: www.spri.kr (accessed on 29 June 2021).
9. IEEE. Available online: http://standards.ieee.org/standard/2755-2017.html (accessed on 2 April 2021).
10. Hyun, Y.; Lee, D.; Chae, U.; Ko, J.; Lee, J. Improvement of business productivity by applying robotic process automation. Appl.
Sci. 2021, 11, 10656. [CrossRef]
11. Sarilo-Kankaanranta, H.; Frank, L. The Continued Innovation-Decision Process—A Case Study of Continued Adoption of Robotic
Process Automation. In Proceedings of the European, Mediterranean, and Middle Eastern Conference on Information Systems,
Virtual Event, 8–9 December 2021; Springer: Cham, Switzerland, 2022; pp. 737–755. [CrossRef]
12. Wewerka, J.; Reichert, M. Robotic Process Automation: A Systematic Literature Review and Assessment Framework [Technical
report]. arXiv 2020, arXiv:2012.11951.
13. Marciniak, P.; Stanisławski, R. Internal determinants in the field of RPA technology implementation on the example of selected
companies in the context of Industry 4.0 Assumptions. Information 2021, 12, 222. [CrossRef]
14. Hyen, Y.G.; Lee, J.Y. Trends analysis and future direction of business process automation, RPA (robotic process automation) in the
times of convergence. J. Digit. Converg. 2018, 16, 313–327. [CrossRef]
15. Asatiani, A.; Penttinen, E. Turning robotic process automation into commercial success—Case opuscapita. J. Inf. Technol. Teach.
Cases 2016, 6, 67–74. [CrossRef]
16. George, A.; Ali, M.; Papakostas, N. Utilising robotic process automation technologies for streamlining the additive manufacturing
design workflow. CIRP Ann. 2021, 70, 119–122. [CrossRef]
17. Lee, T.-L.; Chuang, M.-C. Foresight for public policy of solar energy industry in Taiwan: An application of Delphi method and Q
methodology. In Proceedings of the PICMET’12: Technology Management for Emerging Technologies, Vancouver, BC, Canada,
29 July–2 August 2012; IEEE Publications: New York, NY, USA, 2012.
18. Yoon, S.; Roh, J.; Lee, J. Innovation resistance, satisfaction and performance: Case of robotic process automation. J. Digit. Converg.
2021, 19, 129–138.
19. Gartner. Top 10 Trends in PaaS and Platform Innovation. Available online: https://discover.opscompass.com/en/top-10-trends-
in-paas-and-platform-innovation-2020. (accessed on 10 October 2021).
20. McKinsey. The State of AI in 2020. 2020. Available online: https://www.stateof.ai/. (accessed on 10 October 2021).
21. Schatsky, D.; Muraskin, C.; Iyengar, K. Robotic Process Automation: A Path to the Cognitive Enterprise; Deloitte N Y Consult: New
York, NY, USA, 2016.
22. Skulmoski, G.J.; Hartman, F.T.; Krahn, J. The Delphi Method for Graduate Research. J. Inf. Technol. Educ. Res. 2021, 6, 1. [CrossRef]
23. van Oostenrijk, A. Screen Scraping Web Services; Radboud University of Nijmegen, Department of Computer Science: Nijmegen,
The Netherlands, 2004.
24. Schaffrik, B. The Forrester Wave: Robotic Process Automation, Q1 2021. Herausgegeben von Forrester Research. Available online:
start.uipath.com/rs/995-XLT-886/images/161538_print_DE.PDF (accessed on 10 October 2021).
25. U.S. Fed.: RPA Community Practice RPA Program Playbook. 2020. Available online: https://www.fedscoop.com/rpa-cop-first-
playbook/ (accessed on 3 September 2021).
26. Etnews, J. RPA Introduction Guide Seminar. Available online: https://m.etnews.com/20190226000165?obj=Tzo4OiJzdGRDbGFzcyI6
Mjp7czo3OiJyZWZlcmVyIjtOO3M6NzoiZm9yd2FyZCI7czoxMzoid2ViIHRvIG1vYmlsZSI7fQ%3D%3D
(accessed on 10 February 2021).
27. Séguin, S.; Tremblay, H.; Benkalaï, I.; Perron-Chouinard, D.; Lebeuf, X. Minimizing the number of robots required for a robotic
process automation (RPA) problem. Procedia Comput. Sci. 2021, 192, 2689–2698. [CrossRef]
28. Agostinelli, S.; Marrella, A.; Mecella, M. Research challenges for intelligent robotic process automation. In Proceedings of the
International Conference on Business Process Management, Vienna, Austria, 1–6 September 2019; Springer: Cham, Switzerland,
2019; pp. 12–18.
29. Choi, D.; R’bigui, H.; Cho, C. Candidate digital tasks selection methodology for automation with robotic process automation.
Sustainability 2021, 13, 8980. [CrossRef]
30. Atencio, E.; Komarizadehasl, S.; Lozano-Galant, J.A.; Aguilera, M. Using RPA for performance monitoring of dynamic SHM
applications. Buildings 2022, 12, 1140. [CrossRef]
31. Kim, S.H. Development of satisfaction evaluation items for degree-linked high skills Meister courses using the Delphi method. J.
Inst. Internet Broadcast. Commun. 2020, 20, 163–173.
32. Mitchell, V.-W.; McGoldrick, P.J. The Role of Geodemographics in Segmenting and Targeting Consumer Markets. Eur. J. Mark.
1994, 28, 54–72. [CrossRef]
Electronics 2023, 12, 986 22 of 22

33. Na, Y.-S.; Kim, H.-B. Research articles: A study of developing educational training program for flight attendants using the Delphi
technique. J Tourism. Sci. Soc. Korea 2011, 35, 465–488.
34. Khorramshahgol, R.; Moustakis, V.S. Delphic hierarchy process (DHP): A methodology for priority setting derived from the
Delphi method and analytical hierarchy process. Eur. J. Oper. Res. 1988, 37, 347–354. [CrossRef]
35. Ayre, C.; Scally, A.J. Critical values for Lawshe’s content validity ratio: Revisiting the original methods of calculation. Meas. Eval.
Couns. Dev. 2014, 47, 79–86. [CrossRef]
36. Lawshe, C.H. A quantitative approach to content validity. Pers. Psychol. 1975, 28, 563–575. [CrossRef]
37. Delbecq, A.L.; Van de Ven, A.H.; Gustafson, D.H. Group Techniques for Program Planning: A Guide to Nominal Group and Delphi
Processes. Foresman, Scott Deloitte Analysis dupress.com; Deloitte University Press: Quebec, QC, Canada, 1975.
38. Murry, J.W., Jr.; Hammons, J.O. Delphi: A versatile methodology for conducting qualitative research. Rev. Higher Educ. 1995, 18,
423–436. [CrossRef]
39. Völker, M.; Weske, M. Conceptualizing bots in robotic process automation. In Proceedings of the International Conference on
Conceptual Modeling, Virtual, 18–21 October 2021; Springer: Cham, Switzerland, 2021; pp. 3–13.
40. Banta, V.C. Application of RPA Solutions near ERP Systems-in The business Environment Related to the Production Area. A Case
Study. Ann. Univ. Craiova Econ. Sci. Ser. 2020, 1, 17–24.
41. Banta, V.C. The Current Opportunities Offered by AI and RPA near to the ERP Solution-Proposed Economic Models and Processes,
Inside Production Area. A Case Study. Ann. Constantin Brancusi' Univ. Targu-Jiu. Econ. Ser. 2022, 1, 159–164.
42. Banta, V.C. The Impact of the Implementation of AI and RPA Type Solutions in the Area Related to Forecast and Sequencing in
the Production Area Using Sap. A Case Study. Ann. Univ. Craiova Econ. Sci. Ser. 2020, 2, 121–126.
43. Banta, V.C.; Turcan, C.D.; Babeanu, A. The Impact of the Audit Activity, Using AI, RPA and ML in the Activity of Creating the
Delivery List and the Production Plan in Case of a Production Range. A Case Study. Ann. Univ. Craiova Econ. Sci. Ser. 2022, 1,
98–104.
44. Hsiung, H.H.; Wang, J.L. Research on the Introduction of a Robotic Process Automation (RPA) System in Small Accounting Firms
in Taiwan. Economies 2022, 10, 200. [CrossRef]
45. E-Fatima, K.; Khandan, R.; Hosseinian-Far, A.; Sarwar, D.; Ahmed, H.F. Adoption and Influence of Robotic Process Automation
in Beef Supply Chains. Logistics 2022, 6, 48. [CrossRef]
46. Sobczak, A.; Ziora, L. The use of robotic process automation (RPA) as an element of smart city implementation: A case study of
electricity billing document management at Bydgoszcz city Hall. Energies 2021, 14, 5191. [CrossRef]
47. Jaiwani, M.; Gopalkrishnan, S. Adoption of RPA and AI to Enhance the Productivity of Employees and Overall Efficiency of
Indian Private Banks: An Inquiry. In Proceedings of the 2022 International Seminar on Application for Technology of Information
and Communication (iSemantic), Semarang, Indonesia, 17–18 September 2022; IEEE: New York, NY, USA, 2022; pp. 191–197.
[CrossRef]
48. Agostinelli, S.; Lupia, M.; Marrella, A.; Mecella, M. Reactive synthesis of software robots in RPA from user interface logs. Comput.
Ind. 2022, 142, 103721. [CrossRef]
49. Vijai, C.; Suriyalakshmi, S.M.; Elayaraja, M. The Future of Robotic Process Automation (RPA) in the Banking Sector for Better
Customer Experience. Shanlax Int. J. Commer. 2020, 8, 61–65. [CrossRef]
50. Vinoth, S. Artificial intelligence and transformation to the digital age in Indian banking industry—A case study. Artif. Intell. 2022,
13, 689–695.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.
Reproduced with permission of copyright owner. Further reproduction
prohibited without permission.