Conceptual challenges of researching Artificial Intelligence in

public administrations
Definitional challenges and varying dimensions on the meaning of Ai
Colin van Noordt
Ragnar Nurkse Department of Innovation and Governance Tallinn University of Technology
Tallinn, Estonia [email protected]
ABSTRACT 1 INTRODUCTION
Research has been advancing on the development and deployment Artificial Intelligence (AI) has grasped the attention of many people
of Artificial Intelligence (AI) in public administrations. However, across the world. Many governments have been writing specific AI-
there is limited consensus and agreement on what is considered strategies, conferences on AI are held frequently and new articles
Artificial Intelligence, as different understandings and approaches are being written in rapid succession [1]. All in all, there is a general
in research and practice exist. This paper explores and compares understanding that Artificial Intelligence is going to transform our
the varying ways AI has been described and understood in previ- societies and that governments should do everything to harness
ous Information Systems and eGovernment research. Following, a its full potential [2]. Time will tell whether these transformative
survey amongst Belgium civil servants is analysed to assess what predictions will be correct, as often the transformative potential of
they associate with the term Artificial Intelligence. The findings new technologies does not live up to its expectations [3]. In fact, it
show that many civil servants tend to associate AI with being able is not the first time we are discussing the potential and challenges
to conduct intelligent tasks, have certain capabilities or are specific of AI in the public sector as older research articles show us [4, 5].
applications they are familiar with. Specific algorithms or learning However, despite this immense hype and efforts to be the best
methods, often included in research papers, are not associated with in Artificial Intelligence, there is very limited clarity of what is
the term AI. These results show that researchers and policymak- exactly meant by the term, with different audiences using the term
ers may have opposite or even paradoxical views on what is or is in different ways [6]. This lack of a clear terminology for AI has
not AI, which could have significant consequences for researching led to a variety of critical remarks, stating that people only refer
the adoption of AI in government, as well as comparing different to software as Artificial Intelligence when there is funding to be
research findings. In this respect, the paper proposes to use an held -otherwise, it would just be said to be statistics [7]. This is not
integrative lens to study AI in government, by including different to say that there are no articles or reports present which describe
dimensions and understandings. definitions of AI, in fact, the opposite is true. There are incredibly
large volumes of publications on Artificial Intelligence, especially
CCS CONCEPTS in recent years [1]. Despite the great number of publications, the
• CONCEPTS; • Artificial Intelligence; • Public Sector Inno- term remains unclear and contested due to the various backgrounds
vation,; • Technology adoption; and interests using these terms with many reports referring to AI
as forms of ‘intelligent machines’, ‘machine learning’ or similar
KEYWORDS terms [6, 8–11].
Many of these new applications, systems or applications are
Artificial Intelligence, Public administration, Perceptions, Civil Ser-
considered AI as they can conduct intelligent tasks [9]. Existing
vants, Machine Learning
studies already highlight that there are many, varying forms of
ACM Reference Format: technologies and applications considered to be AI, which may not
Definitional challenges and varying dimensions on the meaning of Ai, be alike [12–14]. What is the most fundamental for research on AI in
Colin van Noordt, and Ragnar Nurkse Department of Innovation and Gov-
the government, however, is that civil servants themselves may use
ernance Tallinn University of Technology Tallinn, Estonia [email protected].
2022. Conceptual challenges of researching Artificial Intelligence in pub-
different terms and concepts to understand and describe Artificial
lic administrations. In DG.O 2022: The 23rd Annual International Confer- Intelligence. Researchers who are researching the next algorithm
ence on Digital Government Research (dg.o 2022), June 15–17, 2022, Vir- to extract patterns in large data sets will use the term AI in very
tual Event, Republic of Korea. ACM, New York, NY, USA, 8 pages. https: different ways (usually regarded as data science) than policymakers
//doi.org/10.1145/3543434.3543441 would refer to it [15, 16]. Now that there has been increased interest
in researching AI in government, one of the more fundamental
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed research gaps relates to the empirical examples and insights into
for profit or commercial advantage and that copies bear this notice and the full citation how AI can create public value [17, 18]. With such strong research
on the first page. Copyrights for components of this work owned by others than ACM needs for additional empirical works, ensuring consistency and
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a generalizations of what is and is not AI in government within the
fee. Request permissions from [email protected]. scope of the study is crucial in advancing knowledge. As Krafft
dg.o 2022, June 15–17, 2022, Virtual Event, Republic of Korea et al., 2019 already indicate, many governmental documents do
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9749-0/22/06. . . $15.00 include policy recommendations on AI, but do not define their
https://doi.org/10.1145/3543434.3543441

183
dg.o 2022, June 15–17, 2022, Virtual Event, Republic of Korea Colin van Noordt

understanding of the term, raising concerns about what exactly ‘as AI’. At the moment, machine learning is the most popular form
policymakers attempt to govern. Already, as a result, discussions on used as the learning method in AI, or considered as a subset thereof
the impacts and consequences remain to be general, more focused [24]. What makes machine learning distinct from ‘traditional AI’ is
on ethics or the development of fair AI and large efforts are required that machine learning algorithms learn by ‘themselves’ on (very)
for any form of systematic and comparative studies; especially large datasets. The field of machine learning, however, consists
cross-domains [19, 20]. Furthermore, it is likely that in the coming of many different approaches and specific analytical techniques.
years, more and more governments will use some form of AI to Often, a distinction is made between supervised machine learn-
improve their practices [21], further emphasizing the need to create ing, unsupervised machine learning and reinforcement learning.
a common baseline of definitional convergence to understand and In supervised learning, labelled training data is used in order to
explain what and how AI develops in the government, and whether predict new cases based on the existing information whereas in
this is desirable [22]. unsupervised learning insights from the data without a clear out-
To highlight how different civil servants define and understand put are derived [32]. In the reinforcement learning approach, an
AI, this research is built on two steps. Firstly, an exploration of the AI system learns how to do a task well by giving a “reward” based
different understandings and definitions of Artificial Intelligence on the output of the task [31]. Within these broad approaches, a
as discussed in existing literature are introduced and discussed, massive amount of different analytical techniques belong to this
illustrating their main differences and similarities, and, in doing so, realm. For instance, the following terms have been used by Sousa
providing a fresh and new perspective on the topic [23]. Secondly, et al., 2019 to measure the progress of AI in public sector research,
a survey has been conducted among Belgium civil servants, asking such as: case-based reasoning (CBR); cognitive mapping fuzzy logic
them what they understand with the term “Artificial Intelligence” (FL), machine learning (ML), artificial neural networks (ANN) genetic
in an open-response format. Using a deductive coding technique, an algorithms (GA), multi-agent systems (MAS) and natural language
overview with different understandings of AI is introduced to assist processing (NLP), amongst many others [33]. These methods are
further research on AI, as well as to highlight a challenge of how often included in the keywords of larger comparisons such as the
civil servants perceive AI could significantly influence research OECD [34] or described in the proposed AI regulation from the
on the adoption of this technology. As such, the article concludes European Commission as AI techniques or approaches.
with a discussion and concluding remarks on the potential research Commonly, the use of these AI learning techniques aims to
and policy implications of the different perceptions in policy and enable an AI system to gain an ability in order to conduct specific
research. tasks. These abilities, or capabilities as others also mention, are what
define for some whether we classify something as AI or not [35].
These capabilities or abilities gained from the learning techniques
2 CHALLENGES OF THE TERM “ARTIFICIAL described earlier allow the AI to conduct tasks considered to require
INTELLIGENCE” intelligence or for the AI to behave rationally [22], or to think and
Despite the historical works on AI, there is still no commonly ac- learn [36]. Consequently, it is not so much the question of how AI
cepted definition and many studies consequently do not provide a learns – but what it is possible to do following the learning– and
clear definition of AI. If a definition is given, already as many as 28 whether it does so successfully. These abilities include, but are not
different definitions were identified [24]. In that respect, research limited to, perceiving, reasoning, learning, interacting, problem-
could describe AI in many different ways. Some papers aim to high- solving, decision-making or being creative [24].
light the difference between the AI as a SuperIntelligence, General Related, but not entirely similar, are some publications which
Intelligence or Narrow Intelligence, to avoid potential confusion refer to AI as specific applications which are consequently used
on these different types. Discussions on AI as a Superintelligence by public administrations and considered AI [37]. Often, these
often refer to futuristic robots which are far more intelligent than are special IT, software or hardware which are used to improve
humans [25]. This type of AI belongs in the realm of science fiction government’s processes or decisions. It has been argued that ap-
– although the potential futuristic perspective of how AI could look proaching AI as a tangible technology such as is more preferred
like has been at the basis for some ethical discussions on how to for research and regulatory purposes as it is more concrete [38].
‘control’ AI even if such a moment comes to pass [26]. Like this Currently, applications mentioned as such include voice assistants,
type of Superintelligence, research could also discuss Artificial Gen- facial recognition software, recommendation software, chatbots,
eral Intelligence. Unlike the Superintelligent variant, this type of and robotic process automation among others [13, 30].
AI refers to robots (or other systems) which are just as intelligent Lastly, some tend to refer to Artificial Intelligence as the general
as humans. This type of AI is commonly capable of learning and academic field – not specific to any kind of applications, but rather
transferring its previous thought knowledge into new domains the full study domain as it has started since the 1950s [39]. Despite
[27, 28]. More relevant for current research on the adoption and the the status as an emerging technology, the study of AI already started
type of AI applications belongs to the Artificial Narrow Intelligence in the 1950s with various researchers such as Gregory Powell, Mike
or ‘weak’ AI understanding. This is often understood as machine Donavan and Alan Turing publishing works on intelligent machines
intelligence that is equal or (slightly) superior to human intelligence [25, 35]. Not much later in 1956, the term Artificial Intelligence was
– but only for a specific task, predictive, reactive and based on rules first brought up during the Dartmouth Summer Research Project
[29–31]. on Artificial Intelligence, often seen as the ‘starting moment’ of the
Within this field of current or Narrow AI, papers often describe study of AI.
AI as the learning methods or techniques used to make the AI systems

184
Conceptual challenges of researching Artificial Intelligence in public administrations dg.o 2022, June 15–17, 2022, Virtual Event, Republic of Korea

Table 1: Varying perspectives of AI, author’s own elaboration

Understanding of AI Explanation As seen in:

AI as Superintelligence A futuristic machine or computer which (far) surpasses human [13, 25, 26]
intelligence
AI as General Intelligence A futuristic machine or computer which displays equal human-like [31, 40]
intelligence in a variety of domains
AI as Narrow Intelligence Current artificial intelligence in which systems display human-like [41, 42]
intelligence in one specific function
AI methods/techniques Techniques and methods that allow the analysis of large volumes of [14, 27, 32, 43–45]
data to develop AI such as case-based reasoning, cognitive mapping,
fuzzy logic, machine learning, multi-agent systems, rule-based systems
amongst many others. These may fall under supervised, unsupervised
and reinforcement learning methods
AI as human-like cognitive Ability of machines to carry out tasks which require human capabilities, [22, 35, 46, 47]
capability by displaying human-like behaviour, to behave rationally, the ability to
solve hard problems
AI as applications A special form of IT systems, applications or software that are capable [12, 13, 30, 38]
of performing tasks that normally need human intelligence
AI as a science The general study and science behind the pursuit of making machines [5, 39, 48]
or computers intelligent

An overview of these different understandings, which at times lead to the same amount of detailed understanding and concep-
can overlap, can be found in Table 1 below. tualisation of AI as one may obtain in an interview or in a focus
As can be seen by this overview, the term of Artificial Intelligence group due to the limited opportunities to express their thoughts, a
is highly fluid and changing – even over time, depending on the survey is an excellent research methodology to many people in a
latest state of the development [49]. short amount of time, and to this extent, a great research method
to research general perceptions to AI [50]. Naturally, one of the key
requirements of a survey design is to obtain a representative sample
3 METHODOLOGY of respondents. To do so, various distribution channels were chosen
to target a diverse number of civil servants as possible, ranging
To assess how civil servants perceive Artificial Intelligence, this from local, regional, and federal levels of government, and also
research uses data from a survey conducted amongst Belgium pub- civil servants who may have limited formal education in Artificial
lic administrations in April and May 2021, as part of the Belgium Intelligence. Firstly, all the members of the AI4GOV network were
AI4GOV programme. The AI4Belgium community is an ecosystem requested to respond to the survey by email. Secondly, the survey
of researchers, policymakers, civil servants, and citizens interested has been distributed within the internal communication networks
in advancing AI within Belgium, and has a strong research inter- of the Belgium government, to also target the various public or-
est and examines the current use of AI in Belgium and is actively ganisations which may not be active members of the AI4Belgium
putting policy in place to boost the uptake of AI in society. As network. Lastly, the survey has been shared on social media and in
such, there is a strong need to identify what is considered AI by the newsletter of the Belgium Digital Government Administration
civil servants themselves, as they would play a key future role in (BOSA) and AI4Belgium.
examining the current level of uptake and impact of this technol- Since the Belgium public administrations have different working
ogy. The survey also included various questions to the respondents languages (mostly Dutch and French), the survey has been trans-
regarding the current level of use of AI in their organisation, and lated to Dutch and French, giving respondents the option to answer
which perceived drivers and barriers are influencing the use of in English, Dutch or French, depending on the respondent’s pref-
their organisation. However, one of the crucial elements of the erence, in order to obtain a higher response rate. The full survey
survey was to gain a comprehend of how civil servants, with vary- was proof tested and reiterated various times to make the questions
ing backgrounds, perceive the term “Artificial Intelligence”. This relatively easy and quick (within 15 minutes) to answer, common
question was stated: “What is according to you Artificial Intelligence characteristics of mail-based surveys [51].
(AI)? Please provide a short answer.”. Respondents were given the The question was answered by different 134 respondents, from
opportunity to provide a short, open answer as a response. the municipal, regional, and federal levels of the Belgium govern-
A web survey was chosen to gain answers on how civil servants ment – although most of the respondents were from the Belgium
from the survey, as the aim was to gain as many civil servants
participating, in a relatively short time. While a survey may not

185
dg.o 2022, June 15–17, 2022, Virtual Event, Republic of Korea Colin van Noordt

Federal government. Following a review of the answers, 18 ques- 4.2 Artificial Intelligence as a learning
tions were either blank or duplication from another answer, which technique:
were consequently removed. This left 116 answers on what the
A substantial part of the respondents responded to the question
respondents found to be “Artificial Intelligence”. As the answers
about what they consider to be Artificial Intelligence a variety
were provided in both Dutch and French, they were both translated
of answers linked algorithms, methods and techniques to learn
to English using DeepL machine translation. Following this trans-
systems and to possible mimic human intelligence. These answers
lation, the answers were deductively coded and analysed using
(29 out the 159 coded answers) referred to this theme. Often, this
MaxQDA software based on the categories identified in Table 1
included answers such as “A set of algorithms capable of solving
except for the Narrow Intelligence, as the techniques and applica-
problems by connecting various sources of information and drawing
tions categories overlapped strongly with this category since they
conclusions from varied or incomplete information” or “Machine and
described current techniques and applications considered AI and
deep learning tools that allow learning from an algorithmic point of
should be better-understood vis a vis General or Superintelligence,
view from input data, or iterative learning loops, to provide a result
rather than a category by itself. As each of the answers could be
as close as possible to the expert’s expectations.”.
coded one, or multiple times, the final number of codes was 159
These answers could be well interpreted as understanding AI as
out of the 134 answers, meaning that there were several answers
the algorithms or techniques dominant in the development of AI
which referred to one or more of the categories identified below.
systems. This, as such, thus focuses on the creation and design of
technological tools which can conduct intelligent tasks. Tradition-
ally, the field of AI has always been about the creation of intelligent
machines and finding the appropriate techniques, methods and
algorithms to do so [25]. This art and science of creating intelligent
machines are therefore referred to as AI [53], which is in line with
4 FINDINGS an answer which mentioned the “training of computers to perform
4.1 Artificial Intelligence as a future intelligent actions by themselves.”.
Superintelligence or General Intelligence In this perspective, the methods or tools during the creation are
leading in explaining AI such as Machine Learning, Deep Learning,
Despite that some academic literature strongly focuses on the fu-
Bayesian methods, Conventional Neural Networks and more, chang-
turistic general or even superintelligence as Artificial Intelligence,
ing over time [14]. In general, discussions within this perspective
only one of all the answers retrieved was coded as such. This re-
will focus mostly on the methods used to create the AI such as the
spondent criticized the use of the term Artificial Intelligence as
various algorithms or approaches used to create higher-performing
part of the survey, as according to him, it did not exist. Rather, the
models [43].
respondent preferred that the term “Super Robots” would be used
One consequence here for research on the adoption of AI is the
instead. Unfortunately, it is not entirely clear why the respondent
assumption that AI thus is used when an organization is utilizing
found the use of the term AI so problematic, nor is it completely
these (one or more) analytical techniques on their data. The under-
straightforward if “Super Robots” would be in the scope of the
standing AI as a learning technique is highly dynamic – as in the
futuristic AI or the current robotic applications that are already
past other techniques were considered AI. Especially AI research
being used. The answer “Super robots are capable of doing things
up to 1980s, Symbolic AI research was the leading method for cre-
much better than humans but none of them are capable of doing
ating intelligent applications (Stone et al., 2016) – but may not be
anything they were not designed to do” does, on the one hand, gives
considered as such anymore.
the reference to superintelligence, as they are better than humans,
Some of the respondents referred to these algorithms as ‘intelli-
but at the same time, seems to refer to existing applications due to
gent algorithms’, which branch a variety of different backgrounds
the incapability of doing actions they are not designed for.
together, including mathematics, logic, computer science, cognitive
However, in any case, this suggests that for most civil servants,
science and neuroscience. In some cases, specific algorithmic tech-
AI is not associated with futuristic technology. General and su-
niques (considered to be AI) were mentioned in the answers of the
perintelligent AI doesn’t exist now and is likely to not make its
respondents, such as machine learning, deep learning or neural net-
appearance for many decades (or even centuries), although some
works, but more advanced jargon or specific statistical techniques,
reports and authors do refer to AI as the futuristic possibility of
such as Bayesian methods, conventional neural networks amongst
superintelligence [26]. For the respondents, however, this is not
others, were not mentioned once. This is, however, in rather sharp
the case, and questions regarding the use of AI thus refer to more
contrast with some of the measurements currently in use to track
practical experiences of technologies they now consider AI, rather
“AI”, as highlighted earlier.
than this potential futuristic technology. At the same time, there
are discussions regarding the ethics of AI and the academic debate
on the AI alignment problem tend to discuss this form of AI. These 4.3 Artificial Intelligence as an ability:
discussions within the academic community may refer to totally Most of the respondents answered various types of abilities and
different iterations of AI with other challenges and governance capabilities that are linked to the term “Artificial Intelligence”. Fol-
difficulties [31] – and this may not fully align or even cause con- lowing the coding process, 59 of the 159 codes are considered to
fusion as civil servants have a different perception of what is AI describe some form of ability, the highest number of coded groups.
technology [20, 52]. In contrast with the previous set of answers, these respondents

186
Conceptual challenges of researching Artificial Intelligence in public administrations dg.o 2022, June 15–17, 2022, Virtual Event, Republic of Korea

do not refer to the algorithms or techniques used in the develop- should be included [38]. Indeed, one respondent even mentioned
ment of the systems, but rather what AI can – or should - do. A that AI is to “have non-intelligent machines do intelligent tasks”.
consensus on the different abilities of AI was not found amongst
the respondents, as many highlighted different actions that they
consider AI to do, although several were mentioned a couple of 4.4 Artificial Intelligence as an application:
times. These include, amongst others: Many of the respondents also described a variety of systems, pro-
grammes or applications that they considered to be Artificial Intelli-
• Performing tasks are previously done by humans or imitating gence. In this sense, 53 out of the 159 answers to the question were
human intelligence coded to be considered part of this theme of answers. The type of
• Learning new things, learning from experience, and improv- applications referred to be Artificial Intelligence varied greatly, but
ing itself are often referred to as either programmes or computer systems,
• Conducting tasks autonomously or without human interfer- that are capable of adapting or learning to their own needs. Respon-
ence dents would mention specific applications, such as “A prediction
• Capable of automating repetitive tasks tool created by humans to assist them in their prediction tasks based
• Find connections, recognise patterns (in data), derive trends, on input data” or “A system that answers your questions from a com-
create or predict puterized system such as a chat box” are examples of such answers
given by the respondents.
If computers, systems, or other forms of ICT are thus capable of Here, applications are considered differently from the learning
portraying one of the following abilities, the more likely it is that techniques or algorithms that were described before. Instead, it
they are considered Artificial Intelligence. In this perspective, the focuses on the technological artefacts which are referred to as
ability to conduct intelligent tasks determines whether we regard Artificial Intelligence [46]. These are the ‘final products’ such as
technology as Artificial Intelligence [54]. These unique capabilities applications, ICT-systems, interfaces or innovations in which AI
are often enabled by the AI learning methods combined with the capabilities are embedded and with whom users interact with [13].
relevant data [43]. Most AI now tend to be able to do the tasks These applications are services, products or information systems
such as digital media recognition, pattern recognition, detection which enable – autonomously or in combination with people – the
of speech and text, clustering and the detection of anomalies in AI to provide services, insights or make decisions. Hence, respon-
large datasets [55]. Many respondents of the survey also identified dents sometimes mentioned this integration of (machine learning)
more specific abilities other than the ones described above linked algorithms other times of software and hardware as AI, such as
with AI, such as “draw conclusions from data”, “capable of solving “technology based on algorithms to enable machines/computers to
problems”, “detecting certain actions by analysing data”, “enabling analyse data, calculate, decide” “(. . .) is software that you can use not
the exchange of data safely and efficiently”, (. . .) “report deviations only to perform repetitive tasks, but that can quickly learn new tasks
(large or small) to the product owner or support” or “the possibility of (. . .)” or “a function you can assign to a program, app, game, machine
having value-added tasks performed automatically”. etc. to create or predict systematic things”. To a certain extent, some
It is possible that thus, solely assessing the AI learning technique of the applications already identified by Wirtz et al., 2019 were
is thus insufficient to identify AI, but its abilities and what tasks also mentioned by some of the respondents. For instance, answers
they perform should be regarded as the leading factor. It may be mentioned a system that automatically recognises a system through
very well possible that some AI learning techniques are used – a recognition matrix, facial recognition, a system for preventing
but that alone is not sufficient to regard something as “AI”. In diseases, robots, and virtual assistants or chatbots.
particular, the answers showed a strong tendency that Artificial Commonly these applications combine other software and func-
Intelligence is either replacing or acting without the supervision tions in combination with the insights of the AI learning methods,
of humans. In some way, AI could thus be better understood as creating a blurry field of how much the system is actual “AI” [57].
‘automation’ – which could be done by technologies that are based Furthermore, there are so many different types of applications that
on techniques other than the learning techniques described before. could be considered “AI” that it broadens the scope of the num-
However, not all – and especially AI used in the public sector – is ber of applications perhaps too wide. Indeed, one of the answers
acting completely autonomous, but often acts as decision support given highlighted this issue: “Very broad: the word suggestions on
in combination with human expertise [56]. This may thus lead to my mobile phone, Google Home, a chatbot (. . .).”.
difficulties in researching the adoption of AI in the public sector Each of these applications functions very differently, and one
since certain applications are not see perceived as AI due to a lack may only wonder if agreement can be found whether each of these
of autonomous decision-making. specific applications is considered “AI” by all – and if so, whether the
Similarly, as several respondents were mentioning that AI is effects as discussed in the existing literature and policy discourse
capable of mimicking human intelligence or performing tasks re- will apply to each of them. Even with chatbots, while the application
quiring human intelligence, it may lead to challenges in defining and purpose may be the same, research has already highlighted
if this task truly requires “intelligence” to be completed. There is the variety of complexity and functionalities each of these chatbots
not a sole answer to what can be considered intelligence. In fact, may have, leaving one to wonder if all chatbots are truly AI or not.
there seems to be little consensus on whether machines will ever The same accounts for the use of robotic process automation, which
be able to be intelligent at all or whether an AI can be considered is considered to be AI by some [58] and not by others due to the
“rational” creates further difficulties regarding which exact abilities lack of ‘cognitive’ decision-making [27].

187
dg.o 2022, June 15–17, 2022, Virtual Event, Republic of Korea Colin van Noordt

4.5 Artificial Intelligence as a science: more crucial due to the apparent gap already in some surveys on
There were 4 answers out of the 159 coded referring to the overall ‘general’ AI usage and more specific applications, as can be seen
branch of the science of Artificial Intelligence, a small minority. in the latest European Commission survey on the uptake of AI in
One of these answers, for instance, mentioned that Artificial In- businesses [59]. A somewhat paradoxical finding of this survey
telligence is “A branch of computer science covering many different was that a significant number of businesses reported that they are
techniques that rely on data and learning to mimic human intelligence” using AI already (45%), but when asked about specific applications,
whereas another highlighted the “set of theories and techniques for such as sentiment analysis or Chatbots, the percentage was much
developing complex computer programs (. . .)” which derive from the significantly lower, sometimes ranging in less than 5%, making the
academic discipline. One of the respondents referred to the goal reader wonder if specific applications of AI are so rarely used, how
and progress-oriented research focus of the field of AI, as he re- come organisation report such a high uptake of AI?
sponded: “Working towards the most autonomous technology possible Furthermore, similar difficulties could be identified if one would
based on self-learning algorithms”, highlighting the alignment of research the adoption of machine learning or other statistical tech-
the goal-oriented field of study. niques within public organisations. It is, indeed, very possible that
However, in general, based on the responses, many civil servants organisations are using these statistical approaches within their
do not often refer to the overall academic discipline or the science organisation – logistic regressions, Bayesian methods or other quan-
of Artificial Intelligence when asked what they mean by AI. Simi- titative analysis tools are commonly used within private and pub-
larly, to the associations given with Superintelligence and General lic organisations. However, just merely using these statistical ap-
Intelligence, most of the associations that the respondents have proaches may not necessarily lead to the development of an AI
with AI are practical – related to applications, capabilities, and the application or active use of the output of such models. As such,
applications. One may find these the results or the output of the the uptake of AI learning methods may thus be higher than AI
science of AI – rather than the academic field itself. applications, as there may be various challenges from moving to
the development of models using AI learning methods to practically
4.6 Other: using applications built on those. This dilemma can already be seen
in various measurements based on tracking the ‘progress’ of AI.
Lastly, there were 13 answers coded in the residual category of
Most of these track the use of various machine learning terms or
“Other”, as these answers did not fit in any of the other categories
methods in scientific papers. This, however, says very little about
identified above. Some of these answers referred to what AI enables
what is the result of these models, how these models are conse-
for society, such as “A way to save time and jobs” or “A necessity”.
quently finalised and whether they are in active use in (public)
These residual answers show that despite the broad and diverse
organisations. It seems by some existing research that the uptake
meanings and understandings that already exist in the literature,
of the final applications seems much lower and more problematic
civil servants may still have a different association with the term.
indeed. For the research on the uptake of AI, one should also keep
Future research could dive more deeper into other, less usual asso-
in mind that many of the ‘successful’ pilots of AI usage in govern-
ciations that civil servants may have with AI.
ment discontinue or are not used anymore after a while. In such
a scenario, indeed, the learning methods of AI have been used in
5 IMPLICATIONS FOR RESEARCH AND a government context, but real adoption and integration of the
PRACTICE applications did not happen [56].
As the responses show, the term Artificial Intelligence indeed has The same accounts for the potential reverse definition difficulties
many different concepts and understandings among Belgium civil of these dimensions. If, for instance, a civil servant is asked about
servants, and, consequently, when asked whether these civil ser- their organisation’s use of AI applications, it may be possible that
vants are using AI could lead to a plethora of different answers and a response consisting of software, machines or other applications
examples. However, based on the answers given, there are some is given – but these may not necessarily be (fully) based on the
main dimensions of categories with which to understand how poli- learning methods considered AI by others. These applications may
cymakers may associate the term AI with. In this respect, this may still be able to do tasks successfully or considered novel enough to
either be AI as specific learning techniques, AI as an ability, AI as a appear like it is AI but may lead to disagreements to which extent
specific application or a futuristic form of AI, or a mixture between it can be seen as AI by all. An example of such disagreements are
them, as these dimensions support and even expand upon one an- recent articles highlighting that a large portion of “AI” start-ups
other. Each of these concepts also follows the different stages of the may sell software or applications they sell as AI, but in fact, are
adoption; from development & design to the application and imple- not based on AI learning methods, leading to sharp criticism as not
mentation of AI, with each phase having different requirements or being ‘truly AI’1 . Future research could focus more deeply on the
study perspectives as a result [32]. professional and/or academic backgrounds of the respondents, as
More fundamentally is that, when researching the use of AI by this could shed a light on how some identify Artificial Intelligence
civil servants, for instance, is that merely using general definitions more closely with the learning methods, applications and/or other
of AI may thus be insufficient to account for all the different ways identified categories of this paper.
this term is used. Thus, it could be more fruitful to be more spe-
cific in the research and aim to differentiate between the different 1 See e.g. “About 40% of Europe’s “AI companies” don’t use any AI at all, MIT Technology
types of AI – whichever is the aim of the researcher – as to assist Review: https://www.technologyreview.com/2019/03/05/65990/about-40-of-europes-
respondents in associating the requested ‘kind’ of AI. This is even ai-companies-dont-actually-use-any-ai-at-all/

188
Conceptual challenges of researching Artificial Intelligence in public administrations dg.o 2022, June 15–17, 2022, Virtual Event, Republic of Korea

These limitations could significantly affect nuanced discussions https://doi.org/10.1108/DPRG-08-2018-0049.

on the uptake, effects and governance of the different facets in- [9] OECD: Hello, World: Artificial Intelligence and its use in the Public Sector. (2019).
https://doi.org/10.1787/726fd39d-en.
volved with AI, as there are indeed many different challenges which [10] Craglia, M., Annoni, A., Benczur, P., Bertoldi, P., Delipetrev, P., De Prato, G.,
require more research to understand the consequences of AI on Feijoo, C., Fernandez-Macias, E., Gomez, E., Iglesias, M., Junklewitz, H., M, L.-C.,
Martens, B., Nascimento, S., Nativi, S., Polvora, A., Sanchez, I., Tolan, S., Tuomi,
society [60]. In this respect, much can for instance be learned from I., Fernandez Macias, E., Gomez, E., Iglesias, M., Junklewitz, H., López Cobo, M.,
previous unclear concepts used in eGovernment research, such as Martens, B., Nascimento, S., Nativi, S., Polvora, A., Sanchez, I., Tolan, S., Tuomi, I.,
Smart Cities. This has also led to various critical reflections on Vesnic Alujevic, L.: Artificial Intelligence - A European perspective. Publications
Office, Luxembourg (2018). https://doi.org/10.2760/11251.
the term, as a recent publication on smart cities highlights as well [11] Preece, A., Ashelford, R., Armstrong, H., Braines, D.: Hows and Whys of Artificial
[61]. Researchers, as well as policymakers, can have completely Intelligence for Public Sector Decisions: Explanation and Evaluation. (2018).
different understandings of what is considered a smart city, as well [12] Misuraca, G., van Noordt, C., Boukli, A.: The use of AI in public services. In:
Proceedings of the 13th International Conference on Theory and Practice of
as how it should function [62]. What may be a potential solution Electronic Governance. pp. 90–99. ACM, New York, NY, USA (2020). https://doi.
for this issue is to conduct research with a more integrated view org/10.1145/3428502.3428513.
[13] Wirtz, B.W., Weyerer, J.C., Geyer, C.: Artificial Intelligence and the Public Sector—
of AI, which aims to combine the different dimensions discussed Applications and Challenges. Int. J. Public Adm. 42, 596–615 (2019). https://doi.
in this paper. It may thus be extremely worthwhile in research to org/10.1080/01900692.2018.1498103.
describe a specific AI system with its ability, as well as which AI [14] Sousa, W.G. de, Melo, E.R.P. de, Bermejo, P.H.D.S., Farias, R.A.S., Gomes, A.O.:
How and where is artificial intelligence in the public sector going? A literature
learning methods were used. Such an integrated approach also fits review and research agenda. Gov. Inf. Q. 101392 (2019). https://doi.org/10.1016/j.
the definition as proposed by the High-Level Expert Group on AI giq.2019.07.004.
by the European Commission, as well as the OECD. [15] Zhang, B., Dafoe, A.: Artificial Intelligence: American Attitudes and Trends. ,
Oxford (2019).
With this paper, the research, as well as the policy community, [16] Carrasco, M., Mills, S., Whybrew, A., Jura, A.: The Citizens Perspective on the
may be better equipped with additional insights on how different Use of AI in Government. (2019).
[17] Medaglia, R., Gil-Garcia, J.R., Pardo, T.A.: Artificial Intelligence in Government:
groups perceive AI differently – and the need to ensure compara- Taking Stock and Moving Forward. Soc. Sci. Comput. Rev. 089443932110340
bility of the findings. This is not only crucial for improving the (2021). https://doi.org/10.1177/08944393211034087.
generalization of research findings beyond individual case studies, [18] Zuiderwijk, A., Chen, Y., Salem, F.: Implications of the use of artificial intelligence
in public governance: A systematic literature review and a research agenda. Gov.
surveys or other studies, but also in ensuring that any negative Inf. Q. 101577 (2021). https://doi.org/10.1016/j.giq.2021.101577.
consequences following the use of any AI system do not lead to [19] Nemitz, P.: Constitutional democracy and technology in the age of artificial
unnecessary restrictions on the many other times of technology intelligence. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 376, 20180089 (2018).
https://doi.org/10.1098/rsta.2018.0089.
referred to as AI, which do other tasks and are developed differently. [20] Cath, C., Wachter, S., Mittelstadt, B., Taddeo, M., Floridi, L., Wachter, S., Taddeo,
Future research may further explore why these different percep- M., Mittelstadt, B., Cath, C.: Artificial Intelligence and the ‘Good Society’: the
US, EU, and UK approach. Sci. Eng. Ethics. 24, 505–528 (2018). https://doi.org/10.
tions differ between civil servants, and to which extent professional 1007/s11948-017-9901-7.
backgrounds, familiarity with AI as well as how civil servants work [21] Valle-Cruz, D., Criado, J.I., Sandoval-Almazán, R., Ruvalcaba-Gomez, E.A.: As-
with AI in their job influence how they perceive what is to be con- sessing the public policy-cycle framework in the age of artificial intelligence:
From agenda-setting to policy evaluation. Gov. Inf. Q. 37, 101509 (2020). https:
sidered AI, and what is not. Furthermore, there is also room to //doi.org/10.1016/j.giq.2020.101509.
explore differences in understanding of other concepts related to AI [22] Sun, T.Q., Medaglia, R.: Mapping the challenges of Artificial Intelligence in the
such as bias, AI governance, manipulation or exploration of various public sector: Evidence from public healthcare. Gov. Inf. Q. 36, 368–383 (2019).
https://doi.org/10.1016/j.giq.2018.09.008.
unethical forms of AI uses in a follow-up study. [23] Torraco, R.J.: Writing Integrative Literature Reviews: Using the Past and Present
to Explore the Future. Hum. Resour. Dev. Rev. 15, 404–428 (2016). https://doi.org/
ACKNOWLEDGMENTS 10.1177/1534484316671606.
[24] Collins, C., Dennehy, D., Conboy, K., Mikalef, P.: Artificial intelligence in infor-
Great thanks to Anu Masso, Joep Crompvoets, Gianluca Misuraca mation systems research: A systematic literature review and research agenda. Int.
J. Inf. Manage. 60, 102383 (2021). https://doi.org/10.1016/j.ijinfomgt.2021.102383.
and Luca Tangi for their comments on an earlier draft of this paper. [25] Russel, S.J., Norvig, P.: Artificial Intelligence - A modern approach. Pearson
(2016).
REFERENCES [26] Müller, V.C., Bostrom, N.: Future progress in artificial intelligence. AI Matters. 1,
9–11 (2014). https://doi.org/10.1145/2639475.2639478.
[1] Perrault, R., Shoham, Y., Brynjolfsson, E., Clark, J., Etchemendy, J., Grosz Harvard,
[27] Harrison, T.M., Luna-Reyes, L.F.: Cultivating Trustworthy Artificial Intelligence
B., Lyons, T., Manyika, J., Carlos Niebles, J., Mishra, S.: Artificial Intelligence
in Digital Government. Soc. Sci. Comput. Rev. 089443932098012 (2020). https:
Index 2019 Annual Report. , Stanford (2019).
//doi.org/10.1177/0894439320980122.
[2] Alexopoulos, C., Lachana, Z., Androutsopoulou, A., Diamantopoulou, V., Charal-
[28] Wirtz, B.W., Weyerer, J.C., Geyer, C.: Artificial Intelligence and the Public
abidis, Y., Loutsaris, M.A.: How Machine Learning is Changing e-Government.
Sector—Applications and Challenges, (2018). https://doi.org/10.1080/01900692.
In: Proceedings of the 12th International Conference on Theory and Practice of
2018.1498103.
Electronic Governance - ICEGOV2019. pp. 354–363. ACM Press, New York, New
[29] Valle-Cruz, D., Alejandro Ruvalcaba-Gomez, E., Sandoval-Almazan, R., Ignacio
York, USA (2019). https://doi.org/10.1145/3326365.3326412.
Criado, J.: A Review of Artificial Intelligence in Government and its Potential
[3] Bannister, F., Connolly, R.: The future ain’t what it used to be: Forecasting the
from a Public Policy Perspective. In: Proceedings of the 20th Annual International
impact of ICT on the public sphere. Gov. Inf. Q. 37, 101410 (2020). https://doi.org/
Conference on Digital Government Research. pp. 91–99. ACM, New York, NY,
10.1016/j.giq.2019.101410.
USA (2019). https://doi.org/10.1145/3325112.3325242.
[4] Schank, R.C.: The Current State of AI: One Man’s Opinion. Artif. Intell. 4, 1, 1–8
(1983). https://doi.org/10.1609/AIMAG.V4I1.382. [30] Cabrera-Sánchez, J.-P., Villarejo-Ramos, Á.F., Liébana-Cabanillas, F., Shaikh,
[5] Barth, T.J., Arnold, E.: Artificial Intelligence and Administrative Discretion: Im- A.A.: Identifying relevant segments of AI applications adopters – Expand-
plications for Public Administration. Am. Rev. Public Adm. 29, 332–351 (1999). ing the UTAUT2’s variables. Telemat. Informatics. 58, 101529 (2021). https:
[6] Krafft, P.M., Young, M., Katell, M., Huang, K., Bugingo, G.: Defining AI in Policy //doi.org/10.1016/j.tele.2020.101529.
versus Practice. (2019). [31] Kaplan, A., Haenlein, M.: Siri, Siri, in my hand: Who’s the fairest in the land? On
[7] Davison, J.: No, Machine Learning is not just glorified statistics, the interpretations, illustrations, and implications of artificial intelligence. Bus.
https://towardsdatascience.com/no-machine-learning-is-not-just-glorified- Horiz. 62, 15–25 (2019). https://doi.org/10.1016/j.bushor.2018.08.004.
statistics-26d3952234e3, last accessed 2020/02/13. [32] Desouza, K.C., Dawson, G.S., Chenok, D.: Designing, developing, and deploying
[8] Vetrò, A., Santangelo, A., Beretta, E., De Martin, J.C.: AI: from rational agents artificial intelligence systems: Lessons from and for the public sector. Bus. Horiz.
to socially responsible agents. Digit. Policy, Regul. Gov. . 21, 291–304 (2019).

189
dg.o 2022, June 15–17, 2022, Virtual Event, Republic of Korea Colin van Noordt

63, 205–213 (2020). https://doi.org/10.1016/j.bushor.2019.11.004. [48] Russel, S.J., Norvig, P. 1956-, Russell, S.J. 1962-, Norvig, P. 1956-, Russel, S.J.,
[33] Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., Norvig, P. 1956-: Artificial Intelligence - A modern approach. Pearson (2016).
Wang, Y.: Artificial intelligence in healthcare: Past, present and future. Stroke [49] Grosz, B.J., Altman, R., Horvitz, E., Mackworth, A., Mitchell, T., Mulligan, D.,
Vasc. Neurol. 2, 230–243 (2017). https://doi.org/10.1136/svn-2017-000101. Shoham, Y., Brynjolfsson, E., Calo, R., Etzioni, O., Hager, G., Hirschberg, J.,
[34] Baruffaldi, S., Van Beuzekom, B., Dernis, H., Harhoff, D., Rao, N., Rosenfeld, D., Kalyanakrishnan, S., Leyton-Brown, K., Parkes, D., Press, W., Shah, J.: Stand-
Squicciarini, M.: Identifying and measuring developments in artificial intelligence: ing Committee of the One Hundred Year Study of Artificial Intelligence. (2016).
Making the impossible possible. OECD Sci. Technol. Ind. Work. Pap. 1–68 (2020). [50] Bryman, A.: Social research methods. Oxford university press (2016).
[35] Haenlein, M., Kaplan, A.: A brief history of artificial intelligence: On the past, [51] Thiel, V.: Research methods in public administration and public management.
present, and future of artificial intelligence. Calif. Manage. Rev. 61, 5–14 (2019). An introduction. Routledge (2014).
https://doi.org/10.1177/0008125619864925. [52] Green, B.P.: Ethical reflections on artificial intelligence. Sci. Fides. 6, 9–31 (2018).
[36] Russell, S.J., Norvig, P., Davis, E., Edwards, D.D., Forsyth, D., Hay, N.J., Malik, https://doi.org/10.12775/SetF.2018.015.
J.M., Mittal, V., Sahami, M., Thrun, S.: Artificial Intelligence A Modern Approach [53] Makridakis, S.: The forthcoming Artificial Intelligence (AI) revolution: Its impact
Third Edition. (2016). on society and firms, (2017). https://doi.org/10.1016/j.futures.2017.03.006.
[37] van Noordt, C., Misuraca, G.: Exploratory Insights on Artificial Intelligence for [54] Engin, Z., Treleaven, P.: Algorithmic Government: Automating Public Services
Government in Europe. Soc. Sci. Comput. Rev. 089443932098044 (2020). https: and Supporting Civil Servants in using Data Science Technologies. Comput. J.
//doi.org/10.1177/0894439320980449. 62, 448–460 (2019). https://doi.org/10.1093/comjnl/bxy082.
[38] Scherer, M.U.: Regulating artificial intelligence systems: Risks, challenges, com- [55] Ojo, A., Mellouli, S., Ahmadi Zeleti, F.: A Realist Perspective on AI-era Public
petencies, and strategies. Harv. J. Law Technol. 29, 353–400 (2016). Management*. In: 20th Annual International Conference on Digital Government
[39] Littman, M.L., Ajunwa, I., Berger, G., Boutilier, C.: Gathering Stregth, Gathering Research on - dg.o 2019. pp. 159–170. ACM Press, New York, New York, USA
Storm: The One Hundred Year Study on Artificial Intelligence (AI100) 2021 Study (2019). https://doi.org/10.1145/3325112.3325261.
Panel Report. Stanford Univ. Stanford, CA. (2021). [56] Misuraca, G., van Noordt, C.: AI Watch - Artificial Intelligence in public services.
[40] Bostrom, N., Yudkowsky, E.: The ethics of artificial intelligence. In: Frankish, K. , Luxembourg (2020). https://doi.org/10.2760/039619.
and Ramsey, W.M. (eds.) The Cambridge Handbook of Artificial Intelligence. pp. [57] Renda, A.: Artificial Intelligence Ethics, governance and policy challenges. Report
316–334. Cambridge University Press, Cambridge (2021). https://doi.org/10.1017/ of a CEPS Task Force, February 2019. aei.pitt.edu (2019). https://doi.org/10.1016/
CBO9781139046855.020. B0-12-227410-5/00027-2.
[41] Pennachin, C., Goertzel, B.: Contemporary Approaches to Artificial General [58] Eggers, W., Schatsky, D., Viechnicki, P., Eggers, D.W.: AI-augmented government:
Intelligence. In: Goertzel, B. and Pennachin, C. (eds.) Cognitive Technologies. pp. Using cognitive technologies to redesign public sector work. (2017).
1–30. Springer Berlin Heidelberg, Berlin, Heidelberg (2007). https://doi.org/10. [59] European Commission: European enterprise survey on the use of technologies
1007/978-3-540-68677-4_1. based on artificial intelligence. (2020). https://doi.org/10.2759/759368.
[42] Gasser, U., Almeida, V.A.F.: A Layered Model for AI Governance. IEEE Internet [60] Dwivedi, Y.K., Hughes, L., Ismagilova, E., Aarts, G., Coombs, C., Crick, T., Duan,
Comput. 21, 58–62 (2017). https://doi.org/10.1109/MIC.2017.4180835. Y., Dwivedi, R., Edwards, J., Eirug, A., Galanos, V., Ilavarasan, P.V., Janssen, M.,
[43] Raaijmakers, S.: Artificial Intelligence for Law Enforcement: Challenges and Jones, P., Kar, A.K., Kizgin, H., Kronemann, B., Lal, B., Lucini, B., Medaglia, R., Le
Opportunities. IEEE Secur. Priv. 17, 74–77 (2019). https://doi.org/10.1109/MSEC. Meunier-FitzHugh, K., Le Meunier-FitzHugh, L.C., Misra, S., Mogaji, E., Sharma,
2019.2925649. S.K., Singh, J.B., Raghavan, V., Raman, R., Rana, N.P., Samothrakis, S., Spencer,
[44] Kaplan, A., Haenlein, M.: Rulers of the world, unite! The challenges and oppor- J., Tamilmani, K., Tubadji, A., Walton, P., Williams, M.D.: Artificial Intelligence
tunities of artificial intelligence. Bus. Horiz. 63, 37–50 (2019). https://doi.org/10. (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and
1016/j.bushor.2019.09.003. agenda for research, practice and policy. Int. J. Inf. Manage. 101994 (2019). https:
[45] Sætra, H.S.: A shallow defence of a technocracy of artificial intelligence: Examin- //doi.org/10.1016/j.ijinfomgt.2019.08.002.
ing the political harms of algorithmic governance in the domain of government. [61] Meijer, A., Webster, W.: Governing Smart Cities: Why Do Academics Need to
Technol. Soc. 62, 101283 (2020). https://doi.org/10.1016/j.techsoc.2020.101283. Study Trendy Concepts? Inf. Polity. 24, 227–228 (2019). https://doi.org/10.3233/IP-
[46] Agarwal, P.K.: Public Administration Challenges in the World of AI and Bots. 190007.
Public Adm. Rev. 78, 917–921 (2018). https://doi.org/10.1111/puar.12979. [62] Soe, R.M., Schuch de Azambuja, L., Toiskallio, K., Nieminen, M., Batty, M.: Institu-
[47] Rai, A., Constantinides, P., Sarker, S.: Next generation digital platforms: toward tionalising smart city research and innovation: from fuzzy definitions to real-life
human-AI hybrids. MIS Q. 44, iii–ix (2019). experiments. Urban Res. Pract. 00, 1–43 (2021). https://doi.org/10.1080/17535069.
2021.1998592.

190