1

Technical Background and

Methodology for the Elsevier’s
Artificial Intelligence Report
Mark Siebert, Curt Kohler, Anthony Scerri, Georgios Tsatsaronis
Elsevier

In this paper, we are describing an end-to-end methodology

Abstract to identify research patterns of AI given a large-scale
collection of scientific articles (see chapter “Approach and
Abstract—In this paper we give the background of the Sources”).
technical work conducted in the framework of preparing First, we used expert knowledge and an in-house solution to
Elsevier’s AI report. It explains how we are using a corpus- extract and refine relevant concepts from selected sources,
based approach and keyword extraction and seeding to such as textbooks, expert panels, patents and public available
focus on articles that potentially represent the core of the sources as described in the chapter “A. Keyword extraction”.
field. We then utilize both supervised and unsupervised Second, we use these search terms to retrieve a collection of
machine learning approaches to filter and analyze this
scientific articles associated with AI from Scopus, an abstract
corpus with the aim to understand its trends, subfields, the
connections between them, as well as how they evolve over and citation database [2]. We trained a classifier to reduce the
time size of this corpus to only high relevant articles, which we
refer as AI corpus, as described in chapter “B – Optimization”
Once the scope was defined, we used an ensemble of
Introduction unsupervised clustering algorithms to learn latent topic
structures within the corpus, as described in “C – Clustering”.
The field of Artificial Intelligence (AI) is broad, dynamic
We believe that with this ’bottom-up’ strategy we can
and rapidly evolving. Despite its rather young history
with the term being coined in 1956 during a workshop represent the structure of the broad and dynamic AI domain.
in Dartmouth, it now encompasses a variety of sub Many alternatives exist along this process and the
“Discussion” suggests continuing research in this field.
fields, ranging from areas of generic content such as
perception, learning or reasoning; to more
application-specific content such as solving games, Approach and sources
diagnosing diseases or real-time translation [1].
Common topics in AI, in turn, are attracting wide Different approaches exist to delineate a research field. They
attention of researchers from different disciplines range from expert-selection of a publication set over expert-
such as mathematics, linguistics and psychology; law- selection of keywords up to unsupervised approaches, like
and policies makers as well as numerous industries. SciVal Topics, based on document citations. They vary in the
This nature and pace expound why the research influence of experts and depend on the availability and trust
landscape still lacks a united definition. of (seed) information.

Elsevier is a registered trademar k of Elsevier B.V. | RELX Group and the RE symbol are trademar ks of RELX Intellectual Properties SA, used under license. © 2018 Elsevier B.V.
 2

As no given set of publications as defined AI corpus

exists, we followed good practice from the past to
work with AI experts on a set of keywords, allowing to
extract a reference corpus of documents. It became
quickly apparent that other than in other fields experts
were not able to come to a joint view on the keywords
or to provide an agreeable initial set of (seed) of
publications. From desk research of other AI reports
we additionally observed the different scoping and
perspectives on AI as a technical, application or Keyword extraction Corpus optimization Clustering

Fig. 1. Scoping and structuring approach for the field of Artificial

business field and the difference over time of what is
Intelligence
labelled as AI.

Those aspects led us to explore the field bottom-up The following Sections specify how we executed them in more
and to identify keywords from content sources detail. Goal of the approach was to offer and explore an
representative to their perspective and use expert expert-guided, bottom-up approach for an emerging and
guidance to validate the selection and outcome. dynamic field, like AI, in contrast to top-down, expert-defined
Internal and external experts acknowledged the approaches of the past. Quantitative testing of the approach
approach to build a comprehensive base and the and results is suggested to be in focus for further research to
outcome as a good and broad scope of the field of AI, help evolve bottom-up approaches and learn how to apply AI
covering the potentially different schools and fields of to define AI.
Perspec- Sources Key word approach Key word
AI not just now, but also into AI history. Attempts to tives results
identify viable patterns in the keywords, using related Two AI textbooks/- Extract content structure, 500k textbook
publications failed as they again relied on individual structure (Norvig abstracts, full-text (if concepts.
structure [3]) (US, Elsevier copyright). Circa 2,500
expert opinion, e.g., assigning, which keyword is an CHN), Nilsson [4] Manual key word concepts from
algorithm and which an approach. Yet, the exploration (Tsinghua), incl. Index). extraction, cleaning and books &
shed deeper light on the keyword selection of the 50 AI course syllabi, de-duplication. courses.
Teaching

incl. Coursera/edX, Fingerprint full-text into

different perspectives, e.g. the Media focus on AI selected from leading key words/concepts, based
applications. Universities (Scival on general fingerprinting
ranking by “AI” ontology.
Due to the absence of an agreed AI ontology to help keyword). Fingerprinting, using full
prune or structure the keyword list, we worked with 10 Amazon books on Scopus publication body,
“AI” keyword. incl. conference papers and
the resulting raw set of 6 million extracted publications multiple cross-sector
from the ~800 keywords with the goal to optimize this thesauri.
instead of the keywords by reducing the number of Suggestions from Expert-validated key word Circa 50 key
researchers in list on AI, based on Scopus words –
false positives. This brings the additional opportunity
Research

government expert analytics, including curated by

to identify the AI share in keywords or other derived panels (funding conference papers and experts.
concepts, such as SciVal topics, although the program design) multiple cross-sector
thesauri.
optimization of the publication set comes with further Cooperative patent Extract CPC category Circa 100 key
expert influence. To optimize the publication set, classification (CPC [5]) structure and key words. concepts from
- patent codes G06N Extract all patent titles, patent
resulting from a Scopus extraction using the 800 (Computer Systems abstracts and full-text categories.
keywords as queries, experts from Elsevier Labs based on specific (circa 7,000 in the G06N Over 300,000
developed a supervised classifier, based on a gold computational category. concepts from
Industry

models) Extract concepts with patent full

training set of 1; 500 documents, manually annotated fingerprinting from the text/abstracts.
based on a mix of criteria. With this the publication set patent info.
Fingerprinting, using full
was reduced by 90%, satisfying expert validation of the Scopus publication body,
resulting document corpus of 600; 000 as the core incl. conference papers and
multiple cross-sector
publication set scoping the field of AI. thesauri
This was used as base for the AI trend analytics as well Annotated concepts Direct use of concept list 210 unique
as further structuring of the field to identify and and technologies; concepts
Media

www.aitopics.org
specify research fields within AI, using two (official publication of
unsupervised clustering approaches to help validate the AAI on AI news)
each other. Fig. 2. Data sources and results for keyword extraction of AI perspectives

Figure 1 summarizes the method and approach of the

The first step was to create a set of keywords, refer to as
three phases. search terms, associated as relevant to AI as possible. We use
several content sources and text mining solutions alongside
with selected domain experts to extract and refine these
search terms as described in III-A. Consequently, we used
these search terms to retrieve a raw collection of over 6

Elsevier is a registered trademar k of Elsevier B.V. | RELX Group and the RE symbol are trademar ks of RELX Intellectual Properties SA, used under license. © 2018 Elsevier B.V.
 3

million scientific articles from Scopus, our in-house between the perspectives. We experimented with multiple
abstract and citation database [2]. This set has been approaches, such as semantic consolidation and simplification
reduced to a final list of approx. 600; 000 articles of similar terms, keyword co-occurrences, or using publication
associated as relevant by means of a classifier weight and expert structures to apply better classifications.
described in III-B.
Optimizing the keyword list by further consolidating the terms
by e.g. truncating them (e.g. bio*, multi-agent*, etc.) led to
A. KEYWO RD EXTRACTION even more general terms and even higher rates of false
To extract bottom-up a set of keywords, referred to as positives. Additionally, the provenance of the data would
search terms, we use several content sources and text have been lost to analyse which perspective originally
mining solutions alongside with selected domain provided the input for the term. An in-depth quantitative
experts to extract and refine them. We selected with sensitivity analysis and testing was left for further research.
expert input content that represents each perspective
(per Figure 2) and were accessible to us. We examined Optimizing the keyword list via keyword co-occurrences
that content in various ways: text-mining bottom-up (using keyword combinations) made the corpus either too
concepts from full-text (e.g. patents), using expert- narrow or still full of false positives as it didn’t reduce the
curated lists of keywords (e.g. research), or referring keywords with broad or secondary meaning, such as
to existing ontologies (e.g., aitopics.org), to obtain a predictive model, visual analytics, or classification tree. Yet
long list of more than 800; 000 keywords, with mixed keyword co-occurrence provided an interesting way of
quality. exploring sub-fields in AI.
To clean and optimize the keyword list we used a Optimizing the keyword list by using corresponding
number of filters and iterations. publication size, publication development or weight and
- Exclude duplications, which were about 50 classifying the keyword frequencies according to initial expert
- Exclude general language words, like articles, input on AI types (e.g. algorithm, approaches, applications)
verbs or stop words and sub-structures (e.g., Learning, Decision making) felt again
- Exclude general terms too biased and did not bring conclusive results on possible
- Reduce term variations to a representative structures in the AI field. Yet they gave indications of the
minimum and test sensitivity in Scopus different focus of the perspectives, e.g., Media focussing on
- Conceptualize it using fingerprints, which applications. Given the challenges to agree on an optimized
reduces connected terms to concepts list of keywords we focused on optimizing the resulting broad
- Exclude clear non-AI or Computer Science terms, corpus of publications, using a supervised machine learning
e.g. stemming from examples/cases in the classifier.
documents
- Exclude specific, non-popular brands
For text-mining we relied on the Elsevier Fingerprint
Engine. This identifies concepts and their importance B. OPTIMIZATION
in any given text by using a wide range of thesauri and The publication corpus was obtained by searching for each
data-driven controlled vocabularies covering all keyword in the titles, abstracts, and keywords of documents
scientific disciplines, and by applying a variety of included in a Scopus May 2018 dataset, retrieving 5:7 million
Natural Language Processing (NLP) techniques. The unique documents. Expert review of the corpus reveals that
long list of keywords was reduced with this to circa this contained a high number of false positives, caused by
20,000 key phrases. application terms (e.g. computer games), historical AI terms
A first round of expert cleaning reduced this list further (e.g. finite elements), broad terms (e.g. ethical values), or
to 3,171 unique keywords, based on general relevance other field terms (e.g. neural networks in biology).
to Computer Science in a broad sense. In a second
round of expert validation we excluded further terms A corpus of 6 million documents was too big to manually
from the list by reducing general Computer Science curate with an expert panel. Therefore, we focused on a
terms and focusing on AI relevance. This resulted in a training set for a supervised classifier, making use of machine
unique keyword master list containing 797 terms, learning technologies itself to scope the field of AI. Although
which we offered, together with the down-selection further expert influence is introduced this way, external
process for external expert comments. experts appreciated the better transparency into the corpus
evolution and control, compared to a full unsupervised
Attempts to find an agreeable set of shared keywords, approach at this stage. Yet, unsupervised approaches were
representative enough to scope the breath of the field also suggested to further cluster and structure the field.
and being at the same time specific enough to AI were
not successful. To develop the training set for the classifier, we randomly
Neither one perspective stood out nor an overlapping selected 10,000 document identifiers from the 5:7 million
kernel was strong or representative enough to documents identified by the Scopus searches. We initially
delineate the field. Yet they brought interesting took 500 of these identifiers augmented with key textual
insights from analysing the overlap of keywords fields from Scopus (abstract, title, keywords, etc.) and fed

Elsevier is a registered trademar k of Elsevier B.V. | RELX Group and the RE symbol are trademar ks of RELX Intellectual Properties SA, used under license. © 2018 Elsevier B.V.
 4

them into a custom AWS Mechanical Turk [6] task allows to rank the AI-relevant and non-AI-relevant ones. To
where internal experts scored them as: Strongly AI, prune the keyword-list one would need to define a threshold
Moderately AI, or not AI. at which level AI-relevance to cut. Since all keywords yield
The distribution of scores from this initial set indicated some AI papers this approach already illustrates the challenge
we would not generate enough positive AI training of losing AI papers. For instance, cutting off by 50%, non-AI
data, so we created a very simplistic screening would exclude a lot of the over 25; 000 AI-related papers
classifier using the intuition being that documents that around Robotics or 14; 000 papers from expert systems.
hit on more terms (with core and moderate terms
more heavily weighted) were more likely to be AI Especially in the middle (20 - 50%) of the ranked list, we can
documents. We ran all the 5,7 million documents identify most of the AI application fields in the keywords, that
through that screening classifier and assigned them to are interesting to keep.
one of the three buckets. We then selected 1,000 With the aim to provide a rather comprehensive view of AI
random documents from the buckets as a second pool and AI publications appearing in many fields and keywords,
of potential training documents. From this second set we decided to not further cut the keyword list, but to work
of 1000 documents, we created two additional with the resulting corpus. Experts suggested that the rate of
random sets of 500 documents and scored them as false positives, as individual impressions from checking the
previously described. The sets were combined to corpus, is low enough to proceed with analytics. Yet, this test
create our 1, 500 document test set, using the provided the opportunity to now illustrate the AI share in
following criteria: other keyword clusters, like SciVal Topics and brought the
- Number, percentage, and weighted value (initial insight that societal relevant AI application fields range in an
expert score of strong AI) of query terms in a AI share of 20 - 50%. This might be an interesting starting
document point for further application-focused AI research.
- Weighed value of all query terms using the length
of the documents title and abstract Initial Keyword AI count Non-AI %
(used as count non-
- Number and percentage of ASJC codes assigned
analytics base) AI
to the document, AI ASJC and Computer Science
Back-propagation 8107 70 0.9%
ASJC codes assigned to the document Neural Network
- Computer Science Subject Area code Back-propagation 4029 36 0.9%
- Presence of an Abstract Algorithm
High- ranked

Cohen-Grossberg Neural 613 6 1.0%

Testing different classifiers, like Decision Trees and Networks
Nave Base to help the classifier differentiate between Genetics-based Machine 170 2 1.2%
Learning
AI and not-AI documents, a Random Forest model
Neural Networks 1390 19 1.3%
predicted the AI scope with circa 85% accuracy best. learning
The entire set of 5,7 million documents was run
through the model to generate predictions that were self-driving car 234 225 49.0%
used to reduce the number of documents identified as Automatic Translation 438 390 47.1%
AI Research to more than 600, 000 documents.
Mid-ranked

Soccer Robots 469 408 46.5%

To help experts validate the results we mirrored the Personal Assistant
appearance of resulting publications back to each Systems 12 10 45.5%
individual keyword. The keywords would on the one Autonomous Mobile
hand-side help to see the focus of the corpus (if e.g. Robot 2115 1438 40.5%
keywords that are less of core AI appear high in the
ranking of “AI-count”) and would provide a fourth nonlocality 19 3768 99.5%
Low- ranked

option to optimize the keyword list, if the corpus Bias Currents 31 6149 99.5%
seemed wrong, e.g., cutting out further keywords in Choice Experiment 26 5608 99.5%
the base corpus using a certain threshold of AI Boltzmann Equation 57 12460 99.5%
Biosensing 161 38938 99.6%
publications per keyword.
Fig. 3. High / Mid-/ Low-ranked keywords by AI-share, applying 600k
AI publication corpus to 797 keywords, covering 1998-2017
Figure 3 shows the absolute number of AI and Non-AI
publications from the 600,000-corpus associated with
the keyword, resulting in a Non-AI share (%). This

Elsevier is a registered trademar k of Elsevier B.V. | RELX Group and the RE symbol are trademar ks of RELX Intellectual Properties SA, used under license. © 2018 Elsevier B.V.
 5

Testing the corpus against single AI keyword results in SciVal, we see similar growth trends, but differences in the content
of the two samples, as illustrated by the two word-clouds in Figure 4(single keyword) and 5 (report corpus).

Fig. 4. Wordcloud, based on single keyword: Artificial Intelligence, Fig. 5. Wordcloud, based on full AI report corpus, SciVal, 2013-2017
SciVal, 2013-2017

While the single keyword sample seems to focus more on general terms. Like AI, algorithms, models, is the report corpus
highlighting specific approaches, like neural networks, classification or learning systems.

In any case, as Figure 6 illustrates, the report corpus covers significantly more subject fields. This underlines the initial
intention to delineate the field broad, yet specific enough. It might provide a richer source, especially for analysis of the
field structure, e.g. in terms of AI relevance in subject or application fields.

Fig. 6. Comparison coverage subject fields between single keyword vs. AI report corpus, SciVal, 2013-2017

then could filter based on core keywords, number of co-

C. CLUSTERING occurrences, or regions.
Learning more about the sub-structures of the AI field This resulting filtered dataset was then transformed into a
informs our report analytics and helps with deeper network structure and processed via a Louvain clustering
insights, e.g. which fields are the drivers of publication algorithm.
growth and how did the field evolve over time.
For this we refer to sub-structures within the now
delineated field of AI, described by the publication
corpus of 600; 000 documents. Further research might
investigate even broader approaches and explore fully
unsupervised approaches for the field, such as other
data sources, other seeds (selected publications) or
testing different clustering algorithms and labelling.
Within the given field, we investigated and applied two
clustering approaches: Louvain-clustering on keyword
co-occurrences,and Bi-secting Kmeans clustering on
the corpus itself. The first provides more supervision
through the focus on the keywords, especially in terms
of labelling. Other approaches, using stronger
hierarchical information could also be considered, and
we refer to future work for their application.

KEYWORD CO-OCCURRENCE CLUSTERING Fig. 7. Number of keyword co-occurrences based on at least 500 co-
We generated the matrix of co-occurrences between occurring documents, for the World, 2017; source: Scopus and Elsevier
the keywords based on the document ID lists per Fingerprint Engine
keyword search. We then generated a record for each The Louvain algorithm attempts to merge individual nodes in
co-occurrence of a pair of terms for each document. the network together to clusters to maximize the number of
This defined the global cooccurrence dataset that we

Elsevier is a registered trademar k of Elsevier B.V. | RELX Group and the RE symbol are trademar ks of RELX Intellectual Properties SA, used under license. © 2018 Elsevier B.V.
 6

connections between the members of the cluster. The are derived from Scopus abstract and citation database [2].
newly generated network of clusters then feeds into
the next
iteration of the Louvain calculations. This continues Cluster Examples
until no further optimizations of the clustering can be Machine Learning and Machine learning, supervised learning,
Probabilistic Nearest neighbor, Baysian networks
made. Although the clusters appear separately, the co-
Reasoning
occurrence points out the closeness of keywords and Neural networks Deep learning, Backpropagation,
indicate that none of the clusters stands alone in the Hopfield (and other) neural networks
field. Fuzzy systems Fuzzy systems, Close-loop control
Search and Swarm intelligence, evolutionary
Optimization algorithms, genetic algorithms
Testing different visualization formats, like matrices or Planning and Decision Distributed, adaptive systems,
graphs, a Chord-Graph structure appeared most making distributed artificial intelligence
appealing to test users. Especially interactive Natural Language Hidden Markov, Word Sense
functions, such as a time slicer or mouse-over received Processing and Disambiguation, Natural Language
Knowledge Generation
good resonance to facilitate the investigation of
representation
individual clusters or keywords, e.g. evolution of deep Face recognition Autonomous vehicles, Computer Vision,
learning. Image understanding, Robotics
Figure 7 illustrates the resulting graph after Fig. 8. Summary Keyword co-occurrence clusters, World, 1998-2017;
experimenting with a series of tests with different source: Scopus
publication thresholds and number of keywords (e.g., After tokenizing, removal of stop-words and stemming the
only core AI keywords from the classifier). We dataset contains a total of ca. 5.6 billion tokens (ca. 0.64
experimented with thresholds of 0, 100, 500 or 1; 000 million unique tokens). Our word embeddings are obtained
co-occurring documents. While 0 and 100 provide a using a spark implementation [11] of the word2vec skip-gram
rather full chart due to the multitude of model with hierarchical softmax as introduced by Mikolov et
cooccurrences, a threshold of 1; 000 became to sparse al [12], [13]. In this shallow neural network architecture, the
and by region not representative anymore. The main word representations are the weights learned during a simple
clustering stayed stable across the thresholds with prediction task. To be precise, given a word, the training
only smaller isolated clusters appearing/disappearing, objective is to maximize the mean log-likelihood of its context.
probably due to similarity of keywords creating anown We have optimized model parameters by means of a word
cluster. similarity task using external evaluation sets [14], [15], [10]
To illustrate the full field and first identify the clusters, and consequently used the best performing model.
we chose for all keyword despite the reduction in
readability as a starting point. We kept a color coding This experiment resulted in five similar clusters to the initial 7
for the provenance of the keywords to illustrate if clusters of the co-occurrence clustering. Using wordcloud
there were clusters representations of the clusters identified that 2 initial clusters
predominantly provided or suggested by a specific were “folded-in”, e.g., machine learning now appears as part
group of AI practitioners. For deeper investigation and of the other clusters. Given their strong co-occurrences this
further research, e.g. on regional differences, by appears realistic and might be first indications for hierarchical
perspective, the threshold and number of keywords relations, e.g. machine learning being an enabler for rather
can be adapted and help increase readability. The applied fields or other approaches, such as Computer Vision.
report provides regional examples and further
semantic analysis.
DISCUSSIO N
Along the scoping and definition process there are many
In summary, the discovered clusters in the field of AI
alternatives ranging from different input data sets, through
are shown in Figure 5.
selection of seed documents up to alternative algorithms that
all might influence the overall outcome. Key to this was to
WORDEMBEDDINGS+KMEANS CLUSTERING develop a bottom-up approach on larger scale data. It is an
Neural network derived word embeddings are dense invitation to further testing and research on the path to a
numerical representations of words that are more robust and commonly accepted AI ontology.
computed by learning local co-occurrences. Their Open to the whole process is the absence of an AI ontology
efficient training and ability to capture semantic and and the missing use of hierarchical information, apart from
syntactical relatedness in various natural language some influence in the conceptualizing of keywords through
processing (NLP) tasks including named entity our fingerprinting that makes use of other related (AI)
recognition [7], part-of-speech tagging [8], and taxonomies.
semantic role labelling [9], [10] of words have brought Other remaining challenges include clustering resolution,
them much popularity. For this experiment, we trained connectivity parameters, design of the graph, and labelling of
our model on title and abstracts of approximately 70 the clusters as well as insights into the machine clustering.
million scientific articles from 30 thousand distinct
sources such as journals and conferences. All articles

Elsevier is a registered trademar k of Elsevier B.V. | RELX Group and the RE symbol are trademar ks of RELX Intellectual Properties SA, used under license. © 2018 Elsevier B.V.
 7

REFERENCES
[1] Stuart Russell, Peter Norvig, and Artificial Intelligence. A modern approach. Artificial Intelligence. Prentice-Hall, Egnlewood Cliffs, 25(27):79–80, 1995.
[2] Scopus search home page: www.scopus.com
[3] Russell S., Norvig P. (2010), Artificial Intelligence: A Modern Approach; (Third edition) - http://aima.cs.berkeley.edu/
[4] Nilsson, N. (1997), Artificial Intelligence: A New Synthesis - https://www.sciencedirect.com/science/book/9781558604674
[5] https://worldwide.espacenet.com/classification?locale=en_EP#!/CPC=G06N (US, EU, CHN); on CHN using CPC: https://www.epo.org/news-
issues/news/2013/20130604.html
[6] AWS mechanical turk: https://docs.aws.amazon.com/mturk/index.html#lang/en_us : “Amazon Mechanical Turk is a web service that provides an on-
demand, scalable, human workforce to complete jobs that humans can do better than computers, such as recognizing objects in photographs.”
[7] Huy Do, Khoat Than, and Pierre Larmande. Evaluating named-entity recognition approaches in plant molecular biology. bioRxiv, page 360966, 2018.
[8] Cicero D Santos and Bianca Zadrozny. Learning character-level representations for part-of-speech tagging. In Proceedings of the 31st Internationa l
Conference on Machine Learning (ICML-14), pages 1818–1826, 2014.
[9] Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. Jointly predicting predicates and arguments in neural semantic role labeling. arXiv preprint
arXiv:1805.04787, 2018.
[10] Thang Luong, Richard Socher, and Christopher Manning. Better word representations with recursive neural networks for morphology. In Proceeding s
of the Seventeenth Conference on Computational Natural Language Learning, pages 104–113, 2013.
[11] Spark home page: https://databricks.com/spark/about
[12] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781,
2013.
[13] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality .
In Advances in neural information processing systems, pages 3111–3119, 2013.
[14] Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. Placing search in context: the concept
revisited. ACM Trans. Inf. Syst., 20:116–131, 2001.
[15] Elia Bruni, Nam-Khanh Tran, and Marco Baroni. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1–47, 2014.

Elsevier is a registered trademar k of Elsevier B.V. | RELX Group and the RE symbol are trademar ks of RELX Intellectual Properties SA, used under license. © 2018 Elsevier B.V.