An ensemble approach for research article classification: a case study in artificial intelligence

Research article classification in emerging fields

Bibliometric methods are frequently employed in the classification of research articles, particularly in emerging fields. These methods involve a comprehensive analysis of publication patterns and trends. For instance, in synthetic biology, a study outlined a bibliometric approach to define the field by analyzing a core set of articles and refining keywords identified from these sources, also including articles from dedicated journals (Shapira, Kwon & Youtie, 2017).

In nano-energy research, Guan & Liu (2014) used bibliometric and social network analysis to investigate the exponential growth of research output and compare scientific performances across countries. This study revealed a shift in the global share of nano-energy research, with emerging economies like China showing significant development momentum. The research also highlighted the evolving patterns of scientific collaboration in this field, marking a shift in influence from traditional scientific powerhouses to emerging economies.

The detection of emerging research topics (ERTs) using bibliometric indicators was the focus of another study (Xu et al., 2021). This research distinguished ERTs from common-related topics and developed a method to uncover high-impact ERTs with a fine level of granularity. The study not only identified ERTs but also proposed different research and development strategies for each topic, emphasizing the future economic and social impact of these ERTs and the reduction of uncertainty in research directions.

Search strategies of artificial intelligence

Previous efforts to classify articles related to AI have employed a variety of strategies. A simple yet commonly used method involves search strategies such as TS=(“artificial intelligence”), which is straightforward but may miss nuanced aspects of AI research. For instance, a study examining AI developments in China used this strategy to analyze the interaction between academic research and policy-making, providing insights into the evolving relationship between these domains (Gao, Huang & Zhang, 2019).

More complex approaches involve refining and complementing keywords with subject categories. Another study adopted a bibliometric definition for AI, starting with core keywords and specialized journal searches. This method was then enhanced by extracting high-frequency keywords from benchmark records, which was compared with other search strategies to profile AI’s growth and diffusion in scientific research. This approach allowed for a more detailed understanding of AI’s multidisciplinary development and the contributions from diverse disciplines (Liu, Shapira & Yue, 2021).

In addition, the World Intellectual Property Organization (WIPO) in 2019 applied a comprehensive search strategy that combined patent classification codes with an extended list of keywords. This strategy was based on a thorough literature review, established hierarchies, web resources, and manual checking. To identify AI-related publications, approximately 60 words or phrases specific to AI concepts were queried across all subject areas in the Scopus scientific publication database. Additionally, about 35 words or phrases related to AI were applied specifically to the Scopus subject areas of Mathematics, Computer Science, and Engineering (WIPO, 2019).

Deep learning models in research article classification

Deep learning models have also been utilized in identifying AI-related articles and patents. Dunham, Melot & Murdick (2020) described a strategy leveraging the arXiv corpus to define AI relevance, employing deep learning techniques like convolutional neural networks. This approach yielded high classification scores and precision in identifying AI-relevant subjects, demonstrating the effectiveness of supervised solutions in updating AI-related publication identification in line with research advancements. Miric, Jia & Huang (2023) demonstrated the use of machine learning tools, including deep learning, for classifying unstructured text data. The study focuses on identifying AI technologies in patents, comparing machine learning (ML) methods with traditional keyword-based approaches. The findings indicate the superiority of ML in terms of accuracy and efficiency, underscoring the advantages of these methods in capturing AI-related content in patents. Siebert et al. (2018) utilized both supervised and unsupervised machine learning approaches for analyzing a large corpus of AI-related scientific articles. This method involved keyword extraction and seeding, followed by machine learning-based optimization and clustering to identify and analyze trends within AI research.

Search strategy-based classification methods, while convenient, often have limitations in terms of recall, potentially missing relevant articles that do not fit neatly into predefined keyword categories. Traditional machine learning methods for classifying AI-related articles, such as using simple models like Random Forest or SciBERT, also face performance constraints. Their limitation lies in not fully utilizing certain important keywords or category names, which can lead to missing relevant content.

In contrast, our approach integrates keyword matching, decision tree based category classification, and the advanced capabilities of SciBERT. This combination allows for a more nuanced and comprehensive classification of AI-related articles, enhancing both accuracy and efficiency. By leveraging the strengths of each method, this integrated approach aims to provide a more robust and effective solution for classifying AI-related scholarly works.

Decision tree

Decision trees are a fundamental machine learning technique used for classification and regression tasks. They operate by breaking down a dataset into smaller subsets while simultaneously developing an associated decision tree incrementally. The final result is a tree with decision nodes and leaf nodes, where each internal node represents a “test” on an attribute, each branch represents the outcome of the test, and each leaf node represents a class label (Myles et al., 2004).

The primary formula for decision tree algorithms involves the concept of information gain, which is derived from entropy. Entropy, a measure of disorder or uncertainty, is calculated using the formula:

(1) Entropy(S)=−∑i=1npilog2piwhere pi is the proportion of the number of elements in class i to the number of elements in set S. Information gain, then, is defined as the difference in entropy before and after a dataset is split on an attribute. It is given by:

(2) InformationGain(S,A)=Entropy(S)−∑v∈Values(A)|Sv||S|Entropy(Sv)where A is the attribute and Sv is the subset of S for which attribute A has value v.

SciBERT

BERT (Bidirectional Encoder Representations from Transformers) works on the principle of the Transformer, an attention mechanism that learns contextual relations between words in a text (Devlin et al., 2018). Unlike traditional models that read text sequentially (left-to-right or right-to-left), BERT reads the entire sequence of words at once. BERT is pre-trained on a large corpus of text and then fine-tuned for specific tasks. The Transformer model uses an attention mechanism that weighs the influence of different words on each other:

(3) Attention(Q,K,V)=softmax(QKTdk)V

Here, Q, K, V are queries, keys, and values respectively, and dk is the dimension of the keys.

BERT uses multiple layers of the Transformer, and each layer outputs transformed representations of the input text.

SciBERT adapts the BERT model specifically for scientific texts. It is pre-trained on a large corpus of scientific literature, encompassing a wide range of domains (Beltagy, Lo & Cohan, 2019). This specialization allows SciBERT to better understand and process the language used in academic and technical documents. It has been proven effective in tasks like classification (Piao, 2021), entity recognition (Albared et al., 2019), and relationship extraction (Gangwar et al., 2021) in scientific texts. As is mentioned before, it has been utilized in the classification of AI-related scientific literatures and patents (Dunham, Melot & Murdick, 2020; Miric, Jia & Huang, 2023). SciBERT’s ability to grasp complex scientific concepts makes it particularly useful in classifying and analyzing scholarly articles, including those related to AI research.

Support vector machine

SVM is a supervised machine learning algorithm used for both classification and regression. SVM works well for both linear and non-linear problems. The basic idea of SVM is to find the best hyperplane that separates data points of different classes in the feature space. In SVM, data points are plotted in a space where each feature is a dimension. The algorithm then identifies the optimal hyperplane that maximizes the margin between different classes. The data points that are closest to the hyperplane and influence its position and orientation are known as support vectors (Jakkula, 2006). The mathematical formulation of SVM involves finding the hyperplane that solves the following optimization problem:

(4) minw,b1/2∥w∥2

Subject to: yi(w⋅xi+b)≥1,foralli

Here, w is the weight vector, b is the bias, xi are the training examples, and yi are the class labels.

For non-linear problems, SVM uses kernel functions to transform the input space into a higher-dimensional space where a linear separation is possible. Common kernels include polynomial, radial basis function (RBF), and sigmoid.

SVM is particularly popular in text classification due to its effectiveness in high-dimensional spaces and its ability to handle overfitting, especially in cases where the number of features exceeds the number of samples (Jakkula, 2006).

SciBERT

In our approach to classifying scientific articles, we leveraged SciBERT, a state-of-the-art pretrained language model specifically designed for scientific text. For each article in our manually labeled dataset, which served as the ground truth, we concatenated and lemmatized its title and abstract to form a single input text sequence. We then fine-tuned sciBERT using this concatenated text, with the following hyperparameters: number of epochs: three, training batch size: eight, evaluation batch size: eight, warmup steps: 500, weight decay: 0.01, learning rate: 5e-, optimizer: AdamW. A five-fold cross-validation was used in the fine-tuning process. In each fold, four sets were used for training and the remaining set for validation. The output of the SciBERT fine-tuning for our task is a probability score between 0 and 1 for each article, representing the likelihood that the article is AI-related. This is achieved by selecting the sigmoid activation function as the final layer of the sciBERT model.

Let T and A be the title and abstract of an article respectively. The probability score P indicating whether the article is AI-related is obtained by:

(5) P=σ(W⋅SciBERT(Lemmatize(T+A))+b)

(6) σ(x)=11+e−x

Here, σ is the sigmoid function, W is the weight matrix, and is the bias term.

The fine-tuned SciBERT model constitutes the foundation of our article classification system. Building upon this base, we incorporated additional components aimed at enhancing the model’s predictive accuracy. These components leveraged other article attributes and were integrated through a ensemble method.

Decision tree

In parallel to the fine-tuning of SciBERT, we sought to exploit the categorical metadata associated with scientific articles. The Web of Science (WoS) categories assigned to each article offer a high-level view of the content domain. To make use of this categorical information, we employed a decision tree classifier.

This classifier was trained to predict the likelihood of an article being related to artificial intelligence based on its WoS categories. Decision trees are particularly advantageous for their interpretability and ease of use. To validate the effectiveness of this approach, the 4,000 labeled articles from our dataset were subjected to a five-fold cross-validation scheme, akin to the method applied during the SciBERT fine-tuning process. A maximum depth of two was used to limit the complexity of the model. Other hyperparameters were kept at their default values.

At each node n of the tree, the algorithm selects the best feature f and a threshold θ that maximizes information gain. This is represented as:

(7) Split(n)=argmaxf,θInformationGain(D,f,θ)where Information Gain is calculated as the difference in entropy (measure of uncertainty or randomness) before and after the split. Once the tree is built, the probability of an article being AI-related is estimated based on the proportion of AI-related articles in the leaf node where the article falls.

Keyword matching

To further refine our classification model, we conducted an in-depth analysis of the high-frequency keywords within the initial corpus. This involved a manual review of the high-frequency keywords in the initial corpus to identify AI-related ones. Our goal was to establish a list of AI-related keywords that could serve as strong indicators for classifying articles as relevant to the field of AI.

For each keyword identified in this process, we randomly selected 20 articles containing the keyword from the initial corpus. This sample was scrutinized not only by the authors but also by external AI experts to ensure that the presence of the keyword was indeed indicative of the article’s relevance to artificial intelligence. If all of the 20 articles were considered AI-related, this keyword would be added to our keyword list. We were able to curate a list of keywords with a high degree of confidence in their relevance to artificial intelligence research. The keywords were summarized into the regular expression in Table 2. We then performed regular expression matching on title, abstract and keywords of articles to determine whether it contains any AI-related terms, and the matching results were included into our ensemble model.

SVM

Building upon the individual strengths of SciBERT, the decision tree, and keyword matching, we integrate these components using a SVM. The SVM acts as a meta-classifier, taking as input the probability outputs from SciBERT and decision tree, and the binary results from the regular expression keyword matching. The SVM is trained to find an optimal boundary that separates AI-related articles from others. The SVM used a linear kernel, and the regularization parameter C was set to its default value of 1.0. For each article, the input features include the probability score from SciBERT, indicating the probability of the article being AI-related; the output of the decision tree, which predicts the probability based on the article’s categorical data; and a binary feature signifying the presence or absence of AI-related terms as determined by our regular expression analysis. Let Tx, Ax, Kx, Cx and represent the title, abstract, keywords and WoS categories of article x respectively, the SVM function incorporating these specific inputs would be represented as:

(8) y=SVM(PsciBERT(Tx+Ax),PDT(Cx),Iregex(Tx+Ax,K))

And Iregex is the binary output from the regular expression matching.

Performance evaluation

At the beginning of this section, we examine the overlap between the different approaches used to identify AI-related articles. As is mentioned above, approaches in retrieving AI-related articles range from simple search term “Artificial Intelligence”, WoS category “Computer Science, Artificial Intelligence”, complicated search terms, to employing deep learning models. Three strategies were selected for further comparison: keyword based search strategy by Liu, Shapira & Yue (2021) (referred to as Liu’s approach), WoS Category=“Artificial Intelligence” Approach (referred to as WoS Category approach), and our Ensemble model (referred to as the EnsembleAI model). We selected Liu’s search strategy because it achieves the highest performance among existing search strategies (Liu, Shapira & Yue, 2021). The simplest approach of using the query TS=(“artificial intelligence”) to search the WoS database yielded a very low recall, therefore it is omitted in the following analysis. Figure 3 illustrates the coverage and intersection of the three approaches.

The WoS category approach, which contains 192,067 articles, is included in both Liu’s approach and EnsembleAI. Liu’s approach is almost included in EnsembleAI, with the exception of 24,606 articles. Our EnsembleAI approach contains the biggest number of articles among the three approaches, with 402,962 unique articles. The significant overlap between the approaches indicate that we share a common understanding of the AI research landscape. However, the disparities in the number of articles identified by Liu’s approach and EnsembleAI suggest that there may be unique aspects captured by our methods.

Next, we evaluated the performance of various methods to identify AI-related articles. Previously manually annotated dataset of 4,000 articles were directly used to evaluate the performance of WoS category and Liu’s approach. For EnsembleAI, 5-fold cross validation was employed during training and performance evaluation was done by applying the corresponding test set in each fold and computing the average performance.

The performance of each method is evaluated in terms of precision and recall. Precision measures the fraction of true AI-related articles among those identified by the method, while recall measures the fraction of true AI-related articles identified by the method out of all the AI-related articles in the sample.

The following table summarizes the results:

Table 3 presents the performance metrics of three distinct approaches for classifying articles as related to artificial intelligence. Precision measures the proportion of true positive results among all positive cases identified by the classifier, while recall (or sensitivity) measures the proportion of true positive results among all actual positive cases. The F1-score is the harmonic mean of precision and recall, providing a single metric that balances both.

The WoS category-based approach shows precision of 0.934, since not all texts in this category were classified as AI related by our experts. For example, articles dealing with human-machine interfaces, chemometrics and bilevel optimization were deemed non-AI by our experts, and we excluded them from the AI-related articles. Its recall is quite low (0.325), suggesting it fails to identify a large number of AI-related articles. Consequently, its F1-score is relatively low (0.482), reflecting the imbalance between precision and recall.

Liu’s approach significantly improves upon the WoS category, particularly in recall (0.669), while maintaining excellent precision (0.980). This balance results in a much higher F1-score (0.795), suggesting a more effective model overall.

The EnsembleAI shows a slight decrease in precision (0.915) compared to Liu’s approach, but has a superior recall (0.973). This high recall indicates that EnsembleAI is particularly adept at identifying most AI-related articles. Despite the slight trade-off in precision, EnsembleAI achieves the highest F1-score (0.943) among the three methods, indicating a robust performance and a well-balanced approach between precision and recall.

To account for AI-related articles not included in our initial search strategy, we randomly downloaded 2,500 articles from the WoS and subjected them to our ensembleAI model. The random sampling is implemented by randomly selecting a day from last year, downloading all articles published on that day, and then randomly select 2,500 articles from the result. An article is considered AI-related if it is classified as relevant by the ensembleAI model. Out of the 110 relevant articles in the 2,500 random samples, 105 of them are included in the final corpus. The results indicates that our search strategy successfully captures 95% of all the AI-related articles in the entire WoS corpus.

Ablation study

The ablation study will methodically examine the EnsembleAI model by selectively removing or isolating specific components, such as the decision tree, sciBERT, keyword matching, and the SVM integrator. This process aims to reveal the impact and value of each component within the ensemble framework. Through this study, we intend to quantify the contribution of each part of the EnsembleAI model to its precision, recall, and F1-score.

Overall, the ablation study highlights the synergistic effect of the combined components in the EnsembleAI model. Each component contributes uniquely to the model’s performance, with the decision tree and abstract elements being particularly crucial for maintaining high recall, and the SVM integrator more significantly affecting precision. The balanced performance of the original EnsembleAI model underscores the importance of integrating multiple approaches for effective identification of AI-related articles.

The data from the ablation study of the EnsembleAI model provides insightful revelations about the contributions of its individual components to the overall performance in identifying AI-related articles. Table 4 shows the performance data, followed by an analysis of each variant:

EnsembleAI (Original Model): With a precision of 0.915, recall of 0.973, and an F1-score of 0.943, the original EnsembleAI model exhibits a well-balanced performance.

EnsembleAI-Decision Tree: Removing the decision tree component slightly increases precision to 0.921 but notably reduces recall to 0.914. The F1-score drops to 0.918.

EnsembleAI-Keywords: The removal of keyword matching maintains the same precision (0.915) but slightly decreases recall to 0.959. The F1-score also sees a slight decrease to 0.937.

EnsembleAI-SVM: Excluding the SVM integrator decreases precision to 0.902 and slightly increases recall to 0.978. The F1-score is relatively stable at 0.939. Removing SVM is done by retaining the results from sciBERT and decision tree with a probability of 0.5 and more, and computing the union of positive results from sciBERT, decision tree and keywords matching. This method outperformed majority voting and soft voting, therefore it was selected as the way to replace the SVM integrator.

EnsembleAI-title: Removing the title from the SciBERT input lowers precision slightly to 0.905 and recall to 0.970, with an F1-score of 0.936.

EnsembleAI-abstract: Excluding the abstract from SciBERT input results in the highest precision (0.930) but lowers recall to 0.936, leading to an F1-score of 0.933.

Furthermore, we explored the effectiveness of using only the SciBERT component, as well as the impact of replacing SciBERT with other BERT variants when classifying AI-related articles.

BERT-base, developed by Google, was one of the first models to use Transformer architecture, significantly changing the landscape of text-based analysis by providing a deep, bidirectional understanding of context within language. It is pre-trained on a vast corpus of text and designed to be fine-tuned for a variety of Natural language processing (NLP) tasks (Devlin et al., 2018).

RoBERTa, which stands for Robustly Optimized BERT Pretraining Approach, is a model developed by Facebook AI that builds upon BERT’s architecture. It has been optimized with more data, larger batch sizes, and longer training times, leading to improved performance over BERT-base in many benchmarks (Liu et al., 2019).

BERT-base and RoBERTa were selected as alternative models and we conducted a series of additional experiments of feeding different combinations of textual data into these models, including title + abstract, and concatenated fields of title + abstract + WoS categories + keywords. The results are shown in Table 5.

EnsembleAI: This is the original EnsembleAI model with a precision of 0.915, recall of 0.973, and F1-score of 0.943. It serves as the baseline for comparison with other model configurations.

sciBERT: By only utilizing sciBERT with the titles and abstracts, we see a slight increase in precision to 0.928 but a notable drop in recall to 0.896. The F1-score of 0.911 indicates that while sciBERT alone is effective, the additional components in the EnsembleAI model contribute to overall performance.

BERT-base: Replacing sciBERT with BERT-base leads to a precision of 0.919 and a further decrease in recall to 0.877. The F1-score drops to 0.898, suggesting that BERT-base is less effective at identifying AI-related articles than the tailored sciBERT in the context of this task.

RoBERTa: With RoBERTa, there is a slight decrease in both precision and recall compared to BERT-base, resulting in an F1-score of 0.895.

sciBERT (all): This configuration, which feeds a concatenation of title, abstract, WoS category, and keywords directly into sciBERT, achieves a precision of 0.935 and a recall of 0.904. The F1-score improves to 0.919, indicating that incorporating category and keywords information directly into sciBERT’s input can enhance performance, though it does not reach the efficacy of the full EnsembleAI model.

These results affirm that while sciBERT is a powerful tool for classifying AI-related articles, the combination of decision trees, keyword matching, and an SVM integrator in the EnsembleAI model leads to superior recall and the best overall performance as measured by the F1-score. Each component of the EnsembleAI model contributes uniquely to its high level of accuracy and ability to comprehensively identify relevant articles, illustrating the benefits of a multifaceted approach to article classification in rapidly evolving research fields like AI.