Since January 2020 Elsevier has created a COVID-19 resource centre with

free information in English and Mandarin on the novel coronavirus COVID-

19. The COVID-19 resource centre is hosted on Elsevier Connect, the
company's public news and information website.

Elsevier hereby grants permission to make all its COVID-19-related

research that is available on the COVID-19 resource centre - including this
research content - immediately available in PubMed Central and other
publicly funded repositories, such as the WHO COVID database with rights
for unrestricted research re-use and analyses in any form or by any means
with acknowledgement of the original source. These permissions are
granted for free by Elsevier for as long as the COVID-19 resource centre
remains active.
 Applied Soft Computing 132 (2023) 109851

Contents lists available at ScienceDirect

Applied Soft Computing

journal homepage: www.elsevier.com/locate/asoc

Automatic detection of Covid-19 from chest X-ray and lung computed

tomography images using deep neural networks and transfer learning
∗
Linh T. Duong a , Phuong T. Nguyen b , , Ludovico Iovino c , Michele Flammini c
a
Institute of Research and Development, Duy Tan University, Viet Nam
b
Department of Information Engineering, Computer Science and Mathematics University of L’Aquila, Italy
c
Gran Sasso Science Institute, Italy

article info a b s t r a c t

Article history: The world has been undergoing the most ever unprecedented circumstances caused by the coronavirus
Received 29 October 2020 pandemic, which is having a devastating global effect in different aspects of life. Since there are not
Received in revised form 2 October 2022 effective antiviral treatments for Covid-19 yet, it is crucial to early detect and monitor the progression
Accepted 14 November 2022
of the disease, thereby helping to reduce mortality. While different measures are being used to
Available online 24 November 2022
combat the virus, medical imaging techniques have been examined to support doctors in diagnosing
Keywords: the disease. In this paper, we present a practical solution for the detection of Covid-19 from chest
COVID-19 X-ray (CXR) and lung computed tomography (LCT) images, exploiting cutting-edge Machine Learning
AI Diagnosis systems techniques. As the main classification engine, we make use of EfficientNet and MixNet, two recently
Expert systems developed families of deep neural networks. Furthermore, to make the training more effective and
Chest X-ray image
efficient, we apply three transfer learning algorithms. The ultimate aim is to build a reliable expert
Lung CT images
system to detect Covid-19 from different sources of images, making it be a multi-purpose AI diagnosing
system. We validated our proposed approach using four real-world datasets. The first two are CXR
datasets consist of 15,000 and 17,905 images, respectively. The other two are LCT datasets with 2,482
and 411,528 images, respectively. The five-fold cross-validation methodology was used to evaluate the
approach, where the dataset is split into five parts, and accordingly the evaluation is conducted in five
rounds. By each evaluation, four parts are combined to form the training data, and the remaining one
is used for testing. We obtained an encouraging prediction performance for all the considered datasets.
In all the configurations, the obtained accuracy is always larger than 95.0%. Compared to various
existing studies, our approach yields a substantial performance gain. Moreover, such an improvement
is statistically significant.
© 2022 Elsevier B.V. All rights reserved.

1. Introduction patients to severe lung dysfunction. Given the circumstances,

patients need ventilation as they are highly exposed to multiple
Covid-19 is a coronavirus-induced infection that can be asso- extra pulmonary organ failure.
ciated with a coagulopathy and infection-induced inflammatory Since so far there have been no effective antiviral vaccines for
changes [1]. The disease poses a serious threat to public health, Covid-19, it is crucial to reduce mortality by early detecting and
and thus in March 2020, the World Health Organization (WHO) monitoring the progression of the disease [2], so as to effectively
declared Covid-19 a pandemic. At the time of writing, the virus personalize patient’s treatment. Radiology is part of a fundamen-
has infected more than four hundreds millions of people, and has tal process to detect whether or not the radiological outcomes are
claimed over six million peoples’ lives worldwide.1 The clinical consistent with the infection and radiologists should expedite as
spectrum of the disease is very wide, ranging from fever, dry much as possible the exploration, and provide accurate reports of
cough and diarrhea, but can be combined with mild pneumo- their findings. Chest X-ray (CXR) and lung computed tomography
nia and mild dyspnoea. In some cases, the infection can evolve (LCT) images of Covid-19 patients usually show multifocal, bilat-
to severe pneumonia, causing approximately 5% of the infected eral and peripheral lesions, but in the early phase of the disease
they may present a unifocal lesion, most commonly located in
∗ Corresponding author. the inferior lobe of the right lung. Considering the fact that the
E-mail addresses: [email protected] (L.T. Duong), number of false positives obtained by swab results is considerably
[email protected] (P.T. Nguyen), [email protected] (L. Iovino), large, an expert system able to provide doctors with a preliminary
[email protected] (M. Flammini). diagnosis by automated recognition of Covid-19 from CXR or LCT
1 https://www.worldometers.info/coronavirus/. images would be of crucial importance.

https://doi.org/10.1016/j.asoc.2022.109851
1568-4946/© 2022 Elsevier B.V. All rights reserved.
L.T. Duong, P.T. Nguyen, L. Iovino et al. Applied Soft Computing 132 (2023) 109851

The proliferation of disruptive Machine Learning (ML) and Generally, kernels of size k(3,3) [12,13], k(5,5) [14], or k(7,7)
especially Deep Learning (DL) algorithms in recent years has [15] are used as filters for deep neural networks. However, larger
enabled a plethora of applications across several domains [3,4]. kernels can potentially improve a model’s accuracy and effi-
Such techniques work on the basis of complex artificial neural ciency. Furthermore, large kernels help to capture high-resolution
networks, which are capable of effectively learning from data patterns, while small kernels allow us to better extract low-
by means of a large number of hyper-parameters distributed in resolution ones. To maintain a balance between accuracy and
different network layers. In this way, they are able to simulate efficiency, the MixNet [8] family has been built based on the
humans’ cognitive functions, aiming to acquire real-world knowl- MobileNets architectures [13,16]. This network family also aims
edge autonomously [5]. In a nutshell, ML/DL techniques are an to reduce the number of parameters as well as FLOPs, i.e., the
advanced paradigm that brings in substantial improvement in metric used to measure the computational complexity counted
performance compared to conventional learning algorithms. In
as the number of float-point operations (in billions). The most
the Health care sector, the potential of ML/DL to allow for rapid
simple architecture of the MixNet family is MixNet-Small, which
diagnosis of diseases has also been proven by various research
consists of a large number of layers and channels. Furthermore,
work [6].
the size of the filters varies depending on the layers. Similar
Aiming to assist the clinical care, this paper presents a prac-
to the EfficientNet family, other configurations of the MixNet
tical solution for the detection of Covid-19 from CXR and LCT
images exploiting two cutting-edge deep neural network families, family, such as MixNet-Medium or MixNet-Large, are derived
i.e., EfficientNet [7] and MixNet [8]. Moreover, we empower the from MixNet-S with different scaling values.
learning process by means of three different transfer learning
strategies, namely ImageNet [9], AdvProp [10] and Noisy Stu- 2.2. Transfer learning
dent [11]. The evaluation on four large CXR and LCT images
datasets demonstrate that our proposed models obtain a superior To fine tune the hyper-parameters, i.e., the internal weights
performance compared to the existing studies that we are aware and biases, deep neural networks need a huge number of la-
of. In this respect, our work makes the following contributions: beled data. Moreover, the deeper/wider a network is, the more
• A system for detection of Covid-19 from CXR and LCT images parameters it possesses. As a result, deeper/wider networks re-
exploiting cutting-edge deep learning algorithms; quire more data to guide the learning, with the aim of avoiding
• A successful empirical evaluation on four large datasets overfitting and being effective. Thus, it is crucial to train them
consisting of CXR and LCT images; with enough data, in order to facilitate the learning. Nevertheless,
• A software prototype in the form of a mobile app ready to such a requirement is hard to come by in real-world settings,
be downloaded. since the labeling process usually is done manually, thus being
time consuming and subject to error [17]. To this end, transfer
The paper is structured as follows. Section 2 briefly introduces learning has been conceptualized as an effective way to extract
EfficientNet and MixNet as well as the transfer learning methods. and transfer the knowledge from a well-defined source domain
In Section 3, we present the dataset and metrics used for our
to a novice target domain [18,19]. In other words, transfer learn-
evaluation, while in Section 4 we analyze the experimental re-
ing facilitates the export of existing convolution weights from a
sults. The related work is reviewed in Section 5. Finally, Section 6
model trained using large datasets to create new accurate models
discusses future work and concludes the paper.
exploiting a relatively lower number of labeled images. As it has
2. Background been shown in various studies [20,21], transfer learning remains
helpful even when the target domain is quite different from the
As a base for our presentation, Section 2.1 provides a back- one in which the original weights have been obtained. In this
ground on two families of deep neural networks, i.e., EfficientNet work, we consider the following learning methods:
and MixNet, which are used as the classification engine in our
• ImageNet [9]: The ImageNet dataset has been widely ex-
work. Afterwards, a brief introduction to transfer learning is given
ploited to apply transfer learning by several studies, since
in Section 2.2.
it contains more than 14 million images, covering miscella-
2.1. EfficientNet and MixNet neous categories;
• AdvProp [10]: adversarial propagation has been proposed
Based on the observation that a better accuracy and efficiency as an improved training scheme, with the ultimate aim of
can be obtained by imposing a balance between all network avoiding overfitting. The method treats adversarial exam-
dimensions, EfficientNet [7] has been proposed by scaling in ples as additional examples, and uses a separate auxiliary
three dimensions, i.e., width, depth, and resolution, using a set batch norm for adversarial examples;
of fixed scaling coefficients that meet some specific constraints. • NS [11]: the Noisy Student learning method attempts to
By the most compact configuration, i.e., EfficientNet-B0, there are improve ImageNet classification Noisy Student Training by:
18 convolution layers in total, i.e., D = 18, and each layer is (i) enlarging the trainee/student equal to or larger than the
equipped with a kernel k(3,3) or k(5,5). The input image con- trainer/teacher, aiming to make the trainee learn better on
tains three color channels R, G, B, each of size 224 × 224. The a large dataset, and (ii) adding noise to the student, thus
next layers are scaled down in resolution to reduce the feature forcing him to learn more.
map size, but scaled up in width to increase accuracy. For in-
stance, the second convolution layer consists of W = 16 filters, To assist doctors in early detecting Covid-19 from CXR and
and the number of filters in the next convolution layer is W = LCT images, we develop an expert system that makes use of
24. The maximum number of filters is D = 1280 in correspon- EfficientNet and MixNet as the classification engine. Moreover, in
dence of the last layer, which is fed to the final fully connected order to accelerate the learning process and to achieve a higher
layer. The other configurations of the EfficientNet family are accuracy, we propose the three different learning strategies for
generated from EfficientNet-B0 by means of different scaling val- obtaining network weights mentioned above, i.e., ImageNet, Ad-
ues [7]. EfficientNet-B7 outperforms a CNN by achieving a better vProp, and NS. The succeeding section introduces the evaluation
accuracy, while considerably reducing the number of parameters. settings used to study the performance of our approach.
2
L.T. Duong, P.T. Nguyen, L. Iovino et al. Applied Soft Computing 132 (2023) 109851

Table 1 Table 2
Chest X-ray datasets. Lung CT datasets.
Dataset Type Categories Total Dataset Type Categories Total
Covid-19 Normal Pneumonia Covid-19 Normal Pneumonia
Train 98 7,966 5,447 13,511 Train 1,001 – 984 1,985
D1 Test 10 885 594 1,489 D3 [24] Test 125 – 123 248
Total 108 8,851 6,041 15,000 Validation 126 – 123 249
Train 261 8,154 5,909 14,324 Total 1,252 – 1,230 2,482

D2 Test 66 2,038 1,477 3,581 Train 109,250 67,029 111,791 288,070

Total 327 10,192 7,386 17,905 D4 [25] Test 23,410 14,364 23,955 61,729
Validation 23,411 14,364 23,955 61,730
Total 156,071 95,756 159,702 411,528

3. Evaluation
Table 3
Hardware and software configurations.
We explain in detail the datasets and methods used to study
Name Description
the performance of our proposed solution. Four real-world
RAM 24 GB
datasets have been used in the evaluation. Moreover, we make ®
CPU Intel Core™ i5-2400 CPU @ 3.10 GHz ×4
use of recent implementations2 of EfficientNet and MixNet, which
GPU GeForce GTX 1080 Ti
have been built atop the PyTorch framework.3 We imported pre- OS Ubuntu 18.04
trained weights from various sources to speed up the training. Python 3.7.5
We publish the tool developed through this paper in GitHub to Pytorch 1.5
facilitate future research.4 Torchvision 0.5.0
Numpy 1.15.4
Git 2.0
3.1. Research objectives Timm 0.1.26

Given an automated classifier, it is of crucial importance to get

accurate predictions with respect to various quality metrics. In
the evaluation, we focus on the following performance charac- Table 2 summarizes the two LCT datasets used in our eval-
teristics: (i) We identify which neural network family obtains the uation. Dataset D3 contains a small number of images in two
best prediction performance for the CXR datasets; (ii) Afterwards, categories, i.e., Covid-19 and Pneumonia. Finally, D4 is much larger
we determine which deep neural network is the most suitable with 156,017 images for the Covid-19 category. In total, D4 sums
one for classifying LCT images; (iii) We ascertain which transfer up to 411,528 images for training and testing. We believe that
learning method among ImageNet, AdvProp, and NS is beneficial such a large dataset resembles a real scenario, and thus the
to EfficientNet or MixNet; (iv) We measure the average recogni- evaluation on it helps to shed light on the performance of the
tion speed to see if the proposed model is feasible in practice in approach in practice.
terms of timing efficiency; Finally, (v) A comparison with well- Concerning the settings, at the beginning of all experiments
established baselines is performed to validate if our approach the datasets were randomly split using a random seeds function
brings advancements in the field. of the Python numpy library. With respect to the strategy it
ensures that the test sets are independent and all the same for
3.2. Datasets the other evaluations of each architecture. In other words, we
can measure and compare prediction performance on the same
To investigate the performance of our approach, with CXR we criterion. This is done by following many existing studies [4,26–
exploited two datasets used in some previous studies [22,23]. 28], which measured the prediction performance on independent
Their characteristics are summarized in Tables 1 and 2. In all of test sets.
them, except D3 , there are three categories, i.e., Covid-19, Normal, Figs. 1 and 2 give examples of CXR and LCT images for the
and Pneumonia. Only the category Normal exhibits no symp- three categories mentioned above. By plain eyes, it is quite dif-
toms, while the other two categories, Covid-19 and Pneumonia, ficult to notice the difference between them, especially for non-
correspond to different levels of infection-induced inflammatory expert users. In this respect, we assume that the application of
changes. deep learning will facilitate the recognition and effectively assist
Dataset D1 consists of 13,511 images for training and 1489 doctors in diagnosing the disease.
images for testing. It can be seen that there is an imbalance
among the categories. In particular, both categories Normal and 3.3. Settings
Pneumonia contain a large number of images, while there are only
108 images in the Covid-19 category. We use D1 as a means to Deep neural networks such as EfficientNet and MixNet require
compare the performance of our approach with some existing a platform with powerful computational performance. To perform
studies that exploited D1 in their evaluation [22,23]. Compared to the evaluation, we used a server with the hardware and software
D1 , D2 is newly updated with more data for training and testing. configurations listed in Table 3.
This means that there are overlaps between the two datasets. In By means of an empirical study, we found out that two config-
particular, D2 consists of 14,324 and 3581 images for training urations, EfficientNet-B0 and EfficientNet-B3, are more effective
and testing, respectively. We made use of D2 to validate the than the others for the EfficientNet family, and thus we selected
performance of our approach on a larger amount of data, proving them for our evaluation. Weights for EfficientNet are obtained by
its feasibility in real-world settings. all the transfer learning techniques mentioned in Section 2.2. In
the MixNet family, we considered four different configurations,
2 https://github.com/rwightman/gen-efficientnet-pytorch. i.e., MixNet-Small, MixNet-Medium, MixNet-Large and MixNet-
3 https://pytorch.org. XL. We obtained weights coming from the available ImageNet
4 https://github.com/linhduongtuan/Covid-19_Xray_Classifier/. dataset. At the end, by combining two network families with
3
L.T. Duong, P.T. Nguyen, L. Iovino et al. Applied Soft Computing 132 (2023) 109851

Fig. 1. Examples of CXR images.

Fig. 2. Examples of LCT images.

Table 4
Experimental configurations.
Conf. Network Batch size # of Params Learning method Size (MB)
C1 EfficientNet-B0 110 7,919,391 ImageNet 53.1
C2 EfficientNet-B0 110 7,919,391 AdvProp 53.1
C3 EfficientNet-B0 110 7,919,391 NS 53.1
C4 EfficientNet-B3 64 14,352,075 ImageNet 106.9
C5 EfficientNet-B3 64 14,352,075 AdvProp 106.9
C6 EfficientNet-B3 64 14,352,075 NS 106.9
C7 MixNet-Small 110 6,253,449 ImageNet 41.8
C8 MixNet-Medium 90 7,133,225 ImageNet 48.9
C9 MixNet-Large 60 9,448,095 ImageNet 67.5
C10 MixNet-XL 60 14,015,611 ImageNet 104.2

three learning strategies mentioned in Section 2.2, we got 10 following reason: precision, recall and F1 are useful, given that the
experimental configurations C1 , . . . , C10 , as shown in Table 4. Batch number of positive images accounts for a very small percentage of
size corresponds to the number of items used for each training all the items in the dataset. In fact, in this case a classifier always
step; # of Params specifies the number of parameters used by providing a negative prediction would have a very high accuracy.
each network; and finally Size is the file size needed to store True positive TPi = |Gi ∩ Si |, i = 1, 2, 3 is defined as the
the parameters. It is evident that EfficientNet-B3 is the largest number of items that appear both in the results and ground-truth
network with respect to the number of parameters as well as the data of class i. The metrics are defined as follows.
file size to store them. To be concrete, i.e., C4 , C5 , and C6 have Accuracy: It is the fraction of correctly classified items with
more than 14 millions of parameters, accounting for more than respect to the total number of images in the test set.
100MB of storage space each. In the evaluation, we employed the ∑3
TPi
five-fold cross-validation methodology on the datasets. Namely, accuracy = ∑3i × 100% (1)
each dataset is split into five equal parts, and in each validation i |Gi |
round one part is used for testing and the other four ones for the
Precision and Recall: Precision is the fraction of classified
training.
images for Si being found in the ground-truth data Gi , while Recall
the fraction of true positives being found in the ground-truth
3.4. Evaluation metrics data.
TPi
Each image in all the datasets has been manually labeled, i.e., precisioni = (2)
either Normal or Pneumonia or Covid-19, resulting in three inde- |Si |
pendent groups, i.e., G = (G1 , G2 , G3 ), called ground-truth data. TPi
Using either EfficientNet or MixNet as the classifier on a test set, recalli = (3)
|Gi |
three predicted sets, i.e., S = (S1 , S2 , S3 ) of images are obtained.
We measured the classification performance by evaluating the F1 score (F-Measure): It is calculated as the average of preci-
similarity of the classified categories with the ground-truth ones. sion and recall using the following formula:
To this end, three metrics, namely accuracy, precision and recall, 2 · precisioni · recalli
F1 = (4)
and F1 score are used [21]. We selected these metrics for the precisioni + recalli
4
L.T. Duong, P.T. Nguyen, L. Iovino et al. Applied Soft Computing 132 (2023) 109851

Table 5
Experimental results on dataset D1 .
Configuration C1 C2 C3 C4 C5 C6 C7 C8 C9 C10
Accuracy (%) 95.64 95.77 95.30 96.17 96.64 95.90 95.30 95.98 96.11 96.37
Covid-19 1.000 1.000 1.000 1.000 0.875 0.857 1.000 1.000 1.000 1.000
Precision Normal 0.952 0.960 0.950 0.957 0.968 0.957 0.953 0.966 0.961 0.964
Pneu. 0.961 0.954 0.956 0.968 0.964 0.963 0.951 0.950 0.960 0.962
Covid-19 0.300 0.300 0.300 0.300 0.700 0.600 0.300 0.400 0.400 0.600
Recall Normal 0.981 0.975 0.977 0.985 0.978 0.978 0.974 0.971 0.978 0.978
Pneu. 0.929 0.942 0.927 0.937 0.952 0.936 0.932 0.951 0.944 0.947
Covid-19 0.461 0.461 0.461 0.461 0.778 0.705 0.461 0.571 0.571 0.750
F1 -score Normal 0.967 0.967 0.963 0.971 0.973 0.967 0.963 0.969 0.969 0.971
Pneu. 0.945 0.948 0.941 0.953 0.958 0.949 0.942 0.954 0.952 0.955

While accuracy is important, we are also interested in effi- Similarly, let us analyze the results on dataset D2 presented
ciency, considering the fact that the model needs to have a high in Table 6 to ascertain the best network configuration. It is ev-
recognition speed in practice. ident that C1 achieves most of the best scores with respect to
Speed: The system presented in Table 3 is used to benchmark various metrics as well as categories. For instance, C1 get 95.82%
the processing time, i.e., the average number of predicted items as accuracy, together with C8 , which is the best over all the
in a second. configurations.
In the next section, we present in detail the experimental Altogether, through Tables 5 and 6 we can see that C1 and
results by referring to the research questions introduced in Sec- C5 are the configurations among the others that bring the best
tion 3.1. prediction performance.
Compared to existing work that performs evaluation on the
4. Results same dataset [22,29], our approach achieves a better performance
with respect to accuracy, precision, recall, and F1 -score. For in-
stance, the work by Wang et al. [22], the maximum accuracy
This section reports and analyzes the results obtained from
is 93.0% with similar experimental settings. In this respect, we
the conducted experiments by referring to the research objectives
conclude that application of the two network families EfficientNet
presented in Section 3.1.
and MixNet as well as the different transfer learning techniques
brings a good prediction performance on the considered dataset.
4.1. Performance on the CXR datasets We conclude that EfficientNet and MixNet can successfully
predict Covid-19 from CXR images, obtaining a high accuracy and
The results obtained by performing the experiments on D1 are precision. Among others, EfficientNet yields the best prediction
shown in Table 5. The accuracy for all configurations is always performance.
larger than 95%, and the maximum accuracy is 96.64% obtained by
C5 , i.e., EfficientNet-B3 using pre-trained weights with AdvProp. 4.2. Performance on the LCT datasets
With respect to Precision, eight among ten configurations get
1.000 as precision for the Covid-19 category. This means that all The experimental results for the two LCT datasets, i.e., D3 and
images classified as Covid-19 by the classifiers are actually Covid- D4 are shown in Tables 7 and 8, respectively. Table 7 demon-
19. For the other two categories, i.e., Normal and Pneumonia, the strates that EfficientNet-B0 trained with ImageNet is the most ef-
maximum precision is 0.968, achieved also by C5 for Category fective configuration, i.e., it obtains 97.99% as accuracy. Moreover,
Normal, and by C4 for Category Pneumonia. Altogether, we see this configuration also gets the best F1 score for both categories.
that all the classifiers are able to recognize the testing images, Concerning D4 , the dataset consists of a large number of im-
obtaining high precision. ages (cf. Table 2). This resembles a real-world scenario where
Concerning recall, all the configurations get a considerably low images are collected and fed to the system on a daily basis,
score for the Covid-19 category: the highest recall is 0.700, ob- resulting in a big database. We see that both network families
tained by C5 . This means that the models are not able to retrieve get a high performance with respect to different metrics. All the
all the items in the ground-truth data, though they can yield good configurations obtain an accuracy larger than 98% with 99.66% as
predictions for the category. We assume that this happens due maximum accuracy.
to the limited data available in the training set. Referring back In comparison to existing studies that performed evaluation
to Table 1, for the Covid-19 category there are only 98 images on same datasets, we can see that our proposed framework
and 10 images for training and testing, respectively. Meanwhile, achieves a better performance. For instance, He et al. [30] obtain
for other two image categories, the recall scores are substantially 87.93% as maximum accuracy, while we get 97.99% and 99.66%
improved. The best performance is seen by category Normal, i.e., for D3 and D4 , respectively.
0.985; while by Pneumonia, recall is 0.952. As depicted in Table 1, On the LCT datasets, EfficientNet is also the network family
these categories consist of a larger number of training and testing that helps to obtain the best prediction performance.
images compared to the Covid-19.
For what concerns the F1 scores, for the Covid-19 category the 4.3. The benefit of transfer learning
maximum F1 is 0.778, obtained by C5 . The classifiers obtain a
low F1 in the other configurations, and this happens due to the We conducted experiments following the five-fold cross-
low recall scores as shown above. For the other two categories validation technique. Moreover, to further investigate the appli-
Normal and Pneumonia, the F1 scores are improved considerably cability of the proposed approach, we made use of D2 and D4 ,
compared to Covid-19. C5 is the most suitable configuration as it which contain more images than D1 and D3 (cf. Table 1). Figs. 3(a),
obtains the best F1 scores for all the categories, i.e., also 0.973 for 3(b), and 3(c) depict the confusion matrices for EfficientNet-B0
Normal and 0.958 for Pneumonia. using the three different transfer learning techniques mentioned
5
L.T. Duong, P.T. Nguyen, L. Iovino et al. Applied Soft Computing 132 (2023) 109851

Table 6
Experimental results on dataset D2 .
Configuration C1 C2 C3 C4 C5 C6 C7 C8 C9 C10
Accuracy (%) 95.82 94.39 93.30 95.05 95.59 95.00 95.79 95.82 95.68 95.53
Covid-19 0.968 0.948 0.889 0.950 0.968 0.978 0.982 0.966 0.983 0.967
Precision Normal 0.958 0.942 0.932 0.942 0.955 0.953 0.957 0.957 0.951 0.951
Pneu. 0.957 0.946 0.935 0.964 0.955 0.944 0.958 0.959 0.963 0.960
Covid-19 0.924 0.560 0.363 0.863 0.924 0.667 0.833 0.863 0.909 0.909
Recall Normal 0.958 0.942 0.932 0.942 0.955 0.944 0.974 0.973 0.975 0.960
Pneu. 0.942 0.922 0.906 0.912 0.939 0.937 0.941 0.941 0.933 0.932
Covid-19 0.945 0.704 0.616 0.905 0.945 0.792 0.901 0.922 0.945 0.937
F1 -score Normal 0.644 0.956 0.951 0.959 0.962 0.960 0.965 0.965 0.963 0.962
Pneu. 0.950 0.934 0.920 0.940 0.947 0.940 0.950 0.950 0.947 0.946

Table 7
Experimental results on dataset D3 .
Configuration C1 C2 C3 C4 C5 C6 C7 C8 C9 C10
Accuracy (%) 97.99 96.24 94.09 97.31 97.58 97.31 96.77 97.31 96.51 97.81
Covid-19 0.991 0.972 0.982 0.978 0.994 0.973 0.973 0.978 0.978 0.957
Prec.
Pneu. 0.968 0.952 0.905 0.967 0.991 0.973 0.962 0.967 0.958 0.957
Covid-19 0.968 0.951 0.898 0.967 0.958 0.972 0.962 0.967 0.951 0.978
Rec.
Pneu. 0.991 0.973 0.983 0.978 0.994 0.973 0.973 0.978 0.978 0.967
Covid-19 0.979 0.962 0.938 0.973 0.975 0.973 0.967 0.973 0.964 0.967
F1
Pneu. 0.979 0.962 0.943 0.973 0.975 0.973 0.967 0.973 0.965 0.967

Table 8
Experimental results on dataset D4 .
Configuration C1 C2 C3 C4 C5 C6 C7 C8 C9 C10
Accuracy (%) 99.66 99.52 98.85 99.59 99.62 99.47 99.52 98.41 99.59 99.53
Covid-19 0.997 0.995 0.991 0.997 0.996 0.995 0.994 0.990 0.996 0.996
Precision Normal 0.990 0.989 0.977 0.988 0.989 0.987 0.990 0.962 0.989 0.986
Pneu. 0.999 0.998 0.992 0.999 0.999 0.998 0.999 0.991 0.999 0.998
Covid-19 0.994 0.994 0.988 0.993 0.994 0.992 0.994 0.983 0.994 0.992
Recall Normal 0.996 0.991 0.983 0.995 0.995 0.991 0.990 0.978 0.994 0.995
Pneu. 0.992 0.998 0.991 0.999 0.998 0.998 0.998 0.988 0.998 0.999
Covid-19 0.995 0.994 0.989 0.995 0.995 0.994 0.994 0.986 0.995 0.994
F1 -score Normal 0.993 0.990 0.980 0.991 0.992 0.989 0.990 0.970 0.991 0.990
Pneu. 0.999 0.998 0.992 0.999 0.999 0.998 0.998 0.989 0.999 0.998

in Section 2.2. The metrics for all the confusion matrices are classifying 1,984 images among 2,038 images, while it suffers of
shown in Table 6. a low precision and recall for the other categories. For instance,
As we can see, each transfer learning method may have dif- with Pneumonia, only 1,377 out of 1,477 images are properly rec-
ferent effects on the different categories. For example, using ognized by MixNet-XL with weights pre-trained with ImageNet.
EfficientNet-B0 with weights pre-trained by ImageNet is benefi- The experimental results demonstrate that, depending on the
cial to Covid-19 and Pneumonia, but not to Normal. As shown in
network family, each transfer learning technique has a diverse
Fig. 3(a), 61 out of 66 images in Covid-19 are correctly classified,
influence on the final outcomes. By considering the results in
while for Pneumonia 1,392 out of 1,477. However, for the Normal
Table 6, it is evident C1 , corresponding to training EfficientNet-
category, only 1978 images are correctly classified over a total of
2,038 images, accounting for 97.05%. On the other hand, transfer B0 with weights by ImageNet, is the most effective configuration
learning with AdvProp (cf. Fig. 3(b)) induces a better performance with respect to accuracy, precision, recall, and F1 for almost all
for Normal, i.e., 1,981 among 2,038 images are classified to the categories. Moreover, together with the results obtained from
correct categories. In Fig. 3(c), we see that compared to the other Section 4.1, we conclude that ImageNet is the best transfer learning
learning methods, NS has an adverse effect on the recognition strategy for both network families on the two datasets D1 and D2 .
of all the categories. Altogether, we come to the conclusion that Next, we consider the results obtained for the LCT dataset in
training EfficientNet-B0 with weights from ImageNet yields the Figs. 4(a)–4(j). In general, training the deep neural networks with
best prediction performance. ImageNet helps to obtain a better performance compared to the
For EfficientNet-B3, we see that weights pre-trained with Im- other transfer learning techniques in all the three categories. As
ageNet are beneficial to the Normal category (cf. Fig. 3(d)). At
an example, with C1 we get 23,281 correctly classified Covid-
the same time, AdvProp is the transfer learning method that is
19 images, while the corresponding numbers for AdvProp and
suitable for recognition of Pneumonia, i.e., it helps to detect 1,388
NS are 23,270 and 23,137, respectively. For other categories,
out of 1,477 pneumonia images, which is the best among the
others. training EfficientNet-B0 with ImageNet weights also gets a good
Among the configurations, C9 is the best one for the Pneumonia performance.
category. Other MixNet configurations do not outperform the The experimental results show that weights pre-trained from
ones of EfficientNet-B0 and EfficientNet-B3. MixNet-XL obtains a the ImageNet dataset contribute to the best prediction perfor-
considerably good performance with Category Normal, correctly mance on all the CXR and LCT datasets.
6
L.T. Duong, P.T. Nguyen, L. Iovino et al. Applied Soft Computing 132 (2023) 109851

Fig. 3. Confusion matrices of EfficientNet and MixNet using different transfer learning techniques on D2 .

Fig. 4. Confusion matrices of EfficientNet-B0 and EfficientNet-B3 using different transfer learning techniques on D4 .

7
L.T. Duong, P.T. Nguyen, L. Iovino et al. Applied Soft Computing 132 (2023) 109851

Fig. 5. Recognition speed for the configurations on dataset D2 and D4 .

4.4. Timing efficiency Table 9

Results obtained with ResNet50.

We investigate whether the model is feasible in practice by Configuration D2 D4

measuring the time efficiency. The system presented in Table 3 is Accuracy (%) 87.53 91.93
used to benchmark the processing time, i.e., the average number Covid-19 0.880 0.909
of predicted items in a second for the cross-validation experi- Precision Normal 0.843 0.922
Pneu. 0.900 0.923
ments. For these experiments, D2 and D4 have been selected as
they contain a larger number of images compared to D1 and D3 . Covid-19 0.846 0.926
Recall Normal 0.902 0.813
We counted the number of predictions returned by the classifiers
Pneu. 0.874 0.952
in a second for the two datasets. The results are depicted in Fig. 5.
Covid-19 0.863 0.918
Concerning D2 , from the figure it is clear that the considered F1 -score Normal 0.872 0.864
configurations have a comparable recognition speed. For instance, Pneu. 0.887 0.938
C1 , C2 , C3 , and C7 corresponding to using EfficientNet-B0 and
MixNet-S as the classification engine, are the most efficient con-
figurations, as they return 206 and 235 images per second in
average. EfficientNet-B3 also yields a good time performance, i.e., Computed Tomography categories, i.e., D2 and D4 , aiming to study
using C4 , C5 , or C6 as the experimental configuration, the system the generalizability.
generates 165 and 183 predictions per second. The experimental results obtained using ResNet50 are shown
The recognition speed is improved a lot by D4 , i.e., on the in Table 9. The approach gets an accuracy of 87.53% and 91.93%
LCT dataset with more images for training, compared to D2 . The on D2 and D4 , respectively. ResNet50 performs the best on the
maximum speed is 965 images per second and reached by C1 , C2 , Pneumonia, i.e., it always gets a maximum value for all the three
i.e., using EfficientNet-B0 together with ImageNet and AdvProp. metrics (Precision, Recall, and F1-score). Concerning Precision,
ResNet50 gets the maximum value of 0.923 by the Pneumonia
Among the MixNet configurations, MixNet-S is the most efficient
category on D4 . The corresponding scores for Recall and F1-score
one, as it classifies 921 images per second, which is much higher
are 0.952 and 0.938, respectively.
compared to 664, 510 and 367 by MixNet-M, MixNet-L, and
While ResNet50 obtains a good prediction, compared to our
MixNet-XL, respectively.
approach, its performance is considerably lower. For instance, as
Together with the results obtained from Section 4.1 and Sec-
shown in Table 8, on the D4 dataset, with Configuration C1 our ap-
tion 4.2, the results demonstrate that the proposed framework is
proach gets 99.66% as accuracy. The maximum Precision, Recall,
feasible in real-world settings, i.e., it can be used to effectively and and F1 scores are 0.999, 0992, and 0.999, respectively. Finally, we
efficiently predict Covid-19 from CXR and LCT images. Moreover, investigate if the gain in performance is statistically significant.
EfficientNet-B0 is the most efficient model on both datasets, To this end, we perform a Wilcoxon signed-rank test [34] with
while the MixNet family yields a low timing efficiency. a significance level of α = 5% on the obtained Precision, Recall,
and F1 scores from both approaches. The final p-value is 1.903 ×
4.5. Comparison with a baseline 10−7 , which means that the difference is statistically significant.
Altogether, this demonstrates that compared to the baselines, we
ResNet is a modern deep neural network, and it has been always achieve a better performance.
widely used in different classification tasks [31]. For the detection The original EfficientNet-B0 configuration contains 5.29M
of Covid, ResNet50 has been applied in various work [32,33], params, 385.89 MMac, and it obtains an accuracy of 76.3% on
obtaining a promising prediction performance. In this section, we the ImageNet-1K dataset [9]. However, we noticed that it cannot
run experiments to compare our approach with two baselines distinguish well between Covid-19 with other pneumonia. Thus,
built on top of the ResNet50 architecture. For the comparison, we attempted to improve this by modifying the original config-
we use the larger dataset among either the Chest X-ray and Lung uration by adding 12 layers to boost up the prediction accuracy
8
L.T. Duong, P.T. Nguyen, L. Iovino et al. Applied Soft Computing 132 (2023) 109851

for the medical image classification for Sars-CoV-2 based on Chest based on his personal data as well as other risk factors [50]. Jiang
X-ray (CXR) and Computed Tomography (CT). The new proposed et al. [51] propose an approach to the identification of clinical
model EfficientNet-B0 contains 7.92 M params, but still keeps characteristics of Covid-19, and develop an AI tool to recognize
the same number of MMac, i.e., 385.89 MMac. In summary, the patients at risk of a more severe impact of the disease.
modification of the original backbones is our key contribution for Ozturk et al. [52] proposed an approach to deliver accurate
the classification task. diagnostics for binary classification, i.e., Covid-19 vs. No-Findings
and multi-nominal classification, i.e., Covid-19 and No-Findings
4.6. Threats to validity and Pneumonia. The DarkNet model has been exploited as the
classification engine, consisting of 17 convolutional layers and a
We discuss the threats that may affect the internal, external, different filtering in each layer. The proposed model has a predic-
construct, and conclusion validity of our findings. tion accuracy of 98.08% and 87.02% for binary and multi-nominal
Internal validity. This concerns the internal factors that might classification, respectively.
have an adverse influence on the findings. A possible threat here Various studies have demonstrated the usefulness of CXR ex-
could come from the results for the Covid-19 category, since they ams in detecting Covid-19. Hall et al. [53] analyzed 135 CXR im-
are obtained with a considerably low number of items for training ages confirmed as Covid-19 and 320 images of viral and bacterial
and testing, i.e., D1 with 98 and 10 images and D2 with 327 and pneumonia. A Resnet50 DNN was trained on 102 Covid-19 and
98 images for training and testing, respectively. This threat is 102 pneumonia cases by means of the ten-fold cross-validation
mitigated by the other two categories in the datasets, as they technique. The experimental results showed an overall accuracy
contain a considerably large number of items. To the best of our of 89.2% with a Covid-19 true positive rate of 0.804 and an area
knowledge, there exists no CXR dataset with more images for under the curve (AUC) of 0.95. As a matter of fact, the dataset used
the Covid-19 category. As a matter of fact, research in medical by Hall et al. [53] is quite small, and it is necessary to investigate
imaging on Covid-19 suffers a general lack of data. For this reason, the proposed model on a larger amount of data to see if it still
unfortunately we are not able to study the models on a larger achieves such a good performance.
scale. Narin et al. [37] developed a system for the detection of
External validity. The main threat to external validity is due to coronavirus patients from CXR images. Three different CNN-
the factors that might hamper the generalizability of our results based models have been exploited, i.e., ResNet50, InceptionV3
and Inception-ResNetV2. The results show that the ResNet50
to other scenarios outside the scope of this work, e.g., in prac-
model achieves the best prediction performance with 98.0% as
tice we may encounter a limited amount of training data. The
the final accuracy. Although the approach obtains a good classi-
threat is moderated by evaluating EfficientNet and MixNet using
fication performance, it has been studied on a considerably small
the experimental settings following the five-fold cross-validation
dataset. It is our belief that performance may substantially change
methodology. In particular, the dataset is split into five parts, and
on larger datasets like the ones used in our evaluation.
accordingly the evaluation is conducted in five rounds. By each
Apostolopoulos et al. [38] evaluated their solution to auto-
round, four parts are combined to form the training data, and the
matic detection of Covid-19, making use of a dataset of CXR
remaining one is used for testing.
images from patients with common bacterial pneumonia, con-
Construction validity. This is related to the experimental settings
firmed Covid-19, and normal incidents. The datasets consists of
presented in the paper, concerning the simulation performed to
1427 CXR images including 224 images with confirmed Covid-19
evaluate the system. To mitigate the threat, the evaluation has
cases, 700 images with common bacterial pneumonia, and 504
been conducted on a training set and a test set, attempting to
images of normal situations. The experimental outcomes demon-
simulate a real usage where training data is already available for
strate that deep neural networks can be exploited to extract
feeding the system, while testing data is the part that needs to
important biomarkers related Covid-19. Nevertheless, like some
be predicted. Depending on the given settings, we ran several other existing studies et al. [37], again the approach has been
rounds of training/testing, to make sure that the final results are studied by means of a small amount of data. It is our assumption
generalizable, i.e., they are not obtained by chance. In the paper, that such a good performance might considerably change with
we reported the most stable experimental results. larger datasets.
Conclusion validity. This is related to the factors that might COVID-Net [54] is a deep convolutional neural network design
have an impact on the obtained outcome. The evaluation metrics tailored for the detection of Covid-19 cases from CXR images.
accuracy, precision, recall, F1 and execution time might induce a COVID-Net achieves an accuracy of 93.3%, with 98.9% positive
conclusion threat. To face the issue, we adopted such measures predictive values that is related to the detection of false positives.
as recommended by the previous scientific literature related to A deep learning model has been proposed [55] to detect Covid-
our setting, and employed the same metrics for evaluating all the 19 and differentiate it from common acquired pneumonia and
classifiers. other lung diseases. The analyzed dataset consists of 4356 chest
CT exams collected from 3322 patients. The per-exam sensitivity
5. Related work and specificity for detecting COVID-19 in the independent test set
was 114 of 127 (90.0%) and 294 of 307 (96.0%), respectively, with
The recent months have witnessed a large number of studies an area under the receiver operating characteristic curve (AUC) of
related to the topic Covid-19 and Machine Learning, and multiple 0.96 (p-value < 0.001). The per-exam sensitivity and specificity
Covid-19/ML applications have been proposed. We summarize for detecting community acquired pneumonia in the independent
in Table 10 some of the most notable research, providing the test set was 87% (152 of 175) and 92% (239 of 259), respectively.
number of considered images for each category as well as the pre- Abbas et al. [36] introduced Decompose, Transfer, and Com-
diction accuracy. In this work, since we support the recognition pose (DeTraC), a deep neural network for automated recognition
of Covid-19 from CXR and LCT images, in the remaining of this of Covid-19 from CXR images. An accuracy of 95% was achieved
section we concentrate on analyzing these studies. in the detection of Covid-19 CXR images from normal, and severe
Deep learning techniques have been exploited to predict which acute respiratory syndrome cases. COVID-CAPS [56] is a capsule
current antivirals might be more effective in patients infected Network-based Framework for Identification of Covid-19 cases
with coronavirus [48,49]. Similarly, a specific model has been de- from CXR Images. The approach yielded a good accuracy when
veloped to forecast if a Covid-19 patient has the chance to survive working with small datasets.
9
L.T. Duong, P.T. Nguyen, L. Iovino et al. Applied Soft Computing 132 (2023) 109851

Table 10
A summary of the related studies on CXR datasets.
Study Number of images Network Acc. (%)
Covid-19 Normal Pneumonia
Ghoshal et al. [35] 68 1583 2786 ResNet 89.82
Abbas et al. [36] 105 80 11 DeTraC based on ResNet-18 95.12
Nari et al. [37] 50 50 – ResNet-50 98.00
Apostolopous et al. [38] 224 504 700 VGG19 93.48
Luz et al. [39] 183 – – EfficientNet-B3 93.90
Zhang et al. [40] 100 1431 1531 ResNet18 96.00
Hemdan et al. [41] 25 25 – VGG19, DenseNet121 90.00

Table 11
A summary of the related studies on LCT datasets.
Study Number of images Network Acc. (%)
Covid-19 Normal Pneumonia
Rahimzadeh et al. [42] 465 7878 – ResNet50V2 98.49
Anwar et al. [43] 98 – 105 EfficientNet 89.70
Gunraj et al. [44] 4346 9450 7395 COVID-Net-CT 99.10
Mobiny et al. [45] 47 58 – DECAPS 87.60
He et al. [30] 25,442 14,471 28,160 ResNet3D34 95.90
Ardakanu et al. [46] 510 510 – ResNet-101 99.63
Bai et al. [47] 5030 – 9152 EfficientNet-B4 96.00

Concerning LCT datasets, we review the most relevant studies Declaration of competing interest
to our work in Table 11. Overall, most of the studies use a small
dataset for their evaluation. Only Gunraj et al. [44] made use of a The authors declare that they have no known competing finan-
considerably large dataset, however the obtained performance is cial interests or personal relationships that could have appeared
much lower than our proposed approach, i.e., 95.90% compared to to influence the work reported in this paper.
99.66%. In this respect, we conclude that by combining different
deep neural networks with transfer learning strategies, we can Data availability
obtain a high prediction accuracy even on a very large dataset.
To the best of our knowledge, compared to different existing The authors do not have permission to share data.
studies [36,38], our work is the first one that deals with big
datasets. In particular, for the CXR datasets, there are 15,000 and
References
17,905 images in D1 and D2 , respectively [57]. While for the
LCT datasets, especially with D4 , there are more than 400K of [1] J.M. Connors, J.H. Levy, COVID-19 and its implications for thrombosis and
images. This suggests that with the proposed model we can work anticoagulation, Blood J. Am. Soc. Hematol. 135 (23) (2020) 2033–2040.
with a huge number of images and still obtain a good timing [2] Q. Sun, H. Qiu, M. Huang, Y. Yang, Lower mortality of COVID-19 by
efficiency. More importantly, the application of the two deep early recognition and intervention: experience from Jiangsu Province, Ann.
Intensive Care 10 (1) (2020) 1–4.
neural network families, i.e., EfficientNet and MixNet, allows us
[3] L. Iovino, P.T. Nguyen, A.D. Salle, F. Gallo, M. Flammini, Unavailable transit
to build an expert system being capable of working with images feed specification: Making it available with recurrent neural networks, IEEE
coming from different sources. Trans. Intell. Transp. Syst. 22 (4) (2021) 2111–2122, http://dx.doi.org/10.
1109/TITS.2021.3053373.
6. Conclusions [4] L.T. Duong, N.H. Le, T.B. Tran, V.M. Ngo, P.T. Nguyen, Detection of
tuberculosis from chest X-ray images: Boosting the performance with
vision transformer and transfer learning, Expert Syst. Appl. 184 (2021)
We presented a workable solution for the detection of Covid- 115519, http://dx.doi.org/10.1016/j.eswa.2021.115519, URL https://www.
19 from chest X-ray and lung computed tomography images. We sciencedirect.com/science/article/pii/S0957417421009295.
designed and implemented the models based on two building [5] I. Portugal, P. Alencar, D. Cowan, The use of machine learning algorithms
blocks: EfficientNet and MixNet as the prediction engine and in recommender systems: A systematic review, Expert Syst. Appl. 97
(2018) 205–227, http://dx.doi.org/10.1016/j.eswa.2017.12.020, URL http:
effective transfer learning algorithms. The proposed models have //www.sciencedirect.com/science/article/pii/S0957417417308333.
been studied by means of four datasets which have been widely [6] X. Mei, H.-C. Lee, K.-y. Diao, M. Huang, B. Lin, C. Liu, Z. Xie, Y. Ma, P.
used in various papers. The experimental results show that our Robson, M. Chung, A. Bernheim, V. Mani, C. Calcagno, K. Li, S. Li, H. Shan, J.
proposed approach obtains a better prediction performance com- Lv, T. Zhao, J. Xia, Y. Yang, Artificial intelligence–enabled rapid diagnosis of
patients with COVID-19, Nat. Med. (2020) 1–5, http://dx.doi.org/10.1038/
pared to some relevant previous studies. To the best of our
s41591-020-0931-3.
knowledge, our work is the first one that deals with images com- [7] M. Tan, Q. Le, EfficientNet: Rethinking model scaling for convolutional
ing from different sources. As future work, we are going to refine neural networks, in: K. Chaudhuri, R. Salakhutdinov (Eds.), Proceedings of
and evaluate the approach by taking into consideration more the 36th International Conference on Machine Learning, in: Proceedings
datasets and tuning other deep neural network configurations. of Machine Learning Research, vol. 97, PMLR, Long Beach, California, USA,
2019, pp. 6105–6114, URL http://proceedings.mlr.press/v97/tan19a.html.
[8] M. Tan, Q.V. Le, MixConv: Mixed depthwise convolutional kernels, 2019,
CRediT authorship contribution statement CoRR abs/1907.09595. arXiv:1907.09595. URL http://arxiv.org/abs/1907.
09595.
Linh T. Duong: Conceptualization, Software, Writing – original [9] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A.
draft. Phuong T. Nguyen: Conceptualization, Methodology, Writ- Karpathy, A. Khosla, M. Bernstein, A.C. Berg, L. Fei-Fei, ImageNet large scale
visual recognition challenge, Int. J. Comput. Vis. 115 (3) (2015) 211–252,
ing – original draft, Writing – review & editing. Ludovico Iovino: http://dx.doi.org/10.1007/s11263-015-0816-y.
Validation, Visualization, Writing – review & editing. Michele [10] C. Xie, M. Tan, B. Gong, J. Wang, A. Yuille, Q.V. Le, Adversarial examples
Flammini: Writing – review & editing, Supervision. improve image recognition, 2019, arXiv e-prints arXiv:1911.09665.

10
L.T. Duong, P.T. Nguyen, L. Iovino et al. Applied Soft Computing 132 (2023) 109851

[11] Q. Xie, E. Hovy, M.-T. Luong, Q.V. Le, Self-training with Noisy Student [30] X. He, S. Wang, S. Shi, X. Chu, J. Tang, X. Liu, C. Yan, J. Zhang, G. Ding,
improves ImageNet classification, 2019, cite arxiv:1911.04252. URL http: Benchmarking deep learning models and automated model design for
//arxiv.org/abs/1911.04252. COVID-19 detection with chest CT scans, MedRxiv (2020) http://dx.doi.
[12] F. Chollet, Xception: Deep learning with depthwise separable convolutions, org/10.1101/2020.06.08.20125963.
2016, cite arxiv:1610.02357. URL http://arxiv.org/abs/1610.02357. [31] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition,
in: 2016 IEEE Conference on Computer Vision and Pattern Recognition,
[13] A.G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. CVPR, 2016, pp. 770–778, http://dx.doi.org/10.1109/CVPR.2016.90.
Andreetto, H. Adam, MobileNets: Efficient convolutional neural networks [32] M. Elpeltagy, H. Sallam, Automatic prediction of COVID-19 from chest
for mobile vision applications, 2017, cite arxiv:1704.04861. URL http: images using modified ResNet50, Multimedia Tools Appl. 80 (17) (2021)
//arxiv.org/abs/1704.04861. 26451–26463, http://dx.doi.org/10.1007/s11042-021-10783-6.
[14] M. Tan, B. Chen, R. Pang, V. Vasudevan, Q.V. Le, MnasNet: Platform- [33] D. Yang, C. Martinez, L. Visuña, H. Khandhar, C. Bhatt, J. Carretero,
aware neural architecture search for mobile, 2018, CoRR abs/1807.11626. Detection and analysis of COVID-19 in medical images using deep learning
arXiv:1807.11626. URL http://arxiv.org/abs/1807.11626. techniques, 11 (1) 19638. http://dx.doi.org/10.1038/s41598-021-99015-3.
[15] H. Cai, L. Zhu, S. Han, ProxylessNAS: Direct neural architecture [34] F. Wilcoxon, Individual comparisons by ranking methods, in: S. Kotz, N.L.
search on target task and hardware, in: International Conference on Johnson (Eds.), Breakthroughs in Statistics: Methodology and Distribution,
Learning Representations, 2019, URL https://openreview.net/forum?id= Springer New York, New York, NY, 1992, pp. 196–202, http://dx.doi.org/
HylVB3AqYm. 10.1007/978-1-4612-4380-9_16.
[35] B. Ghoshal, A. Tucker, Estimating uncertainty and interpretability in deep
[16] M. Sandler, A.G. Howard, M. Zhu, A. Zhmoginov, L. Chen, Inverted residuals learning for coronavirus (COVID-19) detection, 2020, arXiv abs/2003.10769.
and linear bottlenecks: Mobile networks for classification, detection and [36] A. Abbas, M.M. Abdelsamea, M.M. Gaber, Classification of COVID-19 in
segmentation, 2018, CoRR abs/1801.04381. arXiv:1801.04381. URL http: chest X-ray images using DeTraC deep convolutional neural network, 2020,
//arxiv.org/abs/1801.04381. arXiv preprint arXiv:2003.13815.
[17] A. Kamilaris, F.X. Prenafeta-Boldú, Deep learning in agriculture: A survey, [37] A. Narin, C. Kaya, Z. Pamuk, Automatic detection of coronavirus disease
Comput. Electron. Agric. 147 (2018) 70–90, http://dx.doi.org/10.1016/j. (COVID-19) using X-ray images and deep convolutional neural networks,
compag.2018.02.016. 2020, arXiv:2003.10849.
[18] K. Weiss, T. Khoshgoftaar, D. Wang, A survey of transfer learning, J. Big [38] I.D. Apostolopoulos, T.A. Mpesiana, Covid-19: automatic detection from x-
Data 3 (2016) http://dx.doi.org/10.1186/s40537-016-0043-6. ray images utilizing transfer learning with convolutional neural networks,
Phys. Eng. Sci. Med. (2020) 1.
[19] L. Torrey, T. Walker, J. Shavlik, R. Maclin, Using advice to transfer [39] E. Luz, P.L. Silva, R. Silva, G. Moreira, Towards an efficient deep learning
knowledge acquired in one reinforcement learning task to another, in: model for covid-19 patterns detection in x-ray images, 2020, arXiv preprint
Proceedings of the 16th European Conference on Machine Learning, ECML arXiv:2004.05717.
’05, Springer-Verlag, Berlin, Heidelberg, 2005, pp. 412–424, http://dx.doi. [40] J. Zhang, Y. Xie, Y. Li, C. Shen, Y. Xia, Covid-19 screening on chest x-ray
org/10.1007/11564096_40. images using deep learning based anomaly detection, 2020, arXiv preprint
[20] Z. Huang, Z. Pan, B. Lei, Transfer learning with deep convolutional neural arXiv:2003.12338.
network for SAR target classification with limited labeled data, Remote [41] E.E.-D. Hemdan, M.A. Shouman, M.E. Karar, Covidx-net: A framework of
Sens. 9 (9) (2017) http://dx.doi.org/10.3390/rs9090907. deep learning classifiers to diagnose covid-19 in x-ray images, 2020, arXiv
[21] L.T. Duong, P.T. Nguyen, C. Di Sipio, D. Di Ruscio, Automated fruit preprint arXiv:2003.11055.
recognition using EfficientNet and MixNet, Comput. Electron. Agric. 171 [42] M. Rahimzadeh, A. Attar, S.M. Sakhaei, A fully automated deep learning-
(2020) 105326, http://dx.doi.org/10.1016/j.compag.2020.105326, URL http: based network for detecting COVID-19 from a new and large lung
//www.sciencedirect.com/science/article/pii/S0168169919319787. CT scan dataset, MedRxiv (2020) http://dx.doi.org/10.1101/2020.06.08.
20121541, arXiv:https://www.medrxiv.org/content/early/2020/09/01/2020.
[22] Z.Q.L. Linda Wang, A. Wong, COVID-Net: A tailored deep convolutional 06.08.20121541.full.pdf.
neural network design for detection of COVID-19 cases from chest [43] T. Anwar, S. Zakir, Deep learning based diagnosis of COVID-19 using chest
radiography images, 2020, arXiv:2003.09871. CT-scan images, 2020, http://dx.doi.org/10.36227/techrxiv.12328061.
[23] J.P. Cohen, P. Morrison, L. Dao, COVID-19 image data collection, 2020, [44] H. Gunraj, L. Wang, A. Wong, COVIDNet-CT: A tailored deep convolutional
arXiv:2003.11597. URL https://github.com/ieee8023/covid-chestxray- neural network design for detection of COVID-19 cases from chest CT
dataset. images, 2020, arXiv:2009.05383.
[24] E. Soares, P. Angelov, S. Biaso, M. Higa Froes, D. Kanda Abe, SARS-CoV-2 [45] A. Mobiny, P.A. Cicalese, S. Zare, P. Yuan, M. Abavisani, C.C. Wu, J. Ahuja,
CT-scan dataset: A large dataset of real patients CT scans for SARS- P.M. de Groot, H.V. Nguyen, Radiologist-level COVID-19 detection using CT
CoV-2 identification, MedRxiv (2020) http://dx.doi.org/10.1101/2020.04.24. scans with detail-oriented capsule networks, 2020, arXiv:2004.07407.
20078584, arXiv:https://www.medrxiv.org/content/early/2020/05/14/2020. [46] A.A. Ardakani, A.R. Kanafi, U.R. Acharya, N. Khadem, A. Mohammadi,
04.24.20078584.full.pdf. URL https://www.medrxiv.org/content/early/2020/ Application of deep learning technique to manage COVID-19 in routine
05/14/2020.04.24.20078584. clinical practice using CT images: Results of 10 convolutional neu-
ral networks, Comput. Biol. Med. 121 (2020) 103795, http://dx.doi.org/
[25] K. Zhang, X. Liu, J. Shen, Z. Li, Y. Sang, X. Wu, Y. Zha, W. Liang, C. Wang, K. 10.1016/j.compbiomed.2020.103795, URL http://www.sciencedirect.com/
Wang, L. Ye, M. Gao, Z. Zhou, L. Li, J. Wang, Z. Yang, H. Cai, J. Xu, L. Yang, science/article/pii/S0010482520301645.
W. Cai, W. Xu, S. Wu, W. Zhang, S. Jiang, L. Zheng, X. Zhang, L. Wang, L. [47] H.X. Bai, R. Wang, Z. Xiong, B. Hsieh, K. Chang, K. Halsey, T.M.L. Tran, J.W.
Lu, J. Li, H. Yin, W. Wang, O. Li, C. Zhang, L. Liang, T. Wu, R. Deng, K. Wei, Choi, D.-C. Wang, L.-B. Shi, J. Mei, X.-L. Jiang, I. Pan, Q.-H. Zeng, P.-F. Hu,
Y. Zhou, T. Chen, J. Lau, M. Fok, J. He, T. Lin, W. Li, G. Wang, Clinically Y.-H. Li, F.-X. Fu, R.Y. Huang, R. Sebro, Q.-Z. Yu, M.K. Atalay, W.-H. Liao,
applicable AI system for accurate diagnosis, quantitative measurements, Artificial intelligence augmentation of radiologist performance in distin-
and prognosis of COVID-19 pneumonia using computed tomography, Cell guishing COVID-19 from pneumonia of other origin at chest CT, Radiology
181 (6) (2020) http://dx.doi.org/10.1016/j.cell.2020.04.045. 296 (3) (2020) E156–E165, http://dx.doi.org/10.1148/radiol.2020201491,
[26] G. Marques, D. Agarwal, I. de la Torre Díez, Automated medical diagnosis PMID: 32339081. arXiv:https://doi.org/10.1148/radiol.2020201491.
of COVID-19 through EfficientNet convolutional neural network, Appl. Soft [48] H. Zhang, K.M. Saravanan, Y. Yang, M.T. Hossain, J. Li, X. Ren, Y. Pan, Y.
Comput. 96 (2020) 106691, http://dx.doi.org/10.1016/j.asoc.2020.106691, Wei, Deep learning based drug screening for novel coronavirus 2019-nCov,
URL http://www.sciencedirect.com/science/article/pii/S1568494620306293. Interdiscip. Sci. Comput. Life Sci. (2020) 1.
[27] A. Iqbal, M. Usman, Z. Ahmed, An efficient deep learning-based framework [49] B.R. Beck, B. Shin, Y. Choi, S. Park, K. Kang, Predicting commercially
for tuberculosis detection using chest X-ray images, Tuberculosis 136 available antiviral drugs that may act on the novel coronavirus (SARS-
(2022) 102234, http://dx.doi.org/10.1016/j.tube.2022.102234, URL https: CoV-2) through a drug-target interaction deep learning model, Comput.
//www.sciencedirect.com/science/article/pii/S1472979222000713. Struct. Biotechnol. J. (2020).
[50] L. Yan, H.-T. Zhang, Y. Xiao, M. Wang, C. Sun, J. Liang, S. Li, M. Zhang, Y.
[28] A. Saygılı, A new approach for computer-aided detection of coronavirus Guo, Y. Xiao, et al., Prediction of criticality in patients with severe Covid-19
(COVID-19) from CT and X-ray images using machine learning meth- infection using three clinical features: a machine learning-based prognostic
ods, Appl. Soft Comput. 105 (2021) 107323, http://dx.doi.org/10.1016/ model with clinical data in Wuhan, MedRxiv (2020).
j.asoc.2021.107323, URL https://www.sciencedirect.com/science/article/pii/ [51] X. Jiang, M. Coffee, A. Bari, J. Wang, X. Jiang, J. Huang, J. Shi, J. Dai, J. Cai, T.
S1568494621002465. Zhang, et al., Towards an artificial intelligence framework for data-driven
[29] T. Ozturk, M. Talo, E.A. Yildirim, U.B. Baloglu, O. Yildirim, U.R. Acharya, prediction of coronavirus clinical severity, CMC: Comput. Mater. Contin.
Automated detection of COVID-19 cases using deep neural networks with 63 (2020) 537–551.
X-ray images, Comput. Biol. Med. 121 (2020) 103792, http://dx.doi.org/ [52] T. Ozturk, M. Talo, E.A. Yildirim, U.B. Baloglu, O. Yildirim, U.R. Acharya,
10.1016/j.compbiomed.2020.103792, URL http://www.sciencedirect.com/ Automated detection of COVID-19 cases using deep neural networks with
science/article/pii/S0010482520301621. X-ray images, Comput. Biol. Med. (2020) 103792.

11
L.T. Duong, P.T. Nguyen, L. Iovino et al. Applied Soft Computing 132 (2023) 109851

[53] L.O. Hall, R. Paul, D.B. Goldgof, G.M. Goldgof, Finding covid-19 from chest [56] P. Afshar, S. Heidarian, F. Naderkhani, A. Oikonomou, K.N. Plataniotis,
x-rays using deep learning on a small dataset, 2020, arXiv preprint arXiv: A. Mohammadi, Covid-caps: A capsule network-based framework for
2004.02060. identification of covid-19 cases from x-ray images, 2020, arXiv preprint
[54] L. Wang, A. Wong, COVID-net: A tailored deep convolutional neural arXiv:2004.02696.
network design for detection of COVID-19 cases from chest X-Ray images, [57] L.T. Duong, P.T. Nguyen, L. Iovino, M. Flammini, Deep learning for
2020, arXiv preprint arXiv:2003.09871. automated recognition of Covid-19 from chest X-ray images, MedRxiv
[55] L. Li, L. Qin, Z. Xu, Y. Yin, X. Wang, B. Kong, J. Bai, Y. Lu, Z. Fang, Q. Song, et (2020) http://dx.doi.org/10.1101/2020.08.13.20173997, arXiv:https://www.
al., Artificial intelligence distinguishes COVID-19 from community acquired medrxiv.org/content/early/2020/08/14/2020.08.13.20173997.full.pdf. URL
pneumonia on chest CT, Radiology (2020) 200905. https://www.medrxiv.org/content/early/2020/08/14/2020.08.13.20173997.

12