Received April 23, 2020, accepted May 2, 2020, date of publication May 12, 2020, date of current version

May 27, 2020.

Digital Object Identifier 10.1109/ACCESS.2020.2994222

Evaluating Trust Prediction and Confusion Matrix

Measures for Web Services Ranking
MUHAMMAD HASNAIN 1 , MUHAMMAD FERMI PASHA 1 , (Member, IEEE), IMRAN GHANI2 ,
MUHAMMAD IMRAN 3 , MOHAMMED Y. ALZAHRANI 4 , AND RAHMAT BUDIARTO 5
1 School of IT, Monash University Malaysia, Subang Jaya 47500, Malaysia
2 Department of Mathematics and Computer Sciences, Indiana University of Pennsylvania, Indiana, PA 15705, USA
3 Next Bridge (Pvt.) Ltd., Lahore 54000, Pakistan
4 Information Technology Department, College of Computer Science and IT, Albaha University, Al Bahah 65527, Saudi Arabia
5 Computer Engineering and Science Department, College of Computer Science and IT, Albaha University, Al Bahah 65527, Saudi Arabia

Corresponding author: Muhammad Hasnain ([email protected])

ABSTRACT To accurately rank various web services can be a very challenging task depending on the
evaluation criteria used, however, it can play an important role in performing a better selection of web services
afterward. This paper proposes an approach to evaluate trust prediction and confusion matrix to rank web
services from throughput and response time. AdaBoostM1 and J48 classifiers are used as binary classifiers
on a benchmark web services dataset. The trust score (TS) measuring method is proposed by using the
confusion matrix to determine trust scores of all web services. Trust prediction is calculated using 5-Fold,
10-Fold, and 15-Fold cross-validation methods. The reported results showed that the web service 1 (WS1)
was most trusted with (48.5294%) TS value, and web service 2 (WS2) was least trusted with (24.0196%) TS
value by users. Correct prediction of trusted and untrusted users in web services invocation has improved the
overall selection process in a pool of similar web services. Kappa statistics values are used for the evaluation
of the proposed approach and for performance comparison of the two above-mentioned classifiers.

INDEX TERMS Web services, trust prediction, web services selection, binary classification, fuzzy rules,
confusion matrix.

I. INTRODUCTION may affect the results of a subjective category of techniques.

Trustworthiness of web services has a significant role in Subsequently, the hybrid of objective and subjective cate-
the ranking of web services. Web services can be ranked gories of technique can be useful to overcome the limitations
based on the requesters’ demands [1]. For example, two web of the existing techniques. Our proposed fuzzy-based users’
services have similar functionality, one web service is more trust prediction approach involves the end-users values of
used as compared to another one; then possibly the selected quality attributes and then rank web services by calculating
web service is usually a more trusted. Web services selection the trust score of web services.
and ranking is a problem that can be addressed through the The confusion matrix is widely used in machine learning
classification mechanism of non-functional quality attributes. for supervised classification or determination of the behavior
Quality attributes such as response time, throughput, avail- of classification models [3]. The square structure of a confu-
ability, and security have different weights, which are prin- sion matrix is represented through rows and columns, where
cipal for ranking of web services [2]. In the latter study, rows are the actual classes of the instances, and columns
three categories of web services ranking techniques objective, are the predicted classes [4]. For the binary classification,
subjective and hybrid were discussed. Expert opinion is not a confusion matrix is represented as 2 ∗ 2 matrix. For a
considered in the objective category of ranking approaches. confusion matrix, four measures, namely, ’true positive’ (TP),
On the other hand, the subjective category considers expert ’true negative’ (TN), ’false positive’ (FP), and ’false negative’
opinion or subjective judgment. However, lack of experience (FN), have been reported. For the multiclass problem, a con-
fusion matrix with the k class has a k ∗k confusion matrix [5].
The associate editor coordinating the review of this manuscript and Confusion matrix is applied to evaluate the performance
approving it for publication was Zhangbing Zhou . of classifiers on datasets. Font et al. [6] used the confusion

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
VOLUME 8, 2020 90847
 M. Hasnain et al.: Evaluating Trust Prediction and Confusion Matrix Measures for Web Services Ranking

matrix to distinguish the predicted values and real values because web services users mostly expect low response time
of model elements in software engineering. Four confusion and high throughput from service providers [67]. Therefore,
matrix measures such as TP, FP, TN, and FN were used for the trustworthiness of a web service is more relevant to the
the classification of faulty and non-faulty classes of Java performance evaluation of a web service, which is derived by
programs. using QoS attributes.
Although multiple classification has an extensive The proposed approach exploits the values of both
background, but studies with regards to multiple web ser- monitored QoS metrics and mentioned in the ’service level
vices instances classification are relatively scarce [50], [51]. agreement’ SLA document [39]. Untrustworthy users were
Existing studies on the multiple classifications show that identified with the assumption that the majority of users were
classifiers used for multiple classification are relatively low honest as their majority opinions were consistent. In contrast,
in performance accuracy. Due to this reason, we observed dishonest users provided a low rating without any consistency
the restricted applications of existing multiple classifiers. in their opinions. This assumption can be further discussed in
We have found in [51] that multiple classification models future research works because no web services QoS metric
do not outperform the single classifier. The authors in the has been used for the evaluation of the proposed approach.
lateral mentioned study proved their claim by using statistical Trust is defined in different contexts. Trust on eBay, and
analysis of multiple classification and binary classification. Amazon has been measured by using the users’ past inter-
Both standard deviation and coefficient of variations for action because trust is relational [8]. For instance, two users
testing multiple classifiers remained higher for the single of web services interact with each other, as a result of the
classifiers. Based on the findings of these studies, we can interaction, their relationship strengthens, and trust evolves
reveal that still, the classifiers can handle efficiently the from their mutual exchange. In addition to it, trustworthy and
binary classification problem rather than handling the issue reputed web services have been defined as services which are
of multiple classifications of web services instances. inherently secure, reliable, and available despite disruption
The concept of trust prediction for web services is not from the environment, and human errors [41]. The author
new in the research domain of web services and estimation points out the requirements of secure web services that ensure
of ’quality of services’ (QoS). Su et al. [7], proposed a the users’ trust in web services. A trusted web service is
trust–awareness approach for the prediction of reliable and reliable as well as provides high throughput and low response
personalized QoS features. Users’ reputation was determined time [42].
by clustering the information obtained from similar users Suppose a web service consumer asks for the best services
to identify the clusters of users and invoked web services. that meet requirements Re as (r1, r2, r3,... rn). Standard
Web service trustworthiness is dependent on users, and it attributes such as response time, cost, and availability along
may be maintained in the inappropriate clusters. As a result, their levels are well defined in the SLA document by ser-
this inappropriate clustering affects the trustworthiness of vice providers. Trust reputation model proposed in [43] is
certain web services. To address this issue, we propose an evaluated on the latter mentioned three quality attributes
approach with the use of a confusion matrix measures. Our where they find that web services consumers are more inter-
focus is on the binary classification of web services from ested in completing their transactions with the low response
invoked web services by using the obtained feedback in time rather than focusing on the high availability, and cost
terms of the throughput and response time metrics values. attributes. It means, a web service consumer is more oriented
We measure users’ trust from the performance evaluation of towards the short response time to complete transactions
quality metrics. Both, response time and throughput come and shows his trust as feedback. Web services consumers
under performance category of quality metrics. rate their invoked web services differently in terms of QoS
The well-known fact regarding the performance of web properties. For instance, users a and b rate high throughput,
service is its reflection from functional and non-functional and low response time, while another user ’c’ rates the same
quality attributes. Response time and throughput are two vital services with the low performance (throughput) and high
considered attributes in studies [64]. QoS based ranking of response time. Subjective perception of QoS attributes may
web services is appropriate, employing the quality attributes cause the differences in rating by users [44]. As users, a, and
as mentioned earlier. Moreover, Mao et al. [65] considered b may think that it is good if their invoked service responds
throughput and response time as quality attributes to conduct within one second. On the other hand, user c may have not a
the experiments for QoS based ranking of web services. high requirement, and he would like those web services which
Evaluation of the most web services ranking approaches is respond within 20 seconds. We can specify that users a, b, and
performed on the real-world dataset that is composed of c have provided their trust values by differently rating web
two QoS attributes (throughput and response time) [64]. services.
Somu et al. [66] also performed trust centric ranking of web Web services selection approach proposed in [40] is aimed
services by using the throughput and response time quality to evaluate the security as a big challenge of web services.
attributes. Based on the existing literature and understanding Researchers mentioned that the security of web service is
of the QoS attributes, it is appropriate to use throughput further related to confidentiality and privacy aspects. This
and response time as the most popular quality attributes is because a web service is more reliable than another web

90848 VOLUME 8, 2020

M. Hasnain et al.: Evaluating Trust Prediction and Confusion Matrix Measures for Web Services Ranking

service in confidentiality, and the same web service may be a hybrid approach of classifiers to address the issue of high
weaker in security in comparison with another web service. percentages of the FP instances. Along with the proposed
Therefore, web services users find it hard to resolve the hybrid approach, feature selection and reduction are required
selection and ranking of web services as they lack expertise. to find the maximum number of attacks on a network system.
Other than confidentiality and privacy features can be used to Al-Obeidat and El-Alfy [12] proposed an approach to
address the security issue of web services that, in turn, helps address the space issue between yes and no in binary clas-
in determining the trust of users in web services. sification. Their decision tree generates rules, which have
Trust prediction of web services can be approached as a incredibly crisp intervals, and using the fuzzy membership
ranking problem. Ranking encompasses several issues, such to an object of a class can address marginal space issues
as selection, recommendation, and testing of web services. between yes and no. The main objective behind the proposal
The main objective of trust prediction is to calculate the of the hybrid approach is to classify internet traffic through
users’ trust in the invoked web services. Then, the calculated classification and interpretation.
users’ trust is used to rank web services from a pool of web In the literature, the trust prediction of web services is
services accessed by the same users of various regions. Our presented in various means and names. Ding et al. [13],
ultimate goal of ranking is to identify the web services with combined QoS prediction and estimation of customer satis-
the high trust score of users and prioritize them for better faction in their proposed approach known as CSTrust, which
future selection of web services. is used to release the customer satisfaction information on
Contributions of this paper are as follows: web services. The main difference between CSTrust approach
• This paper proposes a trust prediction binary classifica- and our proposed approach is that the CSTrust evaluates
tion approach by using QoS attributes of web services. the cloud web services. In contrast, our proposed approach
• This paper proposes fuzzy rules to provide ground truth focuses on web services that use open standards, such as
for training and evaluation of binary classifiers. ’extensible markup language’ (XML), ’web services descrip-
• This paper proposes an application of the confusion tion language’ (WSDL), and ’simple object access protocol’
matrix measures to evaluate the ranking of web services. (SOAP).
In the remainder of this paper, section 2 presents the relevant To manage the ‘‘system-level agreement’ (SLA), QoS pre-
literature on the existing trust and confusion matrix topics. diction is a significant tool. To know the behavior of services
Section 3 presents the proposed approach; section 4 presents consumers, Hussain et al. [14] compared the results of ML
results and discussion; section 5 presents the impact of dataset approaches to time series approaches. With the objectives of
size on the trust prediction precision; section 6 presents knowing the services violation and avoidance of penalties,
threats to validity; section 7 concludes the proposed work service providers could benefit from the ML-based QoS pre-
along with future research implications. diction. Somu et al. [15] proposed the web services’ ranking
algorithm to identify the most trustworthy web services. The
II. LITERATURE REVIEW proposed approach employed hyper-graph partitioning and
In this section, we present a review of the existing primary time-varying mapping method to identify the similar services
studies on the classification with regards to the confusion providers. Moreover, the use of ‘‘hyper-graph-binary fruit
matrix. We also discuss a few significant approaches pro- fly algorithm’’ (HBFFOA), which employs hypergraph par-
posed for QoS prediction in the literature. titioning, and time-varying function for the identification of
Polat et al. [9] used four measures, namely, TP, TN, FP, and similar services, helped in determining the optimal ranking
FN, of the confusion matrix, to determine whether patients of web services.
have optic nerve disease or not. These researchers exploited Trust assessment of web services through fuzzy-based
TN for patients with optic nerve and TP for healthy indi- credibility was undertaken by Saoud et al. [16], and they
viduals, reported the results with the confusion matrix, and pointed out the limitations of those trust-based web ser-
used TP and TN to classify individuals as either diseased vices selection approaches that involved the end-users rating.
individuals or healthy individuals. For binary classification, Uncertainty and bias were the concerns of researchers that
the use of TP and TN can accurately predict instances. Choud- affected the end-users ratings for web services. A fuzzy-
hury and Bhowal [10] used confusion matrix measures to based model was proposed to address the uncertainty and
predict the true and false instances of the network attacks. biases of end-users’ ratings of web services. The proposed
Binary classifiers were used to represent the attacked and trust-approach was evaluated on a number of experiments.
normal classes for network intrusion detection. Based on the Results indicated that the proposed approach improved trust
confusion matrix measures, these researchers developed a quality and robustness.
’false positive rate’, (FPR), ’false discovery rate’, (FDR), To address the problem of the accurate prediction of
and ’negative prediction rate’ (NPR) measures. To predict the unknown QoS values, Ma et al. [17] proposed the collab-
possibility of the accurate and inaccurate classification of net- orative filtering that outperformed the existing approaches
work attack and normal instances, three developed measures in the accurate prediction of missing values. The main dif-
were used for accuracy metrics. To increase the accuracy of ference between the collaborative filtering approach and our
the anomaly detection system, Aljawarneha et al. [11] used proposed approach is that the former considers the missing

VOLUME 8, 2020 90849

 M. Hasnain et al.: Evaluating Trust Prediction and Confusion Matrix Measures for Web Services Ranking

values, while the latter uses throughput and response time latter study is investigating both trustworthiness and
values as feedback given by users. None of the two studies QoS prediction.
mentioned above show trust prediction via classification. In a recently published work, Tibermacine et al. [47]
Therefore, the proposal of our approach is mainly based proposed a method to determine the reputation of similar
on the binary classification along with the confusion matrix web services. Researchers have employed the application of
and k-fold cross-validation (CV) method, which divides data support vector regression algorithm to estimate the unknown
points into a fixed number of folds of the data. K-fold cross- QoS values of web services from their known values. The
validation ensures that data in each fold is at least once tested. proposed reputation estimation method has been evaluated
Wang et al. [45] proposed a trustworthiness of the web on two web services QoS datasets. The proposed reputation
services selection approach that involved collaboration rep- estimation method is mainly focused on determining the
utation in social networks. This study offers a web services reputation of newcomer web services because reputation is
selection process with the aims of including and excluding a similar issue to trust, and security issues of web services
the services with the high and low reputations, respectively. deployment. Therefore, the trust and security of web services
The reputation of a web service is increased due to more can be undertaken in future works to ensure the quality of
interaction rounds of web services. This study also defines the web services because users show their high confidence in the
reliability levels of web services which are given as follows: high-quality web services as compared to low-quality web
• Level 1: Good web service (0.9-1) services.
• Level 2: Normal web service (0.3-0.9) Mao et al. [52] pointed out that trustworthiness was a
• Level 3: Bad web service (0.0-0.3) significant indicator for the selection and recommendation
The findings of this study indicated that web services’ rep- of services. Trust prediction based on the QoS value is a
utation was fairly computed that distinguished web services challenging task due to a non-linear association between
from the selection process by using the defined three levels. QoS values and the trust rate of services. Although neu-
This study showed the scalability issue because the proposed ral networks (NNs) have the capability of trust prediction,
approach was less effective or even was unable to work but their parameters’ setting further requires research work
on a small community of web services. Therefore, a web to improve their performance. Therefore, researchers in the
service ranking approach is required to address the scalability lateral-mentioned study introduced particle swarm optimiza-
issue within the small community of web services. A ranking tion (PSO) to enhance the environment of NNs to trust pre-
approach can be proposed to rank web services for a small diction of cloud web services accurately. For the evaluation
number of web services. of the PSO supported trust prediction, experiments were per-
Mehdi et al. [46] proposed a trust and reputation-based web formed on the public QoS dataset. The results showed that
services selection approach. This proposed approach used NNs with PSO outperformed the basic NNs in the trust-based
correlation information among various QoS metrics, which classification of web services.
resulted in estimating the trustworthiness of web services. Somu et al. [53] called that trustworthiness was itself a
Researchers exploited two statistical distributions known as quality metric used for the assessment of the quality of web
Dirichlet and generalized Dirichlet, which represented the services. It has been earlier mentioned in [52] that the trust
multiple correlated metrics. For instance, the throughput QoS prediction from QoS attributes is a challenging task. To over-
metric is correlated with response time and availability of come this problem, a multi-level ‘Hypergraph Coarsening
QoS metrics. It has been stated that the increase in through- based Robust Heteroscedastic Probabilistic Neural Network’
put value results in increasing the availability score of web (HC-RHRPNN) was proposed in [53] for trust prediction
services and decreasing the response time value of users’ of cloud web services. Informative samples were identified
requests. Reliability as a QoS metric has a strong correlation by employing the hypergraph coarsening of HC-RHRPNN.
with response time and throughput QoS metrics. Afterward, the training of the proposed model was done by
Moreover, the lateral mentioned study proposed the aggre- using the identified informative samples. Moreover, infor-
gate reputation feedback algorithm to deal with the mali- mative samples improved the prediction accuracy and also
cious feedback, which propagates between interacting web minimized the execution time. The proposed HC-RHRPNN
services. Results endorsed that the proposed approach, along outperformed the earlier proposed neural networks with
with the algorithm is capable of showing the better determi- regards to performance. We have observed an extension in the
nation of trustworthiness in comparison with the state of the latter work in [54] in which researchers used artificial neural
art approaches and algorithms. Before the latter mentioned networks (ANNs). The PSO technique was applied to train
research work, Deng et al. [44] proposed a CTrust framework the ANN. Moreover, ‘binary particle swarm optimization’
to evaluate the trustworthiness of cloud web services by (BPSO) has been used for the selection of quality attributes.
combining the customers’ satisfaction estimation and QoS The evaluation of the proposed approach was performed by
prediction. Both of these research studies were aimed at using a public QoS dataset. The results showed that the
addressing the trustworthiness issue of web services. How- prediction accuracy of the proposed model remained better
ever, the previous research explores QoS metrics for the than the existing models. Further works require to improve
estimation of trustworthiness web services, while the the trust prediction accuracy of the chosen models.

90850 VOLUME 8, 2020

M. Hasnain et al.: Evaluating Trust Prediction and Confusion Matrix Measures for Web Services Ranking

FIGURE 1. Phases of the proposed approach.

→ Researchers in studies [53], [54] studied the trust where xi denotes the value of a quality attribute, and max(x),
prediction of web services regarding the evaluation of users’ and min(x) denote the maximum and minimum values on all
feedback in terms of quality attributes. Contrary to these value of the given quality attributes. Normalized data were
studies, Nivethitha et al. [55] highlighted the issue of the stored as.csv in Excel files, which were subsequently used
selection of trustworthy web cloud services providers (CSPs). for binary classification of web service instances.
This problem arises due to the varying functional and Several normalization methods have been proposed in the
non-functional requirements of web services users. Also, literature. The most popular methods include min-max and
the complexity in the selection of cloud web services is z-score normalization, as discussed in [75]. The first method
increased due to the addition of new web services. To eval- as min-max is used to normalize the features in the range
uate the quality of CSPs, proposed ‘rough set theory-based [0, 1], as shown in Eq. (1). The min-max normalization
hypergraph-binary fruit fly optimization’ (RST-HGBFFO), method helps to preserve the association among the ordinal
a bio-inspired approach was used to select the most optimal input data [76]. Normalization methods based on mean and
trust measure parameters (TMPs). standard deviations of the data do not show consistent per-
formance because values of these measures vary over time.
III. PROPOSED TRUST PREDICTION APPROACH Since the values of both attributes (throughput and response
This section presents the strategy alongside four phases time) are based on historical information and do not change
involved in proposing the trust prediction of web service with time, so the use of min-max normalization is more
users. We present the structure of the proposed approach and appropriate in this study.
then discuss the main phases used for our proposed approach.
The proposed phases of the trust prediction approach are B. FUZZY RULES
shown in Fig. 1. The phases of the proposed approach are In the second phase, the feedback input is given, and every
discussed in the following subsections. input (throughput + response time) is matched to every fuzzy
rule given in the following. Every combined input data from
A. DATA PREPROCESSING TP and RT is processed according to the membership func-
To improve the accuracy of binary classifiers on the tion. Six fuzzy rules are constructed to handle the binary
numerical dataset, we preprocessed the data of chosen web classification of web services instances. A change in the num-
services. This phase of the proposed approach involved ber of quality metrics can be enforced manually by updating
the pre-processing of web services data obtained from the the fuzzy rules. The association between quality metrics and
GitHub WS-Dream data repository. To normalize the data, fuzzy rules can be adjusted by adding new fuzzy rules.
we used the min maxim normalization approach shown in the To convert the crisp input values of response time and
following Eq. (1). throughput metrics, we have proposed a fuzzy system that is
based on three main steps, namely, fuzzification, inference,
xi−min(x) and defuzzification. The first step as fuzzification decom-
Zi = (1) poses the input and output into one or more than one fuzzy set.
max(x) − min(x)
VOLUME 8, 2020 90851
 M. Hasnain et al.: Evaluating Trust Prediction and Confusion Matrix Measures for Web Services Ranking

In the inference step, our proposed IF-THEN rules are used characterize a linguistic variable as follows:
to compute the fuzzy output from fuzzy input, as given in the
following. In the defuzzification step, crisp value is obtained (n, T(n), X, G, M)
from the conversion of fuzzy values by using the membership
where n expresses the name of a variable, T(n) represents the
function. We propose to use the Sugeno Fuzzy Model [18]
term set of n, and it is the set of names of linguistic values
for the identification of non-linear relationships; between two
of n, and each value is defined as a fuzzy variable on X.
variables (response time and throughput).
Moreover, G is called as a syntactic rule to generate the name
Due to a complete set of fuzzy rules, inconsistencies
of values with regards to n; and M represents the semantic rule
among fuzzy rules decrease. With the increased number of the
used to associate each value with its meaning. Also, n being
exponential rules and linguistic variables and labels, domain
a particular, which is produced by G, is known as a term.
experts need to aware of the differences between rules and
Definition: If the trust of a user in web services is repre-
variation demonstrated in output. Therefore, the proposal of
sented by a linguistic variable, then a term set Ta can be of
rules relevant to the problem at hand and which are more intu-
the following form:
itive to a domain expert are generated [77]. Thus, high-level
rules based on the ‘IF-THEN’ expression are preferred over Ta = Very high, high, medium, low, very low
the complex statements. The fuzzy If-Then rules are proposed
to represent the relationship between variables. These rules Each linguistic term above given is associated with the
occupy a form such as ‘‘If antecedent proposition Then the fuzzy set defined on the domain [1, 0]. Very high can be
consequent proposition.’’ A linguistic model is capable of associated with near to 1, and very low can be linked near
capturing the qualitative as well as high uncertain knowledge to 0; high can be linked to 0.8; medium can be linked to 0.6,
by using the If-Then rules as follows: and low can be linked to 0.4.
A fuzzy rule is the combination of linguistic state-
R: If x is Ai then y is Bi ments, which are used for decision making in assigning
inputs or outputs with regards to classification. Hence, this
In terms of classification, response time and throughput decision-making through linguistic statements is known as
instances can be naturally considered fuzzy. Thus their behav- knowledge engineering. A fuzzy rule follows the structure as
ior is not clear cut, especially when different users report the providing input for classification and then making decisions
varying values of instances of both metrics. Before our study, for an output. The fuzzy rule is constructed from various
Liu et al. [48] proposed fuzzy rules to train the classifiers sources, such as the opinion of domain experts, knowledge
on the text data. Ambiguous and unclear speeches cannot engineering, and historical data analysis [19]. We proposed
be easily classified, and hence, fuzzy rules can solve the the use of combined information from existing literature
classification of complex instances into one or more than one and knowledge engineering for the construction of fuzzy
class. The latter-mentioned fuzzy-based study inspired us to rules [20]. Hence, we used the fuzzy information in the exist-
propose a method to make more transparent the instances of ing studies [21]. We used ’AND’ and ’OR’ logical operators
both metrics to which category they belong and then train and to express the rules for the classification of web service
evaluate the classifiers on those instances of web services. instances. For rule construction, we maintained the values
Additionally, a set of fuzzy rules can be designed to decide between 0 and 1. We used data discretization to maintain TP
the complicated classification of instances of web services. and RT values at equal intervals. We proposed to construct six
A simple heuristic rule helps reduce time consumption on rules and maintain values in five intervals. We constructed
the training of models and computation complexity [49]. fuzzy rules with the help of logical operators, which have
However, such types of proposed methods rely on manual been used in the reference [22] to address the binary classi-
observation with regards to the construction of fuzzy rules. fication problem. We presented the construction of six fuzzy
We have used a limited number of linguistic terms trans- rules, as follows.
lation for supporting the binary classification of web service
instances and handle the trust-based ranking of web services. 1) RULE 1
These linguistic terms have been extracted from the prior If the throughput value is very high OR the response time is
knowledge as well as expert experience [68]. We keep linguis- very low, then a user is trusted on certain web service. OR
tic terms translation small due to limited existing knowledge "If TP≤1.0 and >0.8 OR RT>0 and ≤ 0.20 then a user is
and experts’ expertise. trusted."
A natural way to express numerical values is through → We assign a member function value to each part of the
the use of linguistic phrases. It is easier to say, very high, statement above. The statement above indicates that inputs 1
high, medium, low, and very low, rather than providing the (throughput) and 2 (response time) as the feedback from a
numerical values. As in our case, web services instances have user. Output 1 (user’s trust) results from two inputs such as
numerical quantities. The concept of fuzzy sets introduced by throughput and response time. The use of the membership
Zadeh [69] provides a suitable way to express the imprecise function to determine the ’very high’ and ’very low’ values is
statements. A quintuple proposed in [70] has been used to known as fuzzification.

90852 VOLUME 8, 2020

M. Hasnain et al.: Evaluating Trust Prediction and Confusion Matrix Measures for Web Services Ranking

2) RULE 2 0.00, and the upper bound is 0.20. Similarly, linguistic terms
If the throughput value is high OR the response time value is in other rules obtain weights by the increasing linear function.
low, then a user is relatively trusted on certain web services. Fuzzy values approaching the upper bounds or lower bounds
OR have more uncertainties than the centroid.
If TP>0.6 and ≤0.8 OR RT>0.20 and ≤0.40>, then a user Conformance checking is aimed at establishing if a system
is trusted. externally observed presents the satisfaction and fulfills some
→ For rule 2, we used OR operator to mention that either expectations. Therefore, the conformation notion directly
the TP value was high or RT value was low; then, a user is relates to the notion of expectations. Conformance measure
trusted in a web service. is widely applied to different challenges, i.e., instance march-
ing. The formal definition of conformance outlines the prox-
3) RULE 3 imity between linguistic terms. We adopt the proposed fuzzy
If the throughput value AND the response time value are functional dependencies in [71] and highlight the possibility
medium, then a user is untrusted. OR "If TP>0.4 and to determine the conformance of attribute domain, such as
≤0.6 AND RT>0.40 and ≤0.60, then a user is untrusted" (very high, high, medium, low, and very low). More precisely,
the conformance checking of rules provides an effective
4) RULE 4 manipulation of linguistic terms to define data dependencies,
If the throughput value is low AND the response time value which are not adequately measured.
is high, then a user is untrusted. OR We define the attribute of distance (S-distance) to illus-
"If TP>0.2 and ≤0.4 AND RT>0.6 and ≤0.80, then a user trate the proximity relation. This attribute can express the
is untrusted" distance between two points. Furthermore, it can be fuzzi-
fied into a number of fuzzy sets. For instance, in our case,
5) RULE 5 we define five sets of linguistic terms, as shown in Table 2.
If the throughput value is very low AND the response time The proximity depends upon the expert or a user opinion.
value is very high, then a user is untrusted. OR As shown in Table 2, that ‘very high’ and ‘high’ values of
"If TP>0.0 and ≤0.2 AND RT>0.8 and ≤1.00, then a user throughput attributes are close to each other as compared to
is untrusted" the rest of the sets. For response time attribute very low’ and
‘low’ values of sets are close to each other in comparison
6) RULE 6 with the rest of the sets. Based on the defined conformance
If the throughput value is medium AND the response time principle by Sözat and Yazici [72], we present the proximity
value is high, then a user is untrusted. OR "If TP>0.4 and relation in Table 2. Conformance is also aimed to preserve
≤0.6 AND RT>0.6 and ≤0.80, then a user is untrusted" the interpretability when using the granules with the variable
Prior to binary classification phase, we need to translate granularity.
the linguistic terms into a decision group to align the setup
TABLE 2. Proximity Relation.
for binary classification. As shown in Table 1, we translated
the linguistic terms, such as very high, high, medium, low,
and very low, into two groups. Both very high and high
linguistic terms are maintained in the same group called
c1, and the remaining three linguistic terms, namely, as a
medium, low, and very low, were kept in the second group
named c0.

TABLE 1. Linguistic terms translation.

C. BINARY CLASSIFICATION
In the third phase of the proposed approach, the binary
classification of web services is performed to classify the
web services instances. To create a classification model of
web services instances, two top techniques AdaBoostM1 and
J48, are implemented. Both classifiers are trained on web
services datasets and processed for binary classification.
AdaBoostM1 classifier as a boosting algorithm is chosen
Fuzzy intervals were fixed at discrete values. In rule 1, due to its high accuracy in results. The boosting technique
the fuzzy value for TP input is set between 0.8 and 1.0, so the constructs the robust classification model by focusing on the
lower bound is 0.8, and the upper bound is 1.0. Similarly, misclassified records of past models [23]. AdaBoostM1 tech-
terms in other rules obtain weights by decreasing linear func- nique gives value to every record or instance. Subsequently,
tion. On the other hand, RT fuzzy value for the linguistic term the weight at first is set to 1/n and refreshed on each cycle
in rule 1 is fixed between 0.00 and 0.20, so the lower bound is of technique. The mix of two distinct sorts (boosting, and

VOLUME 8, 2020 90853

 M. Hasnain et al.: Evaluating Trust Prediction and Confusion Matrix Measures for Web Services Ranking

decision tree) of techniques is aimed at decreasing the 2) CROSS VALIDATION METHOD

changes in a robust model [24]. Therefore, the selection of We used the k-fold CV method to evaluate the proposed
boosting and decision tree techniques enhance the robustness approach. The CV was used to select a model in practicing
and prediction power aspects. the learning problem in n iterations [28]. Three k folds, such
as 5, 10, and 15 folds were focused in our chosen CV meth-
1) AdaBoostM1 ods. One of the advantages of using many k-fold methods
AdaBoostM1 is one of the most well-known classifiers of in our experiments is to avoid the biases and overfitting
the boosting family implemented in WEKA. This classifier issues. CV minimizes the generalization of errors. For the
works on the sequential training of models, and each round former issue, the CV method fits and evaluates the model
has a trained model. The algorithm of AdaBoostM1 classi- on separate datasets to ensure that performance evaluation is
fier is shown in the following. Misclassified instances are unbiased [29]. For the five-fold CV, data are randomly split
identified at the end of each round, and they are consid- into the k number of subsets. K-1 is used for training, and
ered in a new training set that is processed in training a the remaining subset is used for testing [30]. This process
new model [25]. We are dealing with the binary classifica- continued until all samples were tested. Similarly, 10-fold and
tion in this paper. Therefore, in our experiments, we con- 15-fold CVs were used to train and test the subsets.
sidered binary features, that is, c1 versus c0 classification.
Cortes et al. [26], also emphasized using AdaBoostM1 for D. TRUST PREDICTION
binary classification. In our experiments, we used J48 to To calculate the classification accuracy of classifiers,
evaluate the results from AdaBoostM1 on the web service the accuracy metric has been primarily used in stud-
datasets. The original algorithm of AdaBoostM1 is given by ies [73], [74]. The accuracy metric involves the confusion
Chen and Pan [27], as follows: matrix measures, as shown in the following Eq. (2) and
Eq. (3).
Algorithm 1 AdaBoost.M1
Total no. of corrected predictions
1) Initialize the observation weights wi = 1/N , i = Accuracy = (2)
1, 2, . . . , N Total no. of predictions
2) For m = 1 to M : OR
a) Fit a classifier Gm (x) to the training data using TP + TN
Accuracy = (3)
weights wi . TP + TN + FP + FN
b) Compute → Based on the Eq. (3), we have proposed to use the con-
PN fusion matrix measures in a similar fashion to calculate the
wi I (yi 6 = Gm (xi ))
errm = i=1 PN . trust score (TS) in this study.
i=1 wi Using the confusion matrix, we proposed to determine the
c) Compute αm = log((1 − errm )/errm ). "trust score" (TS), which is measured in a percentage score,
d) Set wi ← wi · exp[αm · I (yi 6 = Gm (xi ))], i = as shown in Eq. (4). The TS prediction denoted the accurate
1, 2, . . . , N . classification of trusted instances resulting from web service
3) Output G(x) = sgn M m=1 αm Gm (x) .
P 
invocations, and then we determine the rank of individual web
service from classification results:
AdaBoostM1 classifier generates a strong classifier from TP
TS% = ∗ 100 (4)
the set of weak classifiers. Because of iterations, and each TP + FP + TN + FN
sample which is not correctly weighted is considered for Eq. (4) shows the TS percentage of instances from invoked
the next iteration. Both J48 and AdaBoostM1, as super- web services.
vised binary classifiers, show a better classification per- Similar to that in the study of Silva-Palacios et al. [31],
formance on the different and multidimensional datasets we derived a relationship between classes from the confusion
in comparison with the other conventional classifiers. In a matrix. The simple interpretation of a confusion matrix was
recently published work, Rhmann et al. [61] stated that the that how a classifier finds it hard to distinguish between the
J48 classifier outperformed the other classifiers in fault pre- classes. Instead of using directly the four measures of a con-
diction. It is proven that both AdaBoostM1 and J48 classi- fusion matrix, we used correctly predicted instances of con-
fiers have a higher prediction accuracy on datasets in com- fusion matrix measures to obtain the maximum information
parison with the rest of the techniques. The selection of on the trusted instances of web services. As shown in Eq. (4),
two classifiers is due to their advantages: AdaBoostM1 is TS percentage was analogous to the accurate prediction of
the productive classification technique with its boosting fea- trusted instances.
tures and its enhanced characterization rate [62]. On the
other hand, J48 construction is based on the simple graphi- IV. RESULTS AND DISCUSSION
cal representation structure for the classification and higher This section presents the evaluation of the proposed
prediction [63]. trust-based ranking approach. We performed experiments on

90854 VOLUME 8, 2020

M. Hasnain et al.: Evaluating Trust Prediction and Confusion Matrix Measures for Web Services Ranking

TABLE 3. WSDL URL of Web Services.

a real-world dataset. Moreover, we report the results and regard to the web service datasets. Kappa statistics were used
findings of the confusion matrix’ evaluation and the proposed to test the inter-rater reliability or agreement between the
trust score (TS) method. predicted and actual instances of web services. The Kappa
statistics value varied between 0 and 1. Kappa statistics
A. DATASET value of <0.4 showed an extremely low similarity; the value
We used the quality WS-Dream dataset to evaluate the perfor- between 0.4 and 0.55 was acceptable; the value between
mance of our proposed approach. This dataset was published 0.55 and 0.70 indicated a good similarity; the value between
by Zheng et al. [34] with the help of the Planet Lab platform, 0.70 and 0.85 indicated an extremely high similarity, and the
consisted of the invocation records of 339 users and 5825 web value of >0.85 showed a perfect matching between predicted
services and is accessible from GitHub and The Chinese and actual web service instances.
University of Hong Kong websites [32], [33]. We have chosen We can see in Table 4 that AdaBoostM1 classifier
five web services randomly because we plan to include more outperformed the J48 in the case of the WS1 dataset.
web services in our future work. Every web service has The values of Kappa statistics along with the Precision,
metadata information along with response time (RT) matrix, Recall, and F-Measure accuracy metrics were better for
and throughput (TP) matrix, which are denoted as rtmatrix, AdaBoostM1 than the J48 classifier. For the WS2 dataset,
and tpmatrix, respectively. This dataset is the collection of the AdaBoostM1 classifier showed a higher accuracy at
real-world web services’ QoS metrics values by users. 10 k-fold compared to the accuracy values achieved by
Our preliminary experiments were performed on web the J48 classifier. For WS3-WS5 datasets, both classifiers
services datasets given in the following. Table 3 displays showed accuracy performance with negligible difference.
web services datasets with their respective WSDL ’universal As we expected, that AdaBoostM1 and J48 classifiers got
resource locator’ (URL) addresses. The WS-Dream dataset better accuracies, because they are capable of capturing the
has been widely used by many researchers in the selection web services instances classification in each web service
and ranking of web services [35]. We used 20% density dataset.
information from our web services datasets. In addition to Fig. 2 shows the average Kappa statistics of the chosen
metric values, each web service has web service Id, WSDL web service dataset for the binary classification of the users’
address, ’internet protocol’ IP address, country, ’autonomous invoked instances. After ranking the web services, we need to
system’(AS), latitude, and longitude properties. evaluate the proposed approach. Therefore, we used the data
of the web services to check the precision of the Kappa coef-
ficient. Kappa coefficient was measured from each classifier.
B. ACCURACY RESULTS
In this section, we compared the performance of AdaBoostM1
and J48 by using the information collected from the experi-
ments. Experiments performed on web service datasets accu-
mulated the results of various evaluation metrics.
We presented the accurate classification, Kappa, Precision,
Recall, and F-Measure statistics for each classifier on the
web service datasets. Table 4 shows the results of these
accuracy metrics. Among these accuracy metrics, we used
Kappa statistics to evaluate our proposed approach because
Ben-David and Frank [36] reported that Kappa statistics
show good prediction performance of classifiers in the binary
classification problem. Kappa statistics does not ignore the
classification that occurs due to mere chances. A high Kappa
statistics value indicates that the assignment of instances to a
group is not random; AdaBoostM1 and J48 are well-trained
to classify web service instances. Therefore, Kappa statis-
tics show the best classification ability of a classifier [37].
We obtained the average Kappa value for each classifier with FIGURE 2. Average Kappa statistics of AdaBoostM1 and J48 classifier.

VOLUME 8, 2020 90855

 M. Hasnain et al.: Evaluating Trust Prediction and Confusion Matrix Measures for Web Services Ranking

TABLE 4. Statistics of accuracy metrics for AdaBoostM1 and J48 classifier.

We obtained the Kappa coefficient average in all cases. The TABLE 5. Confusion matrix for WS1 at 5-Fold cross validation.
Kappa coefficient, as shown in Fig. 2, indicated good agree-
ment between the predicted and actual web service instances
for all web service datasets. The proposed approach, with
the help of data mining, provided high precision, and accu-
racy for all (i.e., WS1 to WS5) datasets. The proposed TABLE 6. Confusion matrix for WS1 at 10-Fold cross validation.
approach was also evaluated using the J48 in the similar ways
as AdaBoostM1. The ability of the proposed approach to
determine the complex interaction between predictive web
service instances and decrease to the biases was indicated
by the Kappa coefficient and other accuracy metrics. For
datasets WS1 and WS3, the average Kappa statistics val-
training several classifiers on our datasets and finally selected
ues of AdabBoostM1 were 0.9118 and 0.8872, respectively,
AdaBoostM1 and J48, which improved numerical prediction
thereby showing a perfect agreement between predictive
of instances.
and actual web service instances. Meanwhile, the Kappa
→ We determined confusion matrix measures for each
coefficient values of J48 for WS1 and WS3 datasets were
of the five web services datasets. The confusion matrix
0.8529 and 0.8980, respectively. For the remaining datasets
contains the information on the actual and predicted clas-
(i.e., WS2, WS4, and WS5), the Kappa coefficient val-
sification of web service instances. Prior to this work,
ues from AdaBoostM1were between 0.70 and 0.85, which
Mehdi et al. [38] used the confusion matrix to present true
showed an extremely high similarity between predicted and
and predicted classes. We used the confusion matrix with all
actual web services instances.
its measures to compute the evaluation parameters. The per-
centage of accurately classified web service instances from
C. CROSS VALIDATION RESULTS 5-, 10-, and 15-fold CVs was used as the measure for the
→ We present our results from three k-fold CV on web model. Tables (5-7) show the confusion matrix results for
services datasets, as follows. We performed experiments on WS1 by using AdaBoostM1 for three different k-fold CV

90856 VOLUME 8, 2020

M. Hasnain et al.: Evaluating Trust Prediction and Confusion Matrix Measures for Web Services Ranking

TABLE 7. Confusion matrix for WS1 at 15-Fold cross validation. TABLE 9. Ranking score of web services.

methods. Table 5 shows the obtained confusion matrix results

of WS1 from AdaBoost M1. A total of 64 out of 68 instances
Table 9 also shows the final ranking of web services from
were accurately classified. Table 6 shows the confusion
the computed average TS percent values. A web service with
matrix results of WS1 in the 10 k-fold CV method. A total
the highest average TS percent value was predicted as the
of 65 out of 68 instances were accurately classified.
most trusted web service from the users. Table 8 shows the
Table 7 shows that by adjusting the desired k-fold at
simple implementation of our proposed Eq. (4) by using the
a 15-fold CV method, the maximum number of instances,
results in Table 9. The ranking method was mainly based
that is, 66 out of 68 web service instances, have been cor-
on trust criteria of the average TS percent, showing that
rectly classified. Confusion matrix results for WS1 to WS5 in
WS1 was the most trusted by users with 48.5294% score,
5-, 10-, and 15-fold CV methods are shown in Table 8.
and WS2 was the least trusted with a 24.0196% score. Sim-
These results are presented using the AdaBoostM1 technique.
ilarly, we computed the ranking score of the remaining web
Furthermore, we list the number of trusted and untrusted
services, namely, WS3, WS4, and WS5, by using our pro-
instances detected in each dataset. In Table 8, TP indicates
posed TS percent ranking criteria. We can interpret the results
the number of web services instances was correctly assigned
shown in Fig. 3 by considering the trust score calculated
from the trusted class of instances, and FP shows those
from the binary classification of web services instances. Our
instances which were wrongly assigned. Similarly, TN indi-
trust-based web service ranking was based on the accurate
cates the number of web services instances were correctly
prediction of true instances of a given dataset. TP, FP, TN,
assigned from the untrusted class of instances, and FN shows
and FN were four measures of a confusion matrix for binary
those instances which were wrongly assigned.
classification.

TABLE 8. Number of trusted and untrusted instances detected in web

services datasets.

FIGURE 3. Presentation of web services ranking.

D. RANKING RESULTS
The main objective of using classifiers within three k-fold E. IMPACT OF QoS ATTRIBUTES VALUES CHANGES ON
validation methods was to identify how the prediction of WEB SERVICES RANKING
trusted and untrusted instances of users was performed and Hasnain et al. [56], in their recently published paper, high-
interpreted. To interpret the predicted instances accurately, lighted the effects of several quality attributes. They found
we ranked the web services in terms of the accurate prediction the dominating metrics which have a higher impact on the
of trusted and untrusted instances. decision making for the selection of web services datasets.
→ Table 9 shows the computed web service ranking by For instance, throughput and response time as quality metrics
using Eq. (4) mentioned above. The results in Table 8 were were among top quality metrics with their effects. Since
used to determine the average TS percent and the web service in this study, we are dealing with the latter mentioned two
ranking. quality attributes; the impact of changes in throughput and

VOLUME 8, 2020 90857

 M. Hasnain et al.: Evaluating Trust Prediction and Confusion Matrix Measures for Web Services Ranking

response time metrics can be easily determined. The higher The first internal threat to the proposed approach is the
value of throughput instances of web services may change the choice of selection of trust subject of users. There are some
ranking of web services. other choices to rank the web services. For instance, the secu-
As observed in [57], the lesser value of the QoS criterion rity of web service is not directly measured in this paper.
has a higher impact on the proposed ranking results. There- The security of web services is more relevant to web services
fore, low values of quality metrics have effects on the rank- standards and can be handled during the development of
ing results of our proposed approach. As can be seen from web services. Our proposed trust-based ranking approach of
Table 9, the increase in the value of the TS percent method web services is evaluated on the web services data, which
may change the ranking results of the proposed approach. For indirectly measures the confidentiality and reliability of web
instance, WS2 web service may get a new ranking if TS services.
percent values are increased. As a result, it can be ranked at The external validity of the proposed approach is the
position four before WS3. selection of web services datasets. Because performed exper-
iments for the evaluation of our trust-based approach are
V. THE IMPACT OF THE DATA-SET SIZE ON THE TRUST undertaken on the five web services datasets, however, exper-
PREDICTION PRECISION iments can be performed on using more web services datasets
It is known that dataset size profoundly influences the per- from the same datasets and other published datasets of web
formance of a machine learning algorithm. A basic algorithm services. We plan to include more web services datasets by
with lots of data shows the performance edge over the modern accessible information from accessible data repositories.
algorithms. Liu et al. [58] mentioned the datasets which
are in tens of thousands in records i.e., Bitcoin and Ciao VII. CONCLUSION AND FUTURE WORKS
datasets. In addition to these datasets, the Epinions dataset We developed the web service ranking approach that uses
with regards to review rating has been also used. Similar to feedback by users in terms of throughput and response
earlier mentioned datasets, our proposed approach provides time. We proposed fuzzy rules to make binary classification
trust and distrust scores. The main difference between the improve the effect by structuring the various conditions
previously used datasets and our dataset is the variance in of users’ feedback. Next, we established the trust predic-
trust and distrust scores. The highest average TS percent score tion formula from confusion matrix measures. We used
of WS1 is 48.5294. The trust score may vary due to class AdaBoostM1 to predict the trusted and untrusted web service
imbalance issues. The class imbalance issue may be due to instances and compared accuracy with J48 classification tech-
variance in the instances of a dataset. nique. From binary classification of web service instances,
The second point is the impact of dataset size on the trust we used three k-fold CV methods and determined the trust
prediction accuracy. To improve trust prediction accuracy, score of web services. Kappa statistics were applied to eval-
correct labeling of classes is significant. To do so, we have uate the proposed approach.
chosen the weak classifier, such as AdaBoostM1 and J48, This paper has implications for software architects and
which improve their learning capability. The smaller dataset managers. The first implication of the proposed approach is
size requirements to train classifiers may improve their pre- that architects can build better web services by using the trust
diction accuracy. In this regard, Wang et al. [59] stated that features of consumers. The second implication is that web
the short training dataset resulted in improving the prediction services managers can use the ranking of web services based
of the random forest algorithm. This explanation appears to on users’ trust to improve the quality of web services.
be convincing in view of the results of this study because
the training dataset size, in our case, is in tens of reviews ACKNOWLEDGMENT
of web services users. In addition to it, Heydari and Moun- It is clearly stated that no funding was available for this
trakis [60] validated that 2% and 5% training dataset size did research. This research article is relevant to the ongoing
not show a massive difference in the prediction accuracy of research in the School of Information Technology, Monash
the classifiers. Referring to Table 4 where prediction accuracy University Malaysia.
results with different K-folds are reported, we observe that the
Kappa value alongside (Precision, Recall and F-Measure) are REFERENCES
almost higher for both classifiers regarding trust prediction [1] A. Bawazir, W. Alhalabi, M. Mohamed, A. Sarirete, and A. Alsaig,
of web services.Table 4 shows us that Precision, Recall, and ‘‘A formal approach for matching and ranking trustworthy context-
dependent services,’’ Appl. Soft Comput., vol. 73, pp. 306–315, Dec. 2018.
F-measure accuracy metrics values for both AdaBoostM1 and [2] M. Almulla, H. Yahyaoui, and K. Al-Matori, ‘‘A new fuzzy hybrid tech-
J48 classifiers are above than 8 value, which indicates a high nique for ranking real world Web services,’’ Knowl.-Based Syst., vol. 77,
accuracy from both classifiers. pp. 1–15, Mar. 2015.
[3] G. James, D. Witten, T. Hastie, and R. Tibshirani, An Introduction to
Statistical Learning. Cham, Switzerland: Springer, 2013.
VI. THREATS TO VALIDITY [4] O. Caelen, ‘‘A Bayesian interpretation of the confusion matrix,’’ Ann.
This section of the paper presents validity threats to our Math. Artif. Intell., vol. 81, nos. 3–4, pp. 429–450, Dec. 2017.
[5] R. Rajalakshmi and C. Aravindan, ‘‘A Naive Bayes approach for URL
trust-based ranking approach from the evaluation of confu- classification with supervised feature selection and rejection framework,’’
sion matrix measures of web services data. Comput. Intell., vol. 34, no. 1, pp. 363–396, Feb. 2018.

90858 VOLUME 8, 2020

M. Hasnain et al.: Evaluating Trust Prediction and Confusion Matrix Measures for Web Services Ranking

[6] J. Font, L. Arcega, O. Haugen, and C. Cetina, ‘‘Achieving feature location [30] G. Chandrashekar and F. Sahin, ‘‘A survey on feature selection methods,’’
in families of models through the use of search-based software engineer- Comput. Elect. Eng., vol. 40, no. 1, pp. 16–28, 2014.
ing,’’ IEEE Trans. Evol. Comput., vol. 22, no. 3, pp. 363–377, Jun. 2018. [31] D. Silva-Palacios, C. Ferri, and M. J. Ramírez-Quintana, ‘‘Probabilistic
[7] K. Su, B. Xiao, B. Liu, H. Zhang, and Z. Zhang, ‘‘TAP: A personalized class hierarchies for multiclass classification,’’ J. Comput. Sci., vol. 26,
trust-aware QoS prediction approach for Web service recommendation,’’ pp. 254–263, May 2018.
Knowl.-Based Syst., vol. 115, pp. 55–65, Jan. 2017. [32] WS-Dream. Accessed: Feb. 5, 2020. [Online]. Available:
[8] S. Hamdi, A. L. Gancarski, A. Bouzeghoub, and S. B. Yahia, https://github.com/wsdream
‘‘TISoN: Trust inference in trust-oriented social networks,’’ ACM Trans. [33] WS-Dream. Towards Open Datasets and Source Code for Web
Inf. Syst., vol. 34, no. 3, pp. 1–32, May 2016. Services Research. Accessed: Feb. 7, 2020. [Online]. Available:
[9] K. Polat, S. Güneş, and A. Arslan, ‘‘A cascade learning system for http://wsdream.github.io/
classification of diabetes disease: Generalized discriminant analysis and [34] Z. Zheng, H. Ma, M. R. Lyu, and I. King, ‘‘Collaborative Web service QoS
least square support vector machine,’’ Expert Syst. Appl., vol. 34, no. 1, prediction via neighborhood integrated matrix factorization,’’ IEEE Trans.
pp. 482–487, Jan. 2008. Services Comput., vol. 6, no. 3, pp. 289–299, Jul. 2013.
[10] S. Choudhury and A. Bhowal, ‘‘Comparative analysis of machine learning [35] Z. Zheng, Y. Zhang, and M. R. Lyu, ‘‘Investigating QoS of real-world
algorithms along with classifiers for network intrusion detection,’’ in Proc. Web services,’’ IEEE Trans. Services Comput., vol. 7, no. 1, pp. 32–39,
Int. Conf. Smart Technol. Manage. Comput., Commun., Controls, Energy Jan./Mar. 2014.
Mater. (ICSTM), May 2015, pp. 89–95. [36] A. Ben-David and E. Frank, ‘‘Accuracy of machine learning models versus
[11] S. Aljawarneh, M. Aldwairi, and M. B. Yassein, ‘‘Anomaly-based intrusion ‘hand crafted’ expert systems—A credit scoring case study,’’ Expert Syst.
detection system through feature selection analysis and building hybrid Appl., vol. 36, no. 3, pp. 5264–5271, 2009.
efficient model,’’ J. Comput. Sci., vol. 25, pp. 152–160, Mar. 2018. [37] P. Shrivastava, K. K. Bhoyar, and A. S. Zadgaonkar, ‘‘Image classification
[12] F. Al-Obeidat and E.-S.-M. El-Alfy, ‘‘Hybrid multicriteria fuzzy clas- using fusion of holistic visual descriptions,’’ Int. J. Image, Graph. Signal
sification of network traffic patterns, anomalies, and protocols,’’ Pers. Process., vol. 8, no. 8, pp. 47–57, Aug. 2016.
Ubiquitous Comput., vol. 23, nos. 5–6, pp. 777–791, Nov. 2019. [38] M. Mehdi, N. Bouguila, and J. Bentahar, ‘‘Probabilistic approach for QoS-
[13] S. Ding, S. Yang, Y. Zhang, C. Liang, and C. Xia, ‘‘Combining QoS aware recommender system for trustworthy Web service selection,’’ Int. J.
prediction and customer satisfaction estimation to solve cloud ser- Speech Technol., vol. 41, no. 2, pp. 503–524, Sep. 2014.
vice trustworthiness evaluation problems,’’ Knowl.-Based Syst., vol. 56, [39] M. Tang, X. Dai, J. Liu, and J. Chen, ‘‘Towards a trust evaluation middle-
pp. 216–225, Jan. 2014. ware for cloud service selection,’’ Future Gener. Comput. Syst., vol. 74,
[14] W. Hussain, F. K. Hussain, M. Saberi, O. K. Hussain, and E. Chang, pp. 302–312, Sep. 2017.
‘‘Comparing time series with machine learning-based prediction [40] B. Zhou, Q. Zhang, Q. Shi, Q. Yang, P. Yang, and Y. Yu, ‘‘Measuring Web
approaches for violation management in cloud SLAs,’’ Future Gener. service security in the era of Internet of Things,’’ Comput. Electr. Eng.,
Comput. Syst., vol. 89, pp. 464–477, Dec. 2018. vol. 66, pp. 305–315, Feb. 2018.
[15] N. Somu, G. R. M. R., K. Kirthivasan, and S. S. V. S., ‘‘A trust centric [41] J. Jang-Jaccard and S. Nepal, ‘‘A survey of emerging threats in cybersecu-
optimal service ranking approach for cloud service selection,’’ Future rity,’’ J. Comput. Syst. Sci., vol. 80, no. 5, pp. 973–993, Aug. 2014.
Gener. Comput. Syst., vol. 86, pp. 234–252, Sep. 2018. [42] Z. M. Aljazzaf, M. A. M. Capretz, and M. Perry, ‘‘Trust bootstrapping
[16] Z. Saoud, N. Faci, Z. Maamar, and D. Benslimane, ‘‘A fuzzy-based credi- services and service providers,’’ in Proc. 9th Annu. Int. Conf. Privacy,
bility model to assess Web services trust under uncertainty,’’ J. Syst. Softw., Secur. Trust, Montreal, QC, Canada, Jul. 2011, pp. 195–200.
vol. 122, pp. 496–506, Dec. 2016. [43] H. T. Nguyen, W. Zhao, and J. Yang, ‘‘A trust and reputation model based
[17] Y. Ma, S. Wang, P. C. K. Hung, C.-H. Hsu, Q. Sun, and F. Yang, ‘‘A highly on Bayesian network for Web services,’’ in Proc. IEEE Int. Conf. Web
accurate prediction algorithm for unknown Web service QoS values,’’ Services, Jul. 2010, pp. 251–258.
IEEE Trans. Services Comput., vol. 9, no. 4, pp. 511–523, Jul. 2016. [44] S.-G. Deng, L.-T. Huang, J. Wu, and Z.-H. Wu, ‘‘Trust-based personalized
[18] M. Sugeno and G. T. Kang, ‘‘Structure identification of fuzzy model,’’ service recommendation: A network perspective,’’ J. Comput. Sci. Tech-
Fuzzy Sets Syst., vol. 28, no. 1, pp. 15–33, Oct. 1988. nol., vol. 29, no. 1, pp. 69–80, Jan. 2014.
[19] H. B. Yadav and D. K. Yadav, ‘‘A fuzzy logic based approach for phase- [45] S. Wang, L. Huang, C.-H. Hsu, and F. Yang, ‘‘Collaboration reputation for
wise software defects prediction using software metrics,’’ Inf. Softw. Tech- trustworthy Web service selection in social networks,’’ J. Comput. Syst.
nol., vol. 63, pp. 44–57, Jul. 2015. Sci., vol. 82, no. 1, pp. 130–143, Feb. 2016.
[20] H. Wang, Z. Xu, and X.-J. Zeng, ‘‘Hesitant fuzzy linguistic term sets for [46] M. Mehdi, N. Bouguila, and J. Bentahar, ‘‘Trust and reputation of Web
linguistic decision making: Current developments, issues and challenges,’’ services through QoS correlation lens,’’ IEEE Trans. Services Comput.,
Inf. Fusion, vol. 43, pp. 1–12, Sep. 2018. vol. 9, no. 6, pp. 968–981, Nov. 2016.
[21] M. Bouhentala, M. Ghanai, and K. Chafaa, ‘‘Interval-valued member- [47] O. Tibermacine, C. Tibermacine, and F. Cherif, ‘‘Estimating the reputation
ship function estimation for fuzzy modeling,’’ Fuzzy Sets Syst., vol. 361, of newcomer Web services using a regression-based method,’’ J. Syst.
pp. 101–113, Apr. 2019. Softw., vol. 145, pp. 112–124, Nov. 2018.
[22] C. J. Mantas, ‘‘A generic fuzzy aggregation operator: Rules extraction from [48] H. Liu, P. Burnap, W. Alorainy, and M. L. Williams, ‘‘A fuzzy approach to
and insertion into artificial neural networks,’’ Soft Comput., vol. 12, no. 5, text classification with two-stage training for ambiguous instances,’’ IEEE
pp. 493–514, Mar. 2008. Trans. Comput. Social Syst., vol. 6, no. 2, pp. 227–240, Apr. 2019.
[23] E. M. Bahgat, S. Rady, W. Gad, and I. F. Moawad, ‘‘Efficient email [49] M.-S. Hosseini and A.-M. Eftekhari-Moghadam, ‘‘Fuzzy rule-based rea-
classification approach based on semantic methods,’’ Ain Shams Eng. J., soning approach for event detection and annotation of broadcast soccer
vol. 9, no. 4, pp. 3259–3269, Dec. 2018. video,’’ Appl. Soft Comput., vol. 13, no. 2, pp. 846–866, Feb. 2013.
[24] M. A. Mohammed, B. Al-Khateeb, A. N. Rashid, D. A. Ibrahim, [50] Y. Liu, J.-W. Bi, and Z.-P. Fan, ‘‘Multi-class sentiment classification:
M. K. A. Ghani, and S. A. Mostafa, ‘‘Neural network and multi- The experimental comparisons of feature selection and machine learning
fractal dimension features for breast cancer classification from ultrasound algorithms,’’ Expert Syst. Appl., vol. 80, pp. 323–339, Sep. 2017.
images,’’ Comput. Elect. Eng., vol. 70, pp. 871–882, Aug. 2018. [51] D. Wang, X. Tong, and Y. Wang, ‘‘An early risk warning system
[25] A. K. Tripathy and P. K. Tripathy, ‘‘Fuzzy QoS requirement-aware for outward foreign direct investment in mineral resource-based enter-
dynamic service discovery and adaptation,’’ Appl. Soft Comput., vol. 68, prises using multi-classifiers fusion,’’ Resour. Policy, vol. 66, Jun. 2020,
pp. 136–146, Jul. 2018. Art. no. 101593.
[26] E. A. Cortés, M. G. Martínez, and N. G. Rubio, ‘‘Multiclass corporate [52] C. Mao, R. Lin, C. Xu, and Q. He, ‘‘Towards a trust prediction framework
failure prediction by Adaboost. M1,’’ Int. Adv. Econ. Res., vol. 13, no. 3, for cloud services based on PSO-driven neural network,’’ IEEE Access,
pp. 301–312, 2007. vol. 5, pp. 2187–2199, 2017.
[27] P. Chen and C. Pan, ‘‘Diabetes classification model based on boosting [53] N. Somu, G. R. M. R., K. V., K. Kirthivasan, and S. S. V. S., ‘‘An improved
algorithms,’’ BMC Bioinf., vol. 19, no. 1, p. 109, Dec. 2018. robust heteroscedastic probabilistic neural network based trust prediction
[28] X. Zhang and Q. Song, ‘‘A multi-label learning based kernel automatic approach for cloud service selection,’’ Neural Netw., vol. 108, pp. 339–354,
recommendation method for support vector machine,’’ PLoS ONE, vol. 10, Dec. 2018.
no. 4, 2015, Art. no. e0120455. [54] M. Bisi and S. Patel, ‘‘A BPSO-ANN model for trust prediction of
[29] J. Lei, ‘‘Cross-validation with confidence,’’ 2017, arXiv:1703.07904. cloud services,’’ in Proc. Global Conf. Advancement Technol. (GCAT),
[Online]. Available: http://arxiv.org/abs/1703.07904 Oct. 2019, pp. 1–5.

VOLUME 8, 2020 90859

 M. Hasnain et al.: Evaluating Trust Prediction and Confusion Matrix Measures for Web Services Ranking

[55] S. Nivethitha, M. R. G. Raman, O. Gireesha, K. Kannan, and [76] P. Hilletofth, M. Sequeira, and A. Adlemo, ‘‘Three novel fuzzy logic con-
V. S. S. Sriram, ‘‘An improved rough set approach for optimal trust measure cepts applied to reshoring decision-making,’’ Expert Syst. Appl., vol. 126,
parameter selection in cloud environments,’’ Soft Comput., vol. 23, no. 22, pp. 133–143, Jul. 2019.
pp. 11979–11999, Nov. 2019.
[56] M. Hasnain, M. F. Pasha, I. Ghani, B. Mehboob, M. Imran, and A. Ali,
‘‘Benchmark dataset selection of Web services technologies: A factor
analysis,’’ IEEE Access, vol. 8, pp. 53649–53665, 2020.
[57] A. Ouadah, A. Hadjali, F. Nader, and K. Benouaret, ‘‘SEFAP: An efficient
approach for ranking skyline Web services,’’ J. Ambient Intell. Hum. MUHAMMAD HASNAIN was born in Bhakkar,
Comput., vol. 10, no. 2, pp. 709–725, Feb. 2019. Punjab, Pakistan, in 1977. He received the M.Sc.
[58] S. Liu, L. Zhang, and Z. Yan, ‘‘Predict pairwise trust based on machine degree in computer science from Abasyn Uni-
learning in online social networks: A survey,’’ IEEE Access, vol. 6,
versity Islamabad, Pakistan, in 2016. He is cur-
pp. 51297–51318, 2018.
rently pursuing the master’s degree with the
[59] H. Wang, R. Magagi, K. Goïta, M. Trudel, H. McNairn, and
J. Powers, ‘‘Crop phenology retrieval via polarimetric SAR decomposition School of Information Technology, Monash Uni-
and random forest algorithm,’’ Remote Sens. Environ., vol. 231, Sep. 2019, versity Malaysia. From 2016 to 2017, he worked
Art. no. 111234. as a Lecturer with the Army Public College of
[60] S. S. Heydari and G. Mountrakis, ‘‘Meta-analysis of deep neural networks Management Sciences, Rawalpindi, Pakistan. His
in remote sensing: A comparative study of mono-temporal classification to research interest is focused on web services quality
support vector machines,’’ ISPRS J. Photogramm. Remote Sens., vol. 152, enhancement.
pp. 192–210, Jun. 2019.
[61] W. Rhmann, B. Pandey, G. Ansari, and D. K. Pandey, ‘‘Software fault
prediction based on change metrics using hybrid algorithms: An empirical
study,’’ J. King Saud Univ.-Comput. Inf. Sci., vol. 32, no. 4, pp. 419–424,
May 2020.
[62] V. Sharma and K. C. Juglan, ‘‘Automated classification of fatty and normal MUHAMMAD FERMI PASHA (Member, IEEE)
liver ultrasound images based on mutual information feature selection,’’ was born in Indonesia. He received the Ph.D.
IRBM, vol. 39, no. 5, pp. 313–323, Nov. 2018. degree in computer science from Universiti Sains
[63] H. Hong, J. Liu, D. T. Bui, B. Pradhan, T. D. Acharya, B. T. Pham, Malaysia, in 2010.
A.-X. Zhu, W. Chen, and B. B. Ahmad, ‘‘Landslide susceptibility mapping After his Ph.D. degree, he worked as a Research
using J48 decision tree with AdaBoost, bagging and rotation forest ensem- Fellow at Universiti Sains Malaysia. He is cur-
bles in the Guangchang area (China),’’ CATENA, vol. 163, pp. 399–413, rently working as a Lecturer at the School of Infor-
Apr. 2018. mation Technology, Monash University Malaysia.
[64] N. Somu, G. R. MR, A. Kaveri, A. Rahul, K. Krithivasan, and S. Sriram,
His research interests are focused on computa-
‘‘BGSS: An improved binary gravitational search algorithm based search
tional neuroimaging, intelligent network security
strategy for QoS and ranking prediction in cloud environments,’’ Appl. Soft
Comput., vol. 88, pp. 1–20, 2020. traffic analysis, and healthcare and radiology IT with emphasis on big data.
[65] C. Mao, J. Chen, D. Towey, J. Chen, and X. Xie, ‘‘Search-based QoS He is also supervising the Ph.D. students in the latter mentioned research
ranking prediction for Web services in cloud environments,’’ Future Gener. areas.
Comput. Syst., vol. 50, pp. 111–126, Sep. 2015.
[66] H. Ma, H. Zhu, Z. Hu, K. Li, and W. Tang, ‘‘Time-aware trustworthiness
ranking prediction for cloud services using interval neutrosophic set and
ELECTRE,’’ Knowl.-Based Syst., vol. 138, pp. 27–45, Dec. 2017.
[67] F. Qu, J. Liu, H. Zhu, and B. Zhou, ‘‘Wind turbine fault detection based
on expanded linguistic terms and rules using non-singleton fuzzy logic,’’ IMRAN GHANI was born in Pakistan. He received
Appl. Energy, vol. 262, Mar. 2020, Art. no. 114469. the Ph.D. degree from Kookmin University,
[68] L. A. Zadeh, ‘‘A computational approach to fuzzy quantifiers in natural lan- South Korea, in 2010, and the M.Sc. degree in
guages,’’ in Computational Linguistics. New York, NY, USA: Pergamon, computer science from UTM, Malaysia, in 2007.
1983, pp. 149–184. He worked as a Senior Lecturer with Monash
[69] R. R. Yager, M. Z. Reformat, and N. D. To, ‘‘Drawing on the iPad to input University Malaysia. He is currently working as
fuzzy sets with an application to linguistic data science,’’ Inf. Sci., vol. 479, an Associate Professor of computer science with
pp. 277–291, Apr. 2019. the Mathematical and Computer Science Depart-
[70] M. Vučetić, M. Hudec, and B. Božilović, ‘‘Fuzzy functional dependen- ment, Indiana University of Pennsylvania. He has
cies and linguistic interpretations employed in knowledge discovery tasks published more than 80 research articles in reputed
from relational databases,’’ Eng. Appl. Artif. Intell., vol. 88, Feb. 2020, journals and also edited two books. His research interests are focused on
Art. no. 103395.
software engineering, web services, web mining, and cloud computing. He is
[71] M. Sözat and A. Yazici, ‘‘A complete axiomatization for fuzzy functional
currently supervising the Ph.D. students in the latter mentioned research
and multivalued dependencies in fuzzy database relations,’’ Fuzzy Sets
Syst., vol. 117, no. 2, pp. 161–181, Jan. 2001. areas.
[72] B. Sheng, O. M. Moosman, B. Del Pozo-Cruz, J. Del Pozo-Cruz,
R. M. Alfonso-Rosa, and Y. Zhang, ‘‘A comparison of different machine
learning algorithms, types and placements of activity monitors for
physical activity classification,’’ Measurement, vol. 154, Mar. 2020,
Art. no. 107480.
[73] F. Lopes, J. Agnelo, C. A. Teixeira, N. Laranjeiro, and J. Bernardino, MUHAMMAD IMRAN was born in Lahore,
‘‘Automating orthogonal defect classification using machine learning algo- Pakistan. He received the master’s degree in
rithms,’’ Future Gener. Comput. Syst., vol. 102, pp. 932–947, Jan. 2020. computer science from COMSATS University,
[74] D. Singh and B. Singh, ‘‘Investigating the impact of data normaliza- Lahore, Pakistan. He is currently working as a
tion on classification performance,’’ Appl. Soft Comput., May 2019, Senior Software Engineer in a software industry in
Art. no. 105524. Pakistan. His research interests include data min-
[75] L. Munkhdalai, T. Munkhdalai, K. H. Park, H. G. Lee, M. Li, and ing, machine learning, and software engineering.
K. H. Ryu, ‘‘Mixture of activation functions with extended min-
max normalization for forex market prediction,’’ IEEE Access, vol. 7,
pp. 183680–183691, 2019.

90860 VOLUME 8, 2020

M. Hasnain et al.: Evaluating Trust Prediction and Confusion Matrix Measures for Web Services Ranking

MOHAMMED Y. ALZAHRANI received the mas- RAHMAT BUDIARTO received the B.Sc. degree
ter’s and Ph.D. degrees in computer science from from the Bandung Institute of Technology,
Heriot-Watt University, U.K., in 2010 and 2015, in 1986, and the M.Eng. and Dr.Eng. degrees in
respectively. He is currently the Dean of the computer science from the Nagoya Institute of
College of Computer Science and Information Technology, in 1995 and 1998, respectively. He is
Technology, Albaha University, Saudi Arabia. His currently a Full Professor at the College of Com-
research interests include model checking and ver- puter Science and IT, Albaha University, Saudi
ification, intelligent healthcare systems, and infor- Arabia. His research interests include intelligent
mation security. systems, brain modeling, IPv6, network security,
wireless sensor networks, and MANETs.

VOLUME 8, 2020 90861