Expert Systems with Applications 40 (2013) 1609–1618

Contents lists available at SciVerse ScienceDirect

Expert Systems with Applications

journal homepage: www.elsevier.com/locate/eswa

A novel Likert scale based on fuzzy sets theory

Qing Li
Mechanical Engineering Department, University of New Haven, 300 Boston Post Road, West Haven, CT 06516, USA

a r t i c l e i n f o a b s t r a c t

Keywords: The Likert method is commonly used as a standard psychometric scale to measure responses. This mea-
Likert scale surement scale has a procedure that facilitates survey construction and administration, and data coding
Fuzzy sets theory and analysis. However, there are some drawbacks in the Likert scaling. This paper addresses the informa-
Fuzzy Likert scale tion distortion and information lost arising from the closed-form scaling and the ordinal nature of this
Information distortion
measurement method. To overcome these problems, a novel fuzzy Likert scale developed based on the
Information lost
Consensus
fuzzy sets theory has been proposed. The major contribution of the fuzzy Likert approach is that it per-
mits partial agreement of a scale point. By incorporating this capability into the measurement process,
the new scale can capture the lost information and regulate the distorted information. A quantitative
analysis based on the concept Consensus has proven that the new scale can provide a more accurate mea-
surement. The implementation feasibility and the improved measurement performance of the fuzzy Lik-
ert scale have been demonstrated via a simulation study on a low birth weight analysis.
Ó 2012 Elsevier Ltd. All rights reserved.

1. Introduction applied to measure latent constructs that are not directly observa-
ble. For example, Springer (1998) used a Likert scale to examine
Likert scaling, originally introduced by Rensis Likert in 1932, is adolescent concerns that foster runaway behavior, Copeland
the most widely used psychometric scale in survey research. It asks (2003) explored the problems faced by young women living in dis-
respondents to indicate their levels of agreement with a declara- advantaged conditions, Acharya, Lee, and Im (2006) identified con-
tive statement. For a 5-point Likert scale, for example, each scale flicting factors in construction projects, Bañuelas and Antony
point could be labeled according to its agreement level: (2007) developed a stochastic analytic hierarchy process in opera-
1 = strongly disagree (SD), 2 = disagree (D), 3 = neither disagree tional research, and Li, McCoach, Swaminathan, and Tang (2008)
nor agree (NN), 4 = agree (A), and 5 = strongly agree (SA). Depend- applied a Likert scale to develop an instrument to measure student
ing on what is being measured, the scale labels may be worded dif- perspectives of engineering education.
ferently. When measuring frequency, for instance, labels like The popularity of the Likert method comes from a number of
‘‘never-always’’ can be used; when measuring attitude, belief, or facts. First, a Likert scale can be easily constructed and modified.
characteristic of the respondent, labels like ‘‘not very much-very Second, the numerical measurement results can be directly used
much’’ are suitable. A well designed Likert scale should state the for statistical inference. Last but not least, measurements based
opinion, attitude, or belief being measured in clear terms and use on Likert scaling have demonstrated a good reliability. In general,
the appropriate wording for scale points. with Likert scaling researchers can collect and analyze a large
Likert scales have been widely used to measure observable quantity of data with less time and effort. Despite these advanta-
attributes in various social science measurement areas. Examples ges, however, Likert scales have several weaknesses.
of measured variables include fondness of music education (Orr One of the major problems comes from the debate about
& Ohlsson, 2005), organizational behavior in learning organization whether a Likert scale is ordinal or interval (Jamieson, 2004).
(Kiedrowski, 2006), satisfaction of journal quality in library science Although Rensis Likert himself assumed that the Likert method
(Yue, Wilson, & Boller, 2007), effectiveness of drugs in pharmaceu- has an interval scale quality, many consider Likert scaling as ordi-
tics (Buncher & Tsay, 2006), patient advocacy in hospital (Seal, nal in nature (Hodge & Gillespie, 2003; Pett, 1997). A conventional
2007), routine prioritization in dentistry (Postma, 2007), customer interval scale implies that the differences between any two consec-
attitudes towards labeling in nutrition (Lindhorst, Corby, Roberts, utive scales reflect equal differences in the variable measured.
& Zeiler, 2007), and athlete characteristics in sports (Brown, Gus- However, as Cohen, Manion, and Morrison (2000) argued, it is ille-
kiewicz, & Bleiberg, 2007), to name a few recent studies in the end- gitimate to assume that the intensity of feeling between ‘‘strongly
less application list. In addition, Likert scales have also been disagree’’ and ‘‘disagree’’ is equivalent to the intensity of feeling
between other consecutive categories on a Likert scale. In fact, it
E-mail address: [email protected] is problematic to treat the Likert method either way. As Russell

0957-4174/$ - see front matter Ó 2012 Elsevier Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.eswa.2012.09.015
1610 Q. Li / Expert Systems with Applications 40 (2013) 1609–1618

and Bobko (1992) put it, ‘‘Likert scales fail to approximate intervals In most practices in the current social science measurement soci-
of ordinal data’’. A typical ordinal scale can measure the orders of ety, these limitations are simply ignored. In this paper, a novel
the responses but tells nothing about the intervals between re- Liker scale designed based on fuzzy sets theory has been proposed.
sponses. On the other hand, a typical interval scale forces an equal The fuzzy sets theory offers psychometricians a new interpretive
interval between consecutive scales. The former scale leads to algebra, ‘‘a language that is half-verbal-conceptual and half-math-
information lost during measurement, while the later results in ematical-analytical’’ (Ragin, 2000). This interpretive mathematical
information distortion. language can transform a discrete ordinal variable into a continu-
The other weakness of the Likert scaling comes from its closed ous variable while retaining the semantic meaning. It thus pro-
response format (Hodge & Gillespie, 2003; Orvik, 1972). The vides us an idea tool to capture the interval details of ordinal
respondents are forced to make a choice from the given options variables in an open response format. With that capability, it is
that may not match their exact responses. They have to either se- possible to reduce information lost and decrease information dis-
lect an answer from an insufficient range of responses or respond tortion during measurement. A consensus model originated from
to an ‘‘acceptable’’ answer in the closed format. This miss-match- communication theory (Tastle & Wierman, 2007) has been applied
ing further worsens the information distortion problem. to rigorously prove that the proposed fuzzy Likert scale can provide
In summary, a significant amount of information is lost and/or a more accurate measurement than the traditional Likert method.
distorted due to the built-in limitations of the Likert method. Over A logistic regression simulation study based on a low birth weight
the years many researchers have tried to solve these problems. To analysis has demonstrated the implementability and effectiveness
address the information lost problem, Chang (1994) recommended of the novel fuzzy Likert scale.
increasing the scale points on a Likert scale. (Russell & Bobko,
1992) also suggested that the more scale points, the closer a Likert
2. Introduction to fuzzy sets theory
scale can approximate a continuous measure, and thus more infor-
mation can be captured. However, criticism comments that
The concept of fuzzy sets was first conceived by Lotfi Zadeh, an
respondents may have difficulty to accurately resolve their inten-
engineering professor at the University of California at Berkley, to
sity of feelings into many scale categories. Through a comparison
deal with reasoning that is approximate rather than precise (Za-
of 4- and 6-point Likert scales, Chang (1994) found that more scale
deh, 1965). Since then, fuzzy sets have been used in many engi-
points may actually increase the measurement error because
neering fields to address a variety of problems, both mundane
respondents can be confused by too many response categories. In
and abstract (Zimmermann, 1996). The ever expanding applica-
addition, Chang (1994) also pointed out that a longer response op-
tions of fuzzy sets have ranged from expert systems (Zimmermann,
tion list may intensify ‘‘laziness’’ in responding questionnaire.
1987), manufacturing systems (Gien, Jacqmart, Seklouli, & Barad,
Therefore, this modification method can increase several typical
2003), operational research (Zimmermann, 1983), to stock market
‘‘primacy effects’’ in Likert scaling, such as the response-order ef-
(Zopounidis, Pardalos, & Baourakis, 2001). Most of the literature in
fect (increasing the tendency for respondents to select the first re-
fuzzy sets applications is concerned with the development of smart
sponse available to them on the answer scale, Chan, 1991), donkey
machines that can act automatically in the face of ambiguity or
vote effect (selecting the same response for all questions, Ray,
complexity (Jamshidi, Titli, Zadeh, & Boverie, 1997).
1990), or central tendency effect (choosing the neutral response,
Although fuzzy sets have found a great success in engineering,
Brown, 2000).
their impact in social sciences has been rather limited. Several
Later on Albaum (1997) proposed a two-stage Likert scale as an
scholars have attempted to introduce fuzzy sets concepts to social
alternative to the traditional scale. In this alternative scale, the first
science community (Ragin, 2000; Smithson, 1987; Smithson &
stage measures the agreement (agree/disagree) to a statement. The
Verkuilen, 2006). The majority has yet to recognize the potential
second stage measures the intensity of agreement (strong or weak)
of fuzzy sets for transforming social science methodology. By
to the statement. It appears that the two-stage Likert scale can cap-
incorporating fuzzy sets into the traditional forms of qualitative
ture more extreme positions than the traditional Likert scale. In a
and quantitative analysis in social sciences, we are equipped with
sense, this design is effective at reducing the central tendency ef-
a powerful mathematical model that is able to retain the substan-
fect. However, it is not clear how this method can collect more
tial meaning of the underlying latent constructs without losing
information between the extreme positions than the traditional
analytic rigor. This capability of fuzzy sets can bring the measure-
method.
ment methodology in social sciences up to a whole new level.
More recently Hodge and Gillespie (2003) proposed a ‘‘phrase
completion’’ Likert scale approach that uses a sentence completion
format to measure agreement. For example, the statement ‘‘My 2.1. Fuzzy sets
religious beliefs affect every aspect of my life’’ is replaced with
an incomplete sentence fragment ‘‘My religious beliefs affect’’ plus Since our language abounds in imprecise and fuzzy information
two phrases ‘‘No aspect of my life’’ and ‘‘Absolutely every aspect of by nature, fuzzy concepts and fuzzy reasoning are commonly seen
my life’’ that are linked to a scale ranging from 0 to 10. These two in social science research. For example, a simple statement like
phrases, which complete the sentence fragment, anchor each end ‘‘This family income is high’’ can be fuzzy because the linguistic
of the scale. This phrase completion method exhibits some advan- label ‘‘High’’ is not precisely defined. Different people will have
tages to the traditional Likert scale. By specifying the underlying different standards in labeling ‘‘High’’. However, it is difficult to
theoretical continuum in the response, this method can capture translate this statement into a more precise language without
more detailed information. It is also proven that the reliability losing some of the semantic meaning. For instance, an alternative
and the inter-item correlations of the measurement are higher. to this statement could be ‘‘This family income is $100K/year’’. It
However, a long list of response choices can lead to the typical does indicate an exact number but no longer explicitly delivers
‘‘laziness’’ phenomenon (Chang, 1994). Furthermore, as the the semantic meaning ‘‘The income is ‘high’’’.
authors commented, the two-phrase completion format may also How about we include both exact number and semantic mean-
confuse respondents and present an operational challenge. ing into the statement: ‘‘If a family has an income of $100K/year,
Good efforts have been spent on the development of alterna- this family income is a high’’? This seems to be crystal clear now.
tives to Likert scales for better measurement results. However, But what if a family earns $90K/year, does it still belong to the
these alternatives have not circumvented the built-in limitations. ‘‘High family income’’ category? The question becomes fuzzy again.
 Q. Li / Expert Systems with Applications 40 (2013) 1609–1618 1611

In the traditional sets theory where crisp boundary is imposed, a its associated agreement level. For example, membership functions
member either belongs to a set (denoted as 1) or does not belong SD and D characterize fuzzy sets Strongly Disagree and Disagree
to the set (denoted as 0). If ‘‘High’’ is defined as ‘‘The family income respectively. The X-axis indicates the input variable, Family Income
is more than $10K/year’’, the membership of ‘‘$90K/year’’ in the set in this case, and the Y-axis represents the membership degree. If
‘‘High family income’’ would be 0, even if many respondents may Family Income is 20K or less (x 6 20), the input value falls on mem-
still feel that $90K/year is somewhat a ‘‘High family income’’. bership function SD, which returns a membership degree 1.0. The
On the other hand, fuzzy sets theory allows the membership of other membership functions D, NN, A, and SA return a degree 0.0
a variable to operate over the range of [0, 1], with 0 representing an to this input. It means that this Family Income absolutely falls on
absolute False (i.e., full non-membership), 1 representing an abso- fuzzy set Strongly Disagree and absolutely does not belong to fuzzy
lute Truth (i.e., full membership), and a value between 0 and 1 rep- sets Disagree, Neither or Nor Agree, Agree, and Strongly Agree. If a
resenting a partial or fuzzy membership of the variable in a set. family income is 30K, Fig. 1 shows that it falls on membership
With this fuzzy set definition, if Family A has an income of functions SD and D. Both functions return the same membership
$100K/year, a membership degree 1 will be assigned to ‘‘Family degree 0.5. It means that this family income partially belongs to
A’’ in the fuzzy set labeled as ‘‘High income’’. If Family B has an in- fuzzy set SD, and partially belongs to fuzzy set D.
come of $90K/year, the membership degree assigned to ‘‘Family B’’
in the same fuzzy set could be lower, such as 0.8, for instance. If 2.3. Fuzzification and defuzzification
Family C has an income of $20K/year, most likely Family C will
be considered as a low income family by all respondents. In this The fuzzification is the process of transforming input variables
case, the membership degree assigned to ‘‘Family C’’ in the fuzzy into fuzzy values that are described by membership degrees and
set ‘‘High income’’ could be 0. On the contrary, ‘‘Family C’’ might their associated membership functions. For example, if Mr. X has
be awarded a high membership degree 0.9 in the fuzzy set with a family income of $96K/year, according to Fig. 2, this income value
the opposite linguistic label ‘‘Low income’’. Symbolically, the falls on fuzzy sets A and SA. This input value is thus fuzzified into
assignments of the membership degrees to these variables in the two fuzzy values: lSA ðXÞ ¼ 0:8 in SA and lA ðXÞ ¼ 0:2 in A. It is
respective fuzzy sets can be represented as follows: noted that these fuzzy values are represented by the two gray
areas in Fig. 2.
lHIGH ðAÞ ¼ 1:0; lHIGH ðBÞ ¼ 0:8; lHIGH ðCÞ ¼ 0:0; lLOW ðCÞ ¼ 0:9 The fuzzification procedure results in a set of paired fuzzy val-
ues flLABEL ðxÞ; LABEL g, with lLABEL ðxÞ representing the membership
where lLABEL ðxÞ is the membership function operating on a fuzzy set. degree and LABEL representing its associated membership func-
This function returns the membership value of variable x. The sub- tion/fuzzy set. We then analyze these fuzzy values using relevant
script LABEL indicates the linguist label of the fuzzy set. In the above social science theories combined with fuzzy sets rules. It is impor-
examples, it has taken on labels ‘‘High Income’’ or ‘‘Low Income’’. tant to note that the analysis results at this step are fuzzy values.
To make sense of these fuzzy values in the context of social sci-
2.2. Fuzzy membership functions ences, we need to defuzzify the results. The defuzzification trans-
forms the results in fuzzy values to common crisp values so that
In fuzzy sets theory, every linguistic label A in a statement is they can be interpreted in the traditional non-fuzzy sense. There
matched with a fuzzy set X. A membership function mA ðÞ is de- are many different defuzzification approaches available. The most
fined on each fuzzy set X to return a membership degree in the unit frequently used approaches are Center of Area (COA) method and
interval [0, 1]. For an element x in X, the membership degree lA ðxÞ Mean of Maxima (MOM) method (Zimmermann, 1996). In the for-
quantifies the grade of membership of element x in fuzzy set X. By mer method, the center of the fuzzy area is taken as the output,
definition, element x can simultaneously belong to another fuzzy thus it is named as ‘‘Center of Area’’. In the later method, the mean
set Y with linguistic label B defined by membership function of the maximum of the fuzzy area is taken as the output, therefore,
mB ðÞ that characterizes the grade of membership of x in Y. it is heuristicly named as ‘‘Mean of Maxima’’.
Several different types of membership functions have been For the example shown in Fig. 2, the two grey pieces (results of
developed in fuzzy sets theory (Yen & Langari, 2004). The most the previous fuzzification procedure) form the fuzzy area. The
commonly used membership function is the evenly-spaced trian- maximal membership degree of this area is 0.8. If we apply the
gular function which allows for simple computation to transform COA method for defuzzification, the crisp output will be at the cen-
input variables into fuzzy variables. Fig. 1 depicts a series of trian- ter of the area, indicated by the solid arrow in the figure. If we use
gular membership functions that define the spectrum of the agree- the MOM method, the defuzzified result, indicated by the dashed
ment levels to the statement ‘‘The family income is high’’. Each arrow, will be the mean of the output range associated with the
membership function matches with a fuzzy set that represents maximal membership degree.

Membership Degree

SD D NN A SA

1.0

0.5

Family Income
0.0
20K 30K 40K 60K 80K 100K

Fig. 1. Triangular membership functions describing ‘‘The family income is high’’.

1612 Q. Li / Expert Systems with Applications 40 (2013) 1609–1618

Membership Degree

A SA
1.0 96K (Mr. X)

0.8

0.2 Family Income

0.0
20K 40K 60K 80K 100K

OutputCOA OutputMOM

Fig. 2. Fuzzifiction and defuzzification.

3. Development of the fuzzy Likert scale candidate’s position in the competence continuum more flexibly.
Unfortunately the author only proposed this concept without val-
3.1. Literature review idating the advantages of the fuzzy method over the traditional
Likert scale via either quantitative analysis or empirical studies.
Back in the late 1980s, Smithson (1987) made the major contri- Later, Lalla, Facchinett, and Mastroleo (2004) proposed a fuzzy
bution to introduce the fuzzy sets theory to the area of social measurement system for the evaluation of teaching activities in
sciences. However, unlike what has happened in engineering, an Italian university. The purpose of this teaching evaluation is
research of fuzzy sets in social sciences has not flourished since to obtain student feedbacks on different aspects of course organi-
then. Although the majority in social sciences has not paid much zation and teachers. The fuzzy measurement system is designed
attention to the potential of fuzzy sets, some researchers have based on three sequential hierarchical steps: (1) fuzzification of in-
strived to build connections between fuzzy sets (and other artifi- put variables, (2) construction and aggregation of control block
cial intelligence theories) and social sciences. Among these contri- rules, and (3) defuzzification of output variables. Among these
butions, several have developed fuzzy scaling algorithms for steps, (1) and (3) construct the base of the fuzzy Likert scale used
program evaluation. in the measurement system, and (2) establishes the causal rela-
Fourali (1997) proposed a fuzzy rating scale for the assessment tions between the input and output variables. The fuzzy Liket scale
of educational achievement. In a portfolio evidence based assess- transforms actual response values into fuzzy values so that the
ment, the evidence presented by a candidate is matched to the pre- fuzzy inference rules can be combined into the analysis of teaching
scribed standards of competence defined in achievement outcome evaluation. The authors found that their ‘‘fuzzy measurement
terms. An assessor then determines whether the portfolio evidence system yielded evaluation scores generally higher but sometimes
is adequate to allow the candidate to obtain a certificate of compe- lower than those obtained using the conventional 4- and 5-point
tence. Vagueness is usually unavoidable in this type of assessment Likert scales’’. However, based on this ambiguous result, one can
because assessors are not always clear where exactly a candidate hardly draw any conclusion about the worth of using fuzzy Likert
fits in a competence continuum. Instead of giving a single score scale, apart from acknowledging that the fuzzy sets theory pro-
in the traditional assessment way, the fuzzy rating scale allows vides an alternative way for scaling.
assessors to identify the range of possible scores that reflect a can- In additional to evaluations of learning and teaching, some fuz-
didate’s performance. Table 1 illustrates this scale on a competence zy scales have appeared in operational research literature. For
continuum ranging from 0 (for definitely incompetent) to 10 example, Chang and Yeh (2002) proposed a fuzzy multi-criteria
(for definitely competent). If an assessor feels that a candidate is analysis model for the evaluation of service quality for domestic
fairly but not totally competent on Criterion 1, for example, a series airlines. Hsu, Chang, and Hung (2007) developed a fuzzy scale to
of ratings of 6, 7, and 8 can be allocated on the scale. Fourali (1997) conduct a supplier evaluation for supply chain management. Lin
argued that this fuzzy rating scale is more accurate than the and Lee (2009) proposed a fuzzy assessment method for a sam-
traditional Likert scale because it allows assessors to locate a pling survey analysis in marketing research. Hu (2009) designed

Table 1
Fuzzy rating ascribed to each criterion of portfolio assessment Scale.
 Q. Li / Expert Systems with Applications 40 (2013) 1609–1618 1613

Table 2 degree 1.0 to agreement level SA. The responses of X, Y, and Z are
Survey form used in the fuzzy Liket scaling. summarized in Table 2.
Respondents Scales
3.2.2. Fuzzification
SD D NN A SA
Once the responses are collected, we need to transform these
Item: My family income is high data to fuzzy values. A set of isosceles triangular membership func-
Mr. X 0.8
tions evenly distributed along the input continuum, as shown in
Mr. Y 0.3
Mr. Z 1.0 Fig. 3, are adopted in this fuzzification procedure. The design of
etc. membership functions actually raises another challenging issue
in the development of the fuzzy Likert scale, which will be dis-
Instruction to respondents.
1. Please choose one agreement level SD, D, NN, A, or SA on the given scale. cussed in a separate work.
2. Please assign a membership degree between (0, 1) to the level chosen. At the end of the data collection, each response is recorded as a
pair of values: i.e., a selected agreement level and its membership
degree. Now these pairs of data need to be transformed into fuzzy
a fuzzy multiple-criteria decision making system for the assess- responses. Table 2 indicates that Mr. X’s response to the survey is
ment of service quality of travel websites. All these studies have {SA, 0.8}. Based on the membership functions given in Fig. 3, we
claimed that fuzzy scales are more accurate than traditional scales can obtain the fuzzy conversion: the response falls on A with a
due to the continuous nature of fuzzy sets. However, they have nei- membership degree 0.2, and on the other agreement levels with
ther attempted to support their conclusions with any mathemati- a membership degree 0. Mathematically, we can denote these re-
cal analysis, nor presented any comparison evaluation results sults as the following set of fuzzy values:
that demonstrate their fuzzy methods worked better.
mSD ðXÞ ¼ 0:0; mD ðXÞ ¼ 0:0; mNN ðXÞ ¼ 0:0; mA ðXÞ ¼ 0:2;
mSA ðXÞ ¼ 0:8
3.2. Development of the fuzzy Likert scale
Similarly, we can obtain the fuzzy responses for Mr. Y and Mr. Z as
The development of the novel fuzzy Likert scale is described in follows:
this section. The survey item ‘‘My family income is high’’ is used as
mSD ðXÞ ¼ 0:0; mD ðXÞ ¼ 0:0; mNN ðXÞ ¼ 0:0; mA ðXÞ ¼ 07;
an illustration example in the following discussion.
mSA ðXÞ ¼ 0:3

3.2.1. Response collection mSD ðXÞ ¼ 0:0; mD ðXÞ ¼ 0:0; mNN ðXÞ ¼ 0:0; mA ðXÞ ¼ 0:0;
As shown in Table 2, the fuzzy Likert scale takes the same for-
mSA ðXÞ ¼ 1:0
mat as we normally use in traditional scales. It requires respon-
dents to choose their agreement levels on the scale. In addition These response patterns are summarized in Table 3.
to this step, the respondents are now given an opportunity to as-
sign a membership degree to their chosen agreement level. This 3.2.3. Defuzzification
membership degree indicates how strongly a respondent complies The fuzzy values obtained previously are not directly interpret-
with the level of agreement. For example, if Mr. X has a family in- able. They need to be defuzzified back to real numbers. In most
come of $96K/year, he might have an agreement level between existing fuzzy logic applications, the defuzzification is operated
‘‘strongly agree’’ and ‘‘agree’’ to the survey item. Using the pro- on several fuzzy if-then rules (Yen & Langari, 2004). A fuzzy if-then
posed fuzzy Likert scale, Mr. X can choose an agreement level be- rule usually determines what fuzzy action to execute according to
tween ‘‘strongly agree’’ and ‘‘agree’’. If he tends to ‘‘strongly an input. In our development of the fuzzy Likert scale, the defuzz-
agree’’ more than ‘‘agree’’ this survey item, he might assign a high ification procedure only involves one if-then rule. For the specific
membership degree 0.8 to level SA. In the traditional 5-point Likert survey example considered in our paper, the if-then rule reads like
scale, however, Mr. X would have to choose either 5 or 4, which this: IF the family income is A, THEN the agreement level to the
does not reflect his exact agreement level. statement is B.
Now let us assume that Mr. Y has a lower family income of Since there is only one if-then rule, the common defuzzification
$84K/year. His agreement level to this survey item could be lower. methods are not applicable to solve our problem. A new defuzzifi-
He might assign a smaller membership degree 0.3 to the agree- cation algorithm is proposed as follows:
ment level SA. If there is another respondent Mr. Z who has a high- R
u  AðuÞdu
er family income of $100K/year, he might have a full agreement to Output ¼ UR ð1Þ
this survey item. In such a case, he could assign a full membership U
AðuÞdu

Membership Degree

100K
SD D NN A SA (Mr. Z)
1.0

0.8
96K
0.7
(Mr. X)
84K
0.5 (Mr.Y)

0.3
0.2 Family Income
0.0
20K 40K 60K 80K 100K

Fig. 3. Triangular isosceles membership functions.

1614 Q. Li / Expert Systems with Applications 40 (2013) 1609–1618

Table 3 While the traditional Likert scale can only measure agreement
Summary of X, Y, and Z’s responses. using the given integer scale points, the response level in between
Respondent i S D NN A SA Likert scale Fuzzy Likert scale scales will be discretized. In the following section, a Consensus
X .0 .0 .0 .2 .8 5 4.6 model, originated from communication studies (Tastle & Wierman,
Y .0 .0 .0 .7 .3 4 4.2 2007), will be applied to compare the measurement accuracies of
Z .0 .0 .0 .0 1.0 5 5.0 both Likert scales.
Mean .0 .0 .0 .3 .7 4.7 4.6

3.3.1. Model of consensus

In the area of communication, Consensus (Cns) is defined as the
where U indicates the universe of the fuzzy sets, u represents an agreement towards a declarative statement among the individuals
agreement level, and AðuÞ is the area of a fuzzy value. of a sample group (Tastle & Wierman, 2007). According to this def-
Now let us apply this algorithm to transform the fuzzy response inition, if an equal number of participants select their responses in
of Mr. X back to real values. As illustrated in Fig. 4, Mr. X’s response the two extreme categories in a Likert scale, i.e., strongly disagree
is represented by two grey areas ASA and AA. Eq. (2) can thus be ex- (SD) and strongly agree (SA), this group absolutely shows no con-
tended as follows: sensus, hence Cns = 0. If all participants select their responses on
uSA  ASA þ uA  AA the same category in a Likert scale, then this group shows a full
Output ¼ ð2Þ consensus, then Cns = 1. Other combinations of response patterns
ASA þ AA
will result in a consensus value within the interval [0, 1].
where uSA ¼ 5 and uA ¼ 4 are the values assigned to the agreement In (Tastle & Wierman, 2007), Cns is calculated as follow:
levels SA and S. Since both ASA and AA are trapezoids, we can use the   
following formula to calculate their areas: Xn xi  l 
x
CnsðXÞ ¼ 1 þ pi log2 1  ð4Þ
dx
1 i¼1
A ¼ ðbase1 þ base2 Þheight ð3Þ
2 where X is the response, n is the number of the categories in an
From Fig. 4, we can find that for AA, base1 = 40, base2 = 32, and ordinal scale, xi is the degree of agreement in category i, pi is the
height = 0.2 for, and for ASA, base1 = 20, base2 = 4, and height2 = 0.8. probability of the occurrence of xi ; dx ¼ X max  X min is the width of
Substitute these values into Eqs. (2) and (3), we obtain that categories on the measurement scale, and lx is the mean of the
Output = 4.6. overall agreement. Applying these definition to the traditional 5-
Applying the same computation procedure to the fuzzy re- point Likert scale for an example, we can find that n ¼ 5, i ranges
sponse of Mr. Y, we find the defuzzification result is Output = 4.2. from 1 to 5, and dx ¼ 5  1 ¼ 4.
Since Mr. Z assigned a full membership degree to SA, the deffuzifi-
cation is straightforwardly, i.e., Output = 5.0. The defuzzification 3.3.2. Accuracy comparison
outputs are summarized in Table 3. In this section, we will compare the measurement accuracies of
Both the fuzzification and defuzzification procedures described both Likert scales by studying their means, standard deviations
above seem to involve quite a bit computation. If a survey is (S.D.), and consensus values. Suppose that 100 respondents are
administered to a large sample, computation of these procedures randomly selected to rate the survey item ‘‘My family income is
could be burdensome. Fortunately many computer softwares such high’’. Their response patterns are recorded in Table 4.
as Fuzzy Logic Toolbox by The MathWorks or Excel by Microsoft In Cases A, B, and E, the traditional 5-point Likert scale (denoted
are available. Transformations between real and fuzzy values can as LS in the table) is administered to collect the responses. In Cases
be made easy by programming the fuzzification and defuzzifica- C, D, and F, the fuzzy Likert scale (denoted as FLS) is administered.
tion procedures. Columns SD, D, NN, A, and SA represent the five linguistic labels in
the Likert scale, and are assigned ordinal scale points 1, 2, 3, 4, and
3.3. Fuzzy Likert scaling vs. Likert scaling 5 respectively. Columns SD/D, D/NN, NN/A, and A/SA represent the
partial agreements to a linguistic label in the fuzzy Likert scale.
For comparison purpose, Table 3 presents both survey re- Although the degree of a partial agreement between any two con-
sponses based on the traditional and the fuzzy Likert scales. Since secutive linguistic labels can take on any decimal number between
the fuzzy Likert scale allows partial agreement to a scale point, the two whole scale points in practice, here the middle point is taken
response in this scale can be approximated to a decimal place. for a succinct explanation purpose. We therefore assign the

Membership Function

SD D NN A SA
96K (Mr. X)
1.0

0.8

ASA

0.2 Family Income

0.0
20K 40K 60K 80K 100K

AA Output

Fig. 4. Defuzzification of the response of Mr. X.

 Q. Li / Expert Systems with Applications 40 (2013) 1609–1618 1615

Table 4
Likert scale vs. fuzzy Likert Scale.

Cases Scales
SD SD/D D D/NN NN NN/A A A/SA SA Mean S.D. Cns
1 1.5 2 2.5 3 3.5 4 4.5 5 / / /
A(LS) 0 / 0 / 100 / 0 / 0 3.00 .00 1.00
B(LS) 50 / 0 / 0 / 0 / 50 3.00 2.01 .00
C(FLS) 0 0 0 30 40 30 0 0 0 3.00 .39 .88
D(FLS) 25 25 0 0 0 0 0 25 25 3.00 1.78 .16
E(LS) 15 / 15 / 40 / 15 / 15 3.00 1.23 .58
F(FLS) 10 10 10 10 20 10 10 10 10 3.00 1.23 .54

LS indicates the Likert scale, and FLS indicates the fuzzy Likert scale.

following values to these partial agreements: SD/D = 1.5, D/ 3.3.3. More discussion on mean, standard deviation and consensus
NN = 2.5, NN/A = 3.5 and A/SA = 4.5. It is noticed that all the mean values in Table 4 are identical for
In Case A, we assume that all respondents feel somewhere the different response cases. This is because mean measures the
around NN to the survey item. Given this closed format of the Lik- central tendency but not the dispersion of a data distribution. Since
ert scale, they will choose NN. The mean of the responses is 3.0, the response differences between the two Likert scales arise from
and there is no variation among the agreement levels, i.e., data dispersion, therefore mean is not a suitable statistical mea-
S.D. = 0.0. Although the respondents do not feel exactly the same, sure for the comparison of the two methods.
the collected responses indicate that the group has a full consensus Standard deviation is a measure of the variability or dispersion
on the survey item, i.e., Cns = 1.0. of a data set. A low standard deviation indicates that the data
Case C is the fuzzy variation of Case A. Since partial agreement points tend to be close to the mean. A high standard deviation indi-
can be detected in the fuzzy Likert scale, data dispersion around cates that the data are spread out over a large range of values. In
NN and response inconsensus can be captured. Assume that the re- Table 4, when the data dispersion increases from Case A to Case
sponses spread around NN symmetrically, with 30 choosing D/NN, C, S.D. is increased from 0.0 to .39. When the data dispersion de-
40 choosing NN, and another 30 choosing NN/A. The mean in this creases from Case B to Case D, S.D. is also decreased from 2.01 to
case is still 3.0, same as Case A, but S.D. = .39, indicating data var- 1.78. These trends have shown that the valued of S.D. is positively
iation. Slight disagreement is also detected, which leads to associated with the degree of data dispersion. However, standard
Cns = .88, meaning there is no longer full consensus among the deviation alone is not adequate to capture the consensus among
responses. dispersed data.
In Case B, we assume that half of the respondents more or less We are going to use Case E and its fuzzy variation Case F to ex-
feel strongly disagree with the survey question and another half plain the above statement. In Case E, assume that 40% of the
strongly agree with it. Using the traditional Likert scale, we find respondents select NN, and the rest respondents evenly select
mean = 3.0 and the variation among agreement levels is SD, D, A, and SA, with 15% on each scale. In Case F, the responses
S.D. = 2.01. Since there is no consensus among the responses, symmetrically spread around NN, with 20% choosing NN, and
Cns = 0.0. 10% on the rest of the scales. Comparing these two cases, intui-
Case D is the fuzzy variation of Case B. We can detect responses tively we can conclude that the consensus among the responses
near the two opposite end points. Assume that the responses in Case E is higher than that in Case F. Examining the Cns values
equally spread on SD, SD/D, A/SA, and SA, with each scale receiving in Table 4, we find that Cns = .58 in Case E, and Cns = .54 in Case
25 responses. The mean of this response pattern is again 3.0, but F. That means the measure of Cns does agree with our intuitive
the data dispersion is decreased to S.D. = 1.78 and the consensus expectation. However, examining the S.D. values, we find that
level is increased to Cns = .016 because half of the responses have S.D. = 1.23 in both Case E and Case F. That shows that the measure
moved towards the center agreement scale. of S.D. fails to distinguish the difference in consensus.
When the true responses spread around rather than exactly fall In summary, we conclude that both mean and standard devia-
on the scale NN, there should be no full consensus among re- tion are not the appropriate measures to examine consensus of re-
sponses and data variation should exhibit. As shown in Table 4, sponses. As Clegg (1998) pointed out, mean and standard deviation
the fuzzy Likert scale can more accurately capture the information are not suitable to measure the central tendency and the disper-
dispersion (with Cns = .88 and S.D. = .39 in Case C), however, the sion of a data set on ordinal Likert scales, where scores usually rep-
traditional Likert scale fails to detect the data variation since all re- resent linguistic statements. By definition, the calculation of a
sponses are forced to choose NN (with Cns = 1.0 and S.D. = 0.0 in mean or a standard deviation requires the presence of a ratio scale,
Case A). Similarly, when the true responses are distributed around which is built upon the assumption of equal intervals between
rather than fall exactly on the two extreme end scales SD and SA, scales with an absolute zero point. On the other hand, Consensus
the fuzzy Likert scale is able to capture some consensus among can appropriately measure the dispersion of a data set on ordinal
all responses and detect data variation in two response groups scales, where equal intervals with a true zero point is not implied
(with Cns = 0.16 and S.D. = 1.78 in Case D). However, the traditional (Tastle, Russell, & Wierman, 2005; Tastle & Wierman, 2007). There-
Likert scale fails to capture the hidden consensus and variation fore Cns is a reliable measure to be used to compare the measure-
(with Cns = 0.0 and S.D. = 2.01 in Case B). ment accuracies of both Likert scales.
By analyzing the above response patterns in Cases A, B, C and D,
we can conclude that the fuzzy Likert scale is a more accurate mea-
surement tool than the traditional Likert scale. The key leading to 4. Case study
the advantage of the fuzzy Likert scale is the partial agreement fea-
ture. Once the restriction of discretization of responses on scale In this section, the implementability of the fuzzy Likert scale
points is removed, less information will be lost or distorted during will be demonstrated through a simulation study based on a
measurement. low-birth-weight analysis. The advantages of the proposed method
1616 Q. Li / Expert Systems with Applications 40 (2013) 1609–1618

Table 5 From the table, it is observed that the deviance for the null
List of variables for the case study. model
Variables Identification code Abbreviation
LOW ¼ b0 ð5Þ
Low birth weight 0 = Birth LOW
weight > = 2500 g, is D0 ¼ 234:7, and the deviance for the full model
1 = Birth
weight < 2500 g LOW ¼ b0 þ b1 AGE þ b2 LWT þ b3 BLACK þ b4 OTHERS
AGE
Weight in pounds at the last menstrual LWT þ b5 SMOKE þ b6 PTL þ b7 HT þ b8 UI ð6Þ
period
Race 1 = White, RACE is Dm ¼ 196:8. Since the difference in deviance is 37.9 and df = 8,
2 = Black,
therefore p < .05, and the Likelihood Ratio test result is statistically
3 = Others
Smoking status during pregnancy 1 = Yes, 0 = No SMOKE
significant. Hosmer and Lemeshow (2000) then drew the following
History of premature labor 0 = None, 1 = One PTL conclusions. The logistic regression model in Eq. (6) provides an
and others adequate fit to the data. Since p > :05 for AGE, OTHERS, and UI,
History of hypertension 1 = Yes, 0 = No HT these are not significant predictors for output LWT. Among the sig-
1 = Yes, 0 = No UI
nificant predictors, PTL and HT are the most critical ones in detect-
ing LWT, as these two variables exhibit the largest Wald statistics.

will also be verified through a comparison study between two Lik-

4.3. Logistic regression – Data from the fuzzy Likert scale
ert scales.
This section presents the logistic regression based on the simu-
4.1. Data description
lated fuzzy responses.

The Low Birth Weight Data (LOWBWT.DAT) used in this simula-

tion is drawn from Hosmer and Lemeshow (2000). Low birth 4.3.1. Fuzzy data simulation
weight (baby weighting less than 2500 g) has been of concerns to In LOWBWT.DAT, the ordinal variables SMOKE, HT, and UI were
physicians for many years. Researchers have been trying to identify coded by integers, with 0 and 1 representing the absence and pres-
risk factors associated with giving birth to a low birth weight baby. ence of a behavior respectively. In reality, the occurrence of these
The LOWBWT.DAT was collected on 189 women, 59 of which had behaviors usually falls in between a total absence and a total pres-
low birth weight babies and 130 of which had normal birth weight ence. Coding these behaviors with a 2-point Likert scale or dichot-
babies. omous variable will inevitably leads to errors in measurement. For
Table 5 lists all the variables that were thought to be of impor- example, biochemical studies have shown that the impact of light
tance in the obstetrical literature. The goal of the obstetrical inves- smoking on giving birth to a light weighted baby is smaller than
tigation was to ascertain if these variables were importantly the impact of heavy smoking (da Veiga & Wilder, 2008). However,
associated with low birth weight in the population being served in the 2-point Likert scale, mothers with various smoking behav-
by the medical center where the data were collected. iors during pregnancy are all quantified as a full smoker. The differ-
ent effects between light and heavy smoking is not distinguished.
4.2. Logistic regression – Data from the Likert scale On the other hand, with the proposed fuzzy Likert scale, we can
precisely measure the partial presence of smoking behavior.
The original data set LOWBWT.DAT is collected based on the For an idea comparison study, we should conduct another sur-
Likert scale. The outcome LOW is a dichotomous variable with 1 vey using the fuzzy Likert scale to the same sample where the
representing the absence of low birth weight and 0 representing LOWBWT.DAT was drawn. However in reality this is not feasible.
the presence of low birth weight. Hosmer and Lemeshow (2000) So in this study we are going to simulate the fuzzy responses of
suggested running a multiple logistic regression model to fit the the mothers by performing the following fuzzification procedure
data. Among all the variables measured, AGE and LWT are contin- to the original data set:
uous values, and the rest are categorical or ordinal values. Categor- IF VAR ¼ 0; THEN VAR f ¼ Random Number ½0:0; 0:5Þ
ical variable RACE was dummy-coded into groups BLACK, WHITE,
and OTHERS (including races other than black and white) for mod-
IF VAR ¼ 1; THEN VAR f ¼ Random Number ½0:5; 1:0
el fitting. The results of the logistic regression provided by Hosmer
and Lemeshow (2000) are presented in Table 6. where VAR is the original response value and VAR_f is its fuzz-
ified result. When the traditional Likert scale was used for survey, if
a mother did not have certain behavior or that behavior was less
Table 6
frequently present during her pregnancy, she would likely choose
Results of the logistic regression – Data from the Likert scale.
0 as her response. Similarly, if a behavior was more frequently or
IV’s B S.E. Wald p-value Exp(B) 95.0% C.I .for EXP(B) always occurred, the mother would select 1 as the response. In
Lower Upper our simulation, we replace VAR = 0 with a random value
AGE .038 .038 .997 .318 .963 .894 1.037 2 ½0:0; 0:5Þ and VAR = 1 with a random value 2 ½0:5; 1:0 to simu-
LWT .015 .007 4.485 .034 .985 .972 .999 late the fuzzy response. We assume these fuzzy values reflect the
BLACK 1.213 .532 5.187 .023 3.363 1.184 9.549 actual presence frequencies of the behaviors.
OTHERS .804 .448 3.215 .073 2.235 .928 5.382 In addition to the ordinal variables SMOKE, HT, and UI, the
SMOKE .846 .408 4.302 .038 2.331 1.048 5.187
PTL 1.222 .463 6.963 .008 3.393 1.369 8.409
dummy coded variable BLACK in the data set also needs to be fuzz-
HT 1.839 .703 6.836 .009 6.288 1.585 24.954 ified. RACE usually appears as a categorical variable in the domains
UI .711 .463 2.358 .125 2.036 .822 5.047 of social and political studies, however in medical and genetic
Constant .637 1.230 .268 .605 1.891 studies, we should remove the political and social influences and
Deviance for the null model = 234.7. treat it as a ratio variable. In this simulation, the degree of being
Deviance for the full model = 196.8. ‘‘Black’’ has been quantified in accordance with the amount of
 Q. Li / Expert Systems with Applications 40 (2013) 1609–1618 1617

Table 7 of fuzzy sets for transforming social science methodology has not
Results of the logistic regression – Data from the fuzzy Likert scale. been well recognized yet.
IV’s B S.E. Wald p-value Exp(B) 95.0% C.I.for EXP(B) The major contribution of this paper lies in a successful explo-
Lower Upper ration in bridging the fuzzy sets theory with social science practice.
In particular, it has incorporated the concepts of partial member-
AGE .050 .038 1.756 .185 .951 .883 1.024
LWT .011 .007 2.664 .103 .989 .975 1.002
ship and defuzzification from the fuzzy sets theory into the con-
BLACK_f 2.492 .899 7.677 .006 12.081 2.073 70.393 struction of the Likert scaling for improved measurement
OTHERS .882 .424 4.326 .038 2.416 1.052 5.547 accuracy. This new fuzzy Likert scale has overcome the information
SMOKE_f 1.711 .661 6.704 .010 5.535 1.516 20.211 lost problem arising from the ordinal natural of the traditional Lik-
PTL 1.266 .456 7.707 .005 3.547 1.451 8.672
ert method. It has also reduced the information miss-matching ef-
HT_f 1.193 .922 1.673 .196 3.297 .541 20.099
UI_f 1.119 .767 2.132 .144 3.063 .682 13.761 fect due to the closed-response format in the traditional scale. The
Constant .517 1.291 .160 .689 .596 analysis of the consensus values has proven that the proposed fuz-
zy Likert scale can provide a more accurate measurement result
Deviance for the null model = 234.7.
Deviance for the full model = 197.0. than the traditional Likert method. Compared with other alterna-
tive Likert scales existing in current literature, this fuzzy Likert
scale has an advantage that it does not increase the length of the
the black gene of a mother, with 0 meaning an absolute absence of answer options, nor need complicated phrase completion format.
black gene, and 1 meaning a pure black gene. The simple answer format can therefore avoid administrative chal-
lenges and will not invoke the typical ‘‘primacy effects’’ like other
4.3.2. Logistic regression result methods. The applicability and effectiveness of the fuzzy Likert
The simulated fuzzy data has been fitted into the logistic regres- scale has been demonstrated via a case study investigating the
sion model given below, where label ‘‘_f’’ indicates a fuzzified relationship between low birth weight and mother’s behavior dur-
variable: ing pregnancy. The case study results have provided strong evi-
dences that the proposed fuzzy Likert scale has the ability to
LOW ¼ b0 þ b1 AGE þ b2 LWT þ b3 BLACK f þ b4 OTHERS identify the measurement and analysis errors caused by the lost
þ b5 SMOKE f þ b6 PTL þ b7 HT f þ b8 UI f ð7Þ and distorted information in the traditional Likert scale.

The regression results are presented in Table 7. The deviance differ-

ence between the null model in Eq. (5) and the full model in Eq. (7) References
is now 37.7. Since df = 8, and p < .05, we can conclude that the Like- Acharya, N. K., Lee, Y. D., & Im, H. M. (2006). Conflicting factors in construction
lihood Ratio test result is again significant. That means that the new projects: Korean perspective. Engineering, Construction and Architectural
logistic regression model in Eq. (7) fits the fuzzified data well. Management, 13, 543.
Albaum, G. (1997). The Likert scale revisited: An alternate version. Journal of the
However, some regression results based on the fuzzy data are
Market Research Society, 39, 331–349.
different from the results using the original data. According to Bañuelas, R., & Antony, J. (2007). Application of stochastic analytic hierarchy
the p values in Table 7, LWT and HT are no longer significant pre- process within a domestic appliance manufacturer. The Journal of the
dictors for outcome LWT, whereas OTHERS becomes a significant Operational Research Society, 58, 29–39.
Brown, J. D. (2000). Using surveys in language programs. Cambridge: Cambridge
predictor. Furthermore, according to the Wald statistics, BLACK_f University Press.
and PTL are the strongest predictors in detecting low birth weight. Brown, C. N., Guskiewicz, K. M., & Bleiberg, J. (2007). Athlete characteristics and
Most noticeably, the strongest predictor HT (Wald = 6.836, outcome cores for computerized neuropsychological assessment: A preliminary
analysis. Journal of Athletic Training, 42, 515–524.
p = .009) based on the original data regression becomes a non-sig- Buncher, C. R., & Tsay, J. Y. (Eds.). (2006). Statistics in the pharmaceutical industry.
nificant predictor HT_f (Wald = 1.673, p = .144). New York: Chapman and Hall–CRC. 3rd Ed.
Chan, J. C. (1991). Response-order effect in Likert-type scales. Educational Psychology
Measurements, 51, 531–540.
4.4. Discussion Chang, L. (1994). A psychometric evaluation of 4-poing and 6-point Likert-type
scales in relation to reliability and validity. Applied Psychological Measurement,
The above logistic regressions based on the original and fuzzy 18, 205–215.
Chang, Y. H., & Yeh, C. H. (2002). A survey analysis of service quality for domestic
data sets have exhibited quite difference results. Since in Section 3 airlines. European Journal of Operational Research, 139, 166–177.
we proved that the fuzzy Likert scale is a more accurate psycho- Clegg, C. (1998). Simple statistics. Cambridge: Cambridge University Press.
metric tool, we believe that the regression results in Table 7 can Cohen, L., Manion, L., & Morrison, K. (2000). Research methods in education. London:
Routledge Falmer.
better explain the relationships between the low birth weight
Copeland, L. (2003). An exploration of the problems faced by young women living in
(LOW) of a new born and the mother behaviors during pregnancy. disadvantaged circumstances if they want to give up smoking: Can more be
However, we do not imply that the results in Table 7 should be rec- done at general practice level? Family Practice, 20, 393.
ommended to doctors because this is only a simulation study. The da Veiga, P. V., & Wilder, R. P. (2008). Maternal smoking during pregnancy and
birthweight: A propensity score matching approach. Maternal and Child Health
statistical analysis results would have more practical value if we Journal, 12, 194–203.
could collect a set of real fuzzy responses by using the proposed Fourali, C. (1997). Using fuzzy logic in educational measurement: The case of
fuzzy Likert scale. portfolio assessment. Evaluation and Research in Education, 11, 129–148.
Gien, D., Jacqmart, S., Seklouli, A., & Barad, M. (2003). An approach based on fuzzy
sets for manufacturing system design. International Journal of Production
5. Conclusion Research, 41, 315–335.
Hodge, D. R., & Gillespie, D. (2003). Phrase completions: An alternative to Likert
scales. Social Work Research, 27, 45–55.
Since the concept of fuzzy sets conceived in 1965, it has been Hosmer, D. W., & Lemeshow, S. (2000). Applied logistic regression. John Wiley & Sons,
widely applied in areas like automation engineering and opera- Inc.
Hsu, C.F., Chang, B., Hung, H.F. (2007). Applying SVM to build supplier evaluation
tional research to deal with a variety of practical or abstract prob-
model – Comparing Likert scale and fuzzy scale. Paper presented at the 2007
lems involving ambiguity. The advantages of fuzzy sets theory IEEE Industrial Engineering and Engineering Management, Singapore.
come from its feature in combining both qualitative and quantita- Hu, Y. C. (2009). Fuzzy multiple-criteria decision making in the determination of
tive analyses in one single algorithm. It provides scientists with a critical criteria for assessing service quality of travel websites. Expert Systems
with Application, 36, 6439–6445.
very powerful new mathematical tool that is half-verbal-concep- Jamieson, S. (2004). Likert scales: How to (ab)use them. Medical Education, 38,
tual and half-mathematical-analytical. Unfortunately, the potential 1217–1218.
1618 Q. Li / Expert Systems with Applications 40 (2013) 1609–1618

Jamshidi, M., Titli, A., Zadeh, L., & Boverie, S. (1997). Applications of fuzzy logic: Seal, M. (2007). Patient advocacy and advance care planning in the acute hospital
Towards high machine intelligence quotient systems. Prentice Hall. setting. Australian Journal of Advanced Nursing, 24, 29–37.
Kiedrowski, P. J. (2006). Quantitative assessment of a Senge learning organization Smithson, M. J. (1987). Fuzzy set analysis for the behavioral and social sciences (Recent
intervention. The Learning Organization, 13, 369–383. Research in Psychology). New York: Springer-Verlag.
Lalla, M., Facchinett, G., & Mastroleo, G. (2004). Ordinal scales and fuzzy set systems Smithson, M., & Verkuilen, J. (2006). Fuzzy set theory: applications in the social
to measure agreement: An application to the evaluation of teaching activity. sciences. Quantitative Applications in the social sciences series. Belmont, CA: Sage.
Quality & Quantity, 38, 577–601. Springer, D. W. (1998). Validation of the adolescent concerns evaluation: Detecting
Li, Q., McCoach, D. B., Swaminathan, H., & Tang, J. (2008). Development of an indicators of runaway behavior in adolescents. Social Work Research, 22,
instrument to measure perspectives of engineering education among college 241–253.
students. Journal of Engineering Education, 97, 47–56. Tastle, W. J., Russell, J., Wierman, M. J. (2005). A new measure to analyze student
Lin, L., & Lee, H. M. (2009). Fuzzy assessment method on sampling survey analysis. performance using the Likert scale. In The Proceedings of the Information Systems
Expert Systems with Applications, 36, 5955–5961. Education Conference 22, Columbus OH.
Lindhorst, K., Corby, L., Roberts, S., & Zeiler, S. (2007). Rural consumers’ attitudes Tastle, W. J., & Wierman, M. J. (2007). Consensus and dissention: A measure of
towards nutrition labeling. Canadian Journal of Dietetic Practice and Research, 68, ordinal dispersion. International Journal of Approximate Reasoning, 45, 531–545.
146–150. Yen, J., & Langari, R. (2004). Fuzzy logic: Intelligence, control and information. India:
Orr, M. G., & Ohlsson, S. (2005). Relationship between complexity and liking as a Pearson Education.
function of expertise. Music Perception, 22, 583. Yue, W., Wilson, C. S., & Boller, F. (2007). Peer assessment of journal quality in
Orvik, J. M. (1972). Social desirability for individual, his group, and society. clinical neurology. Journal of Medical Library Association, 95, 70–77.
Multivariate Behavioral Research, 7, 3–32. Zadeh, L. A. (1965). Fuzzy sets. Information and Control, 6, 338–353.
Pett, M. A. (1997). Nonparametric statistics for health care research: Statistics for small Zimmermann, H. J. (1983). Using fuzzy sets in operational research. European
samples and unusual distribution. London: SAGE Publication. Journal of Operational Research, 13, 201–216.
Postma, T. C. (2007). Screening tools to prioritize routine dental care in an Zimmermann, H. J. (1987). Fuzzy sets, decision making, and expert. Kluwer Academic
institutional environment. Military Medicine, 172, 1287–1293. Publishers.
Ragin, C. C (2000). Fuzzy-Set Social Science. London: The University of Chicago Press, Zimmermann, H. J. (1996). Fuzzy set theory – and its applications. Kluwer Academic
Ltd. Publishers.
Ray, J. J. (1990). Acquiescence and problems with forced-choice scale. Journal of Zopounidis, C., Pardalos, P. M., & Baourakis, G. (Eds.). (2001). Fuzzy sets in
Social Psychology, 130, 397–399. management. World Scientific Publishing Co Economics and Marketing.
Russell, C. J., & Bobko, P. (1992). Moderated regression analysis and Likert scales:
Too coarse for comfort. Journal of Applied Psychology, 77, 336–342.