European Journal of Operational Research 246 (2015) 1–19

Contents lists available at ScienceDirect

European Journal of Operational Research

journal homepage: www.elsevier.com/locate/ejor

Invited Review

A review of theory and practice in scientometrics ✩

John Mingers a,∗, Loet Leydesdorff b
a
Kent Business School, University of Kent, Parkwood Road, Canterbury CT7 2PE, UK
b
Amsterdam School of Communication Research (ASCoR), University of Amsterdam, PO Box 15793, 1001 NG Amsterdam, The Netherlands

a r t i c l e i n f o a b s t r a c t

Article history: Scientometrics is the study of the quantitative aspects of the process of science as a communication system.
Received 14 January 2015 It is centrally, but not only, concerned with the analysis of citations in the academic literature. In recent
Accepted 1 April 2015
years it has come to play a major role in the measurement and evaluation of research performance. In this
Available online 9 April 2015
review we consider: the historical development of scientometrics, sources of citation data, citation metrics
Keywords: and the “laws” of scientometrics, normalisation, journal impact factors and other journal metrics, visualising
Altmetrics and mapping science, evaluation and policy, and future developments.
Citations © 2015 Elsevier B.V. and Association of European Operational Research Societies (EURO) within the
H-index International Federation of Operational Research Societies (IFORS). All rights reserved.
Impact factor
Normalisation

1. History and development of scientometrics gies on the Web drawing on bibliometric and informetric approaches
(Björneborn & Ingwersen, 2004, p. 1217; Thelwall & Vaughan, 2004;
Scientometrics was first defined by Nalimov (1971, p. 2) as devel- Thelwall, Vaughan, & Björneborn, 2005). This field mainly concerns
oping “the quantitative methods of the research on the development the analysis of web pages as if they were documents.
of science as an informational process”. It can be considered as the Altmetrics – “The study and use of scholarly impact measures based
study of the quantitative aspects of science and technology seen as on activity in online tools and environments” (Priem, 2014, p. 266).
a process of communication. Some of the main themes include ways Also called Scientometrics 2.0, this field replaces journal citations
of measuring research quality and impact, understanding the pro- with impacts in social networking tools such as views, downloads,
cesses of citations, mapping scientific fields and the use of indicators “likes”, blogs, Twitter, Mendelay, CiteULike.
in research policy and management. Scientometrics focuses on com- In this review we concentrate on scientometrics as that is the field
munication in the sciences, the social sciences, and the humanities most directly concerned with the exploration and evaluation of scien-
among several related fields: tific research. In fact, traditionally these fields have concentrated on
Bibliometrics – “The application of mathematics and statistical the observable or measurable aspects of communications – external
methods to books and other media of communication” (Pritchard, borrowings of books rather than in-library usage; citations of papers
1969, p. 349). This is the original area of study covering books and rather than their reading – but currently online access and downloads
publications generally. The term “bibliometrics” was first proposed provide new modes of usage and this leads to the developments in
by Otlet (1934); see also Rousseau (2014). webometrics and altmetrics that will be discussed later. In this sec-
Informetrics – “The study of the application of mathematical meth- tion we describe the history and development of scientometrics (de
ods to the objects of information science” (Nacke, 1979, p. 220). Per- Bellis, 2014; Leydesdorff & Milojevic, 2015) and in the next sections
haps the most general field covering all types of information regard- explore the main research areas and issues.
less of form or origin (Bar-Ilan, 2008; Egghe & Rousseau, 1990; Egghe Whilst scientometrics can, and to some extent does, study many
& Rousseau, 1988; Wilson, 1999). other aspects of the dynamics of science and technology, in prac-
Webometrics – “The study of the quantitative aspects of the con- tice it has developed around one core notion – that of the citation.
struction and use of information resources, structures and technolo- The act of citing another person’s research provides the necessary
linkages between people, ideas, journals and institutions to consti-
tute an empirical field or network that can be analysed quantitatively.
✩
We would like to acknowledge helpful comments from Anne-Wil Harzing, David Furthermore, the citation also provides a linkage in time – between
Pendlebury, Ronald Rousseau and an anonymous referee. the previous publications of its references and the later appearance
∗
Corresponding author. Tel.: +44 1227 8240080.
of its citations. This in turn stems largely from the work of one per-
E-mail addresses: [email protected] (J. Mingers), [email protected]
(L. Leydesdorff). son – Eugene Garfield – who identified the importance of the citation

http://dx.doi.org/10.1016/j.ejor.2015.04.002
0377-2217/© 2015 Elsevier B.V. and Association of European Operational Research Societies (EURO) within the International Federation of Operational Research Societies (IFORS).
All rights reserved.
2 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19

and then promulgated the idea of the Science Citation Index (SCI) in but there are many criticisms of its biases. In terms of journal evalu-
the 1950s (and the company the Institute for Scientific Information, ISI, ation, several new metrics have been developed such as SNIP (Moed,
to maintain it) as a database for capturing citations (Garfield, 1955; 2010b) and SCImago Journal Rank (SJR) (González-Pereira, Guerrero-
Garfield, 1979).1 Its initial purpose was not research evaluation, but Bote, & Moya-Anegón, 2010; Guerrero-Bote & Moya-Anegón, 2012)
rather help for researchers to search the literature more effectively – which aim to take into account the differential citation behaviours of
citations could work well as index or search terms, and also enabled different disciplines, e.g., some areas of science such as biomedicine
unfamiliar authors to be discovered. The SCI was soon joined by the cite very highly and have many authors per paper; other areas, par-
Social Sciences Citation Index (SSCI, in 1973) and the Arts & Humani- ticularly some of the social sciences, mathematics and the humanities
ties Citation Index (A&HCI; since 1978), and eventually taken over by do not cite so highly.
Thomson Corporation who converted it into the Web of Science as part A third, technical, development has been in the mapping and vi-
of their Web of Knowledge platform.2 In 2013, the SCI covered 8539 sualization of bibliometric networks. This idea was also initiated by
journals, the SSCI 3080 journals, and the A&HCI approximately 1700 Garfield who developed the concept of “historiographs” (Garfield,
journals. Sher, & Thorpie, 1964), maps of connections between key papers,
The SCI was soon recognised as having great value for the empirical to reconstruct the intellectual forebears of an important discovery.
study of the practice of science. The historian, Price (1963, 1965), was This was followed by co-citation analysis which used multivariate
one of the first to see the importance of networks of papers and techniques such as factor analysis, multi-dimensional scaling and
authors and also began to analyse scientometric processes, leading to cluster analysis to analyse and map the networks of highly related
the idea of cumulative advantage (Price, 1976), a version of “success papers which pointed the way to identifying research domains and
to the successful” (Senge, 1990) or “success breeds success (SBS)” frontiers (Marshakova, 1973; Small, 1973). And also co-word analysis
also known as the Matthew3 effect (Merton, 1968; Merton, 1988). that looked at word pairs from titles, abstracts or keywords and drew
Price identified some of the key problems that would be addressed on the actor network theory of Callon and Latour (Callon, Courtial,
by scientometricians: mapping the “invisible colleges” (Crane, 1972) Turner, & Bauin, 1983). New algorithms and mapping techniques such
informally linking highly cited researchers at the research frontiers (cf as the Blondel algorithm (Blondel, Guillaume, Lambiotte, & Lefebvre,
co-authorship networks and co-citation analysis (Marshakova, 1973; 2008) and the Pajek mapping software have greatly enhanced the vi-
Small, 1973)): studying the links between productivity and quality sualization of high-dimensional datasets (de Nooy, Mrvar, & Batgelj,
in that the most productive are often the most highly cited (cf the 2011).
h-index); and investigating citation practices in different fields (cf But perhaps the most significant change, which has taken sciento-
normalization). In 1978, Robert K. Merton, a major sociologist, was metrics from relative obscurity as a statistical branch of information
one of the editors of a volume called Towards a Metric of Science: The science to playing a major, and often much criticised, role within the
Advent of Science Indicators (Elkana, Lederberg, Merton, Thackray, & social and political processes of the academic community, is the drive
Zuckerman, 1978) which explored many of these new approaches. of governments and official bodies to monitor, record and evaluate
Scientometrics was also developing as a discipline with the advent of research performance. This itself is an effect of the neo-liberal agenda
the journal Scientometrics in 1978; a research unit in the Hungarian of “new public management” (NPM) (Ferlie, Ashburner, Fitzgerald, &
Academy of Sciences and scientific conferences and associations. Pettigrew, 1996) and its requirements of transparency and account-
At the same time as scientometrics research programs were be- ability. This occurs at multiple levels – individuals, departments and
ginning, the first links to research evaluation and the use of citation research groups, institutions and, of course, journals – and has signif-
analysis in policy making also occurred. For example, the ISI data was icant consequences in terms of jobs and promotion, research grants,
included in the (US) National Science Board’s Science Indicators Re- and league tables. In the past, to the extent that this occurred it did
ports in 1972 and was used by the OECD. Garfield and Sher (1963) so through a process of peer review with the obvious drawbacks of
developed a measure for evaluating journals – the impact factor (IF) – subjectivity, favouritism and conservatism (Bornmann, 2011; Irvine,
that has been for many years a standard despite its many flaws. Jour- Martin, Peacock, & Turner, 1985). But now, partly on cost grounds,
nals with this specific policy focus appeared such as Research Policy, scientometrics are being called into play and the rather ironic result
R&D Management and Research Evaluation. is that instead of merely reflecting or mapping a pre-given reality,
During the 1990s and 2000s several developments have occurred. scientometric methods are actually shaping that reality through their
The availability and coverage of the citation databases has increased performative effects on academics and researchers (Wouters, 2014).
immensely. The WoS itself includes many more journals and also con- At the same time, the discipline of science studies itself has bi- (or
ference proceedings, although its coverage in the social sciences and tri-) furcated into at least three elements – the quantitative study of
humanities is still limited. It also does not yet cover books adequately science indicators and their behaviour, analysis and metrication from
although there are moves in that direction. A rival, Scopus, has also a positivist perspective. A more qualitative, sociology-of-science, ap-
appeared from the publisher Elsevier. However, the most interest- proach that studies the social and political processes lying behind
ing challenger is Google Scholar which works in an entirely different the generation and effects of citations, generally from a constructivist
way – searching the web rather than collecting data directly. While perspective. And a third stream of research that is interested in policy
this extension of coverage is valuable, it also leads to problems of implications and draws on both the other two.
comparison with quite different results appearing depending on the Finally, in this brief overview, we must mention the advent of
databases used. the Web and social networking. This has brought in the possi-
Secondly, a whole new range of metrics has appeared superseding, bility of alternatives to citations as ways of measuring impact (if
in some ways, the original ones such as total number of citations and not quality) such as downloads, views, “tweets”, “likes”, and men-
citations per paper (cpp). The h-index (Costas & Bordons, 2007; Egghe, tions in blogs. Together, these are known as “altmetrics” (Priem,
2010; Glänzel, 2006; Hirsch, 2005; Mingers, 2008b; Mingers, Macri, & 2014), and while they are currently underdeveloped, they may well
Petrovici, 2012) is one that has become particularly prominent, now come to rival citations in the future. There are also academic social
available automatically in the databases. It is transparent and robust networking sites such as ResearchGate (www.researchgate.net),
CiteULike (citeulike.org), academia.edu (www.academia.edu), RePEc
1
(repec.org) and Mendeley (www.mendeley.com) which in some cases
It was first realised in 1964.
2
Web of Knowledge has now reverted to Web of Science.
have their own research metrics. Google Scholar can produce pro-
3
Named after St Matthew (25:29): “For unto everyone that hath shall be given . . . files of researchers, including their h-index, and Publish or Per-
from him that hath not shall be taken away”. ish (Harzing, 2007) enhances searches of Scholar with the Harzing
 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19 3

website (www.harzing.com) being a repository for multiple journals seen as the major form of research output. Secondly, there is a greater
ranking lists in the field of business and management. prevalence of the “lone scholar” as opposed to the team approach
that is necessary in the experimental sciences and which results in
2. Sources of citations data a greater number of publications (and hence citations) overall. As an
extreme example, a paper in Physics Letters B (Aad et al., 2012) in
Clearly for the quantitative analysis of citations to be successful, 2012 announcing the discovery of the Higgs Boson has 2932 authors
there must be comprehensive and accurate sources of citation data. and already has over 4000 citations. These outliers can distort biblio-
The major source of citations in the past was the Thomson Reuters ISI metrics analyses as we shall see (Cronin, 2001). Thirdly, a significant
Web of Science (WoS) which is a specialised database covering all the number of social science and humanities journals are not, or have not
papers in around 12,000 journals.4 It also covers conference proceed- chosen to become, included in WoS, the accounting and finance field
ings5 and is beginning to cover books.6 Since 2004, a very similar rival being a prime example. Finally, in social science and humanities a
database is available from Elsevier called Scopus7 that covers 20,000 greater proportion of publications are directed at the general public
journals and also conferences and books. Scopus retrieves back un- or specialised constituencies such as practitioners and these “trade”
til 1996, while WoS is available for all years since 1900. These two publications or reports are not included in the databases.
databases have been the traditional source for most major sciento- There have also been many comparisons of WoS, Scopus and
metrics exercises, for example by the Centre for Science and Technology Google Scholar across a range of disciplines (Adriaanse & Rensleigh,
Studies (CWTS) which has specialised access to them. More recently 2013; Amara & Landry, 2012; Franceschet, 2010; García-Pérez, 2010;
(2004), an alternative source has been provided by Google Scholar Harzing & van der Wal, 2008; Jacso, 2005; Meho & Rogers, 2008; Meho
(GS). This works in an entirely different way, by searching the Web & Yang, 2007). The general conclusions of these studies are:
for documents that have references to papers and books rather than • That the coverage of research outputs, including books and re-
inputting data from journals. It is best accessed through a software
ports, is much higher in GS, usually around 90 percent, and that
program called Publish or Perish.8 Both of these resources are free
this is reasonably constant across the subjects. This means that GS
while access to WoS and Scopus are subscription-based and offer dif-
has a comparatively greater advantage in the non-science subjects
ferent levels of accessibility depending on the amount of payment
where Scopus and WoS are weak.
thus leading to differential access for researchers. • Partly, but not wholly, because of the coverage, GS generates a sig-
Many studies have shown that the coverage of WoS and Scopus
nificantly greater number of citations for any particular work. This
differs significantly between different fields, particularly between the
can range from two times to five times as many. This is because
natural sciences, where coverage is very good, the social sciences
the citations come from a wide range of sources, not being limited
where it is moderate and variable, and the arts and humanities where
to the journals that are included in the other databases.
it is generally poor (HEFCE, 20089 ; Larivière, Archambault, Gingras, & • However, the data quality in GS is very poor with many entries
Vignola-Gagné, 2006; Mahdi, D’Este, & Neely, 2008; Moed & Visser,
being duplicated because of small differences in spellings or dates
2008). In contrast, the coverage of GS is generally higher, and does not
and many of the citations coming from a variety of non-research
differ so much between subject areas, but the reliability and quality
sources. With regard to the last point, it could be argued that the
of its data can be poor (Amara & Landry, 2012).
type of citation does not necessarily matter – it is still impact.
Van Leeuwen (2006), in a study of Delft University between 1991
and 2001 found that in fields such as architecture and technology, Typical of these comparisons is Mingers and Lipitakis (2010) who
policy and management the proportion of publication in WoS and reviewed all the publications of three UK business schools from 1980
the proportion of references to ISI material was under 30 percent to 2008. Of the 4600 publications in total, 3023 were found in GS,
while for applied science it was between 70 percent and 80 percent. but only 1004 in WoS. None of the books, book chapters, conference
Across the social sciences, the proportions varied between 20 per- papers or working papers were in WoS.10 In terms of number of ci-
cent for political science and 50 percent for psychology. Mahdi et al. tations, the overall mean cites per paper (cpp) in GS was 14.7 but
(2008) studied the results of the 2001 RAE in the UK and found that only 8.4 in WoS. It was also found that these rates varied consider-
while 89 percent of the outputs in biomedicine were in WoS, the ably between fields in business and management, a topic to be taken
figures for social science and arts & humanities were 35 percent and up in the section on normalization. When taken down to the level of
13 percent respectively. CWTS (Moed, Visser, & Buter, 2008) was com- individual researchers the variation was even more noticeable both
missioned to analyse the 2001 RAE and found that the proportions of in terms of the proportion of outputs in WoS and the average number
outputs contained in WoS and Scopus respectively were: Economics of citations. For example, the most prolific researcher had 109 pub-
(66 percent, 72 percent), Business and Management (38 percent, 46 lications. Ninety-two percent were in GS, but only 40 percent were
percent), Library and Information Science (32 percent, 34 percent) in WoS. The cpp in GS was 31.5, but in WoS it was 12.3. Generally,
and Accounting and Finance (22 percent, 35 percent). where papers were included in both sources GS cites were around
There are several reasons for the differential coverage in these three times greater.
databases (Archambault, Vignola-Gagné, Côté, Larivière, & Gingras, With regard to data quality, Garcia-Perez (2010) studied papers of
2006; Larivière et al., 2006; Nederhof, 2006) and we should also note psychologists in WoS, GS, and PsycINFO.11 GS recorded more publica-
that the problem is not just the publications that are not included, tions and citations than either of the other sources, but also had a large
but also that the publications that are included have lower citations proportion of incorrect citations (16.5 percent) in comparison with
recorded since many of the citing sources are not themselves included. 1 percent or less in the other sources. The errors included not sup-
The first reason is that in science almost all research publications ap- plying usable links to the citation, phantom citations, duplicate links
pear in journal papers (which are largely included in the databases), pointing to the same citing paper, or reprints published in different
but in the social sciences and even more so in humanities books are sources. Adriaanse and Rensleigh (2013) studied environmental sci-
entists in WoS, Scopus and GS and made a comprehensive record of
4
http://wokinfo.com/essays/journal-selection-process/.
5
http://wokinfo.com/products_tools/multidisciplinary/webofscience/cpci/cpciessay/. 10
Most studies do not include WoS for books, which is still developing (Leydesdorff
6
http://thomsonreuters.com/book-citation-index/. & Felt, 2012).
7
http://www.elsevier.com/online-tools/scopus/content-overview. 11
PsycINFO is an abstracting and indexing database of the American Psychological
8
http://www.harzing.com/pop.htm. Association with more than 3 million records devoted to peer-reviewed literature in
9
Higher Education Funding Council for England. the behavioural sciences and mental health.
4 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19

Table 1
Comparison of WoS and GS for one of the authors.

Cites from outputs in WoS using WoS Cites from all sources using GS
n, c, h, cpp n, c, h, cpp

Cites to outputs in WoS 88, 1684, 21, 19.1 87, 4890, 31, 56.2
Cites to all outputs 349, 3796, 30, 10.8 316, 13,063, 48, 41.3

n = no. of papers, c = no. of citations, h = h-index (defined below), cpp = cites per paper.

the inconsistencies that occurred in all three across all bibliometric proportion of producers are responsible for a high proportion of out-
record fields – data export, author, article title, page numbers, refer- puts. This also means that the statistical distributions associated with
ences and document type. There were clear differences with GS having these phenomena are generally highly skewed. It should be said that
14.0 percent inconsistencies, WoS 5.4 percent, and Scopus only 0.4 the original works were quite approximate and actually provided few
percent. Similar problems with GS were also found by Jacso (2008) examples. They have been formalised by later researchers.
and Harzing and van der Wal (2008). Lotka (1926) studied the frequency distribution of numbers of
To summarise this section, there is general agreement at this point publications per author, concluding that “the number of authors mak-
in time that bibliometric data from WoS or Scopus is adequate to con- ing n contributions is about 1/n2 of those making one” from which can
duct research evaluations in the natural and formal sciences where be derived Price’s (1963) “square root law” that “half the scientific
the coverage of publications is high, but it is not adequate in the social papers are contributed by the top square root of the total number of
sciences or humanities, although, of course, it can be used as an aid scientific authors”. So, typically there are 1/4 the number of authors
to peer review in these areas (Abramo & D’Angelo, 2011; Abramo, publishing two papers than one; 1/9 publishing three papers and so
D’Angelo, & Di Costa, 2011; van Raan, 2005b). GS is more comprehen- on. Lotka’s Law generates the following distribution:
sive across all areas but suffers from poor data, especially in terms of  
  6
multiple versions of the same paper, and also has limitations on data P X=k = · k−2 where k = 1, 2, . . .
access – no more than 1000 results per query. This particularly affects
π2
the calculation of cites per paper (because the number of papers is Glänzel and Schubert (1985) showed that a special case of the
the divisor) but it does not affect the h-index which only includes the Waring distribution satisfies the square root law.
top h papers. Bradford (1934) hypothesised that if one ranks journals in terms
These varied sources do pose the problem that the number of of number of articles they publish on a particular subject, then there
papers and citations may vary significantly and one needs to be aware will be a core that publish the most. If you then group the rest into
of this in interpreting any metrics. To illustrate this with a simple zones such that each zone has about the same number of articles,
example, we have looked up data for one of the authors on WoS and then the number of journals in each zone follows this law:
GS. The results are shown in Table 1.
Nn = kn N0
The first thing to note is that there are two different ways of ac-
cessing citation data in WoS. (a) One can do an author search and where k = Bradford coefficient, N0 = number in core zone, Nn = jour-
find all their papers, and then do a citation analysis of those papers. nals in the nth zone.
This generates the citations from WoS papers to WoS papers. (b) One Thus the number of journals needed to publish the same number
can do a cited reference search on an author. This generates all the of articles grows with a power law.
citations from papers in WoS to the author’s work whether the cited Zipf (1936) studied the frequency of words in a text and postu-
work is in WoS or not. This therefore generates a much larger number lated that the rank of the frequency of a word and the actual frequency,
of cited publications and a larger number of citations for them. The when multiplied together, are a constant. That is, the number of oc-
results are shown in the first column of Table 1. Option (a) finds 88 currences is inversely related to the rank of the frequency. In a simple
papers in WoS and 1684 citations for them from WoS papers. The cor- case, the most frequent word will occur twice as often as the second
responding h-index is 21. Option (b) finds 349 (!) papers with 3796 most frequent, and three times as often as the third.
citations and an h-index of 30. The 349 papers include many cases of
illegitimate duplicates just as does GS. If we repeat the search in GS,
rf (r) = C
we find a total of 316 cited items (cf 349) with 13,063 citations giving r is the rank, f (r) is the frequency of that rank, C is a constant
an h-index of 48. If we include only the papers that are in WoS we
find 87 of the 88, but with 4890 citations and an h-index of 31. So, 1
one could justifiably argue for an h-index ranging from 21 to 48, and f (r) = C
r
a cpp from 10.8 to 56.2.
More generally:
1/rs
f (r) =   N is the number of items, s is a parameter
3. Metrics and the “laws” of scientometrics 1N 1
ns

In this section we will consider the main areas of scientometrics The Zipf distribution has been found to apply in many other con-
analysis – indicators of productivity and indicators of citation impact. texts such as the size of city by population. All three of these be-
haviours ultimately rest on the same cumulative advantage mecha-
3.1. Indicators of productivity nisms (SBS) mentioned above and, indeed, under certain conditions
all three can be shown to be mathematically equivalent and a conse-
Some of the very early work, from the 1920s onwards, concerned quence of SBS (Egghe, 2005, Chaps. 2 and 3).
productivity in terms of the number of papers produced by an author However, empirical data on the number of publications per year
or research unit; the number of papers journals produce on a partic- by, for example, a particular author shows that the Lotka distribution
ular subject; and the number of key words that texts generate. They by itself is too simplistic as it does not take into account productivity
all point to a similar phenomenon – the Paretian one that a small varying over time (including periods of inactivity) or subject. One
 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19 5

approach is to model the process as a mixture of distributions (Sichel,

1985). For example, we could assume that the number of papers per
year followed a Poisson distribution with parameter λ, but that the
parameter itself varied with a particular distribution depending on
age, activity, discipline. If we assume that the parameter follows a
Gamma distribution, then this mixture results in a negative-binomial
which has been found to have a good empirical fit (Mingers & Burrell,
2006). Moreover, this approach (Burrell, 2003) shows that SBS is a
consequence of the underlying model.

3.2. Indicators of impact: citations

Fig. 1. Histograms for papers published in 1990 in six management science journals,
We should begin by noting that the whole idea of the citation be- from (Mingers & Burrell, 2006).
ing a fundamental indicator of impact, let alone quality, is itself the
subject of considerable debate. This concerns: the reasons for citing Table 2
others’ work, Weinstock (1971) lists 15, or not citing it; the meaning Summary statistics for citations in six OR journals 1990–2004, from (Mingers & Burrell,
or interpretation to be given to citations (Cozzens, 1989; Day, 2014; 2006).

Leydesdorff, 1998); their place within scientific culture (Wouters, JORS Omega EJOR Dec Sci Ops Res Man Sci
2014); and the practical problems and biases of citation analysis
Actual mean 7.3 7.2 11.3 11.1 14.6 38.6
(Chapman, 1989). This wider context will be discussed later; this
Actual SD 17.9 15.5 19.0 14.0 28.6 42.4
section will concentrate on the technical aspects of citation metrics. Percent zero cites 18 22 14 12 10 5
The basic unit of analysis is a collection of papers (or more gener- Max cites 176 87 140 66 277 181
ally research outputs including books, reports, etc., but as pointed out
in Section 2 the main databases only cover journal papers) and the
number of citations they have received over a certain period of time. also have modes (except ManSci) at zero, i.e., many papers have never
There are three possible situations: a fixed collection observed over been cited in all that time. The proportion of zero cites varies from 5
a fixed period of time (e.g., computing JIFs); a fixed collection over percent in Management Science to 22 percent in Omega.
an extending period of time (e.g., computing JIFs over different time The issue of zero cites is of concern. On the one hand, that a paper
windows); or a collection that is developing over time (e.g., observing has never been cited does not imply that it is of zero quality, espe-
the dynamic behaviour of citations over time (Mingers, 2008a)). cially when it has been through rigorous reviewing processes in a top
journal, which is evidence that citations are not synonymous with
3.2.1. Citation patterns quality. On the other hand, as Braun, Glänzel, and Schubert (1985)
If we look at the number of citations per year received by a paper argue, a paper that has never been cited must at the least be discon-
over time it shows a typical birth–death process. Initially there are few nected from the field in question. The mean cites per paper (over 15
citations; then the number increases to a maximum; finally they die years) vary considerably between journals from 7.2 to 38.6 showing
away as the content becomes obsolete. Note that the total number the major differences between journals (to be covered in a later sec-
of citations can only increase over time but the rate of increase of tion), although it is difficult to disentangle whether this is because
citations can decrease as obsolescence sets in. There are many variants of the intrinsically better quality of the papers or simply the reputa-
to this basic pattern, for example “shooting stars” that are highly tion of the journal. Bornmann, Leydesdorff, and Wang (2013a) found
cited but die quickly, and “sleeping beauties” that are ahead of their that the journal can be considered as a significant co-variate in the
time (van Raan, 2004). There are also significantly different patterns prediction of citation impact.
of citation behaviour between disciplines that will be discussed in Obsolescence can be incorporated into the model by including a
the normalization section. There are several statistical models of this time-based function in the distribution. This would generally be an
process. Glänzel and Schoepflin (1995) use a linear birth process; S-shaped curve that would alter the value of λ over time, but there are
Egghe (2000) assumed citations were exponential and deterministic. many possibilities (Meade & Islam, 1998) and the empirical results did
Perhaps the most usual is to conceptualise the process as basically not identify any particular one although the gamma and the Weibull
random from year to year but with some underlying mean (λ) and distributions provided the best fits. It is also possible to statistically
use the Poisson distribution. There can then be two extensions – the predict how many additional citations will be generated if a particular
move from a single paper to a collection of papers with differing mean number have been received so far. The main results are that, at time
rates (Burrell, 2001), and the incorporation of obsolescence in the rate t, the future citations are a linear function of the citations received so
of citations (Burrell, 2002, 2003). far, and the slope of the increment line decreases over the lifetime of
If we assume a Gamma distribution for the variability of the pa- the papers. These results applied to collections of papers, but do not
rameter λ, then the result is a negative binomial of the form: seem to apply to the dynamics of individual papers.
  v  r In a further study of the same data set, the citation patterns of the
r+v−1 α α
P (Xt = r) = 1− , r = 0, 1, 2, . . . individual papers were modelled (Mingers, 2008a). The main conclu-
v−1 α+t α+t sions were twofold: (i) that individual papers were highly variable
With mean = vt/α variance = vt(t + α )/α 2 where v and α are param- and it was almost impossible to predict the final number of citations
eters to be determined empirically. based on the number in the early years, in fact up to about 10 years.
The negative binomial is a highly skewed distribution which, as This was partly because of sleeping beauty and shooting star effects.
we have seen, is generally the case with bibliometric data. Mingers (ii) The time period for papers to mature was quite long – the maxi-
and Burrell (2006) tested the fit on a sample of 600 papers published mum citations were not reached until years eight and nine, and many
in 1990 in six MS/OR journals – Management Science, Operations Re- papers were still being strongly cited at the end of 14 years. This is
search, Omega, EJOR, JORS and Decision Sciences – looking at 14 years very different from the natural sciences where the pace of citations
of citations. Histograms are shown in Fig. 1 and summary statistics in is very much quicker for most papers (Baumgartner & Leydesdorff,
Table 2. As can be seen, the distributions are highly skewed, and they 2014).
6 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19

If we wish to use citations as a basis for comparative evaluation, and number of papers. These h papers are generally called the h-core.
whether of researchers, journals or departments, we must consider The h-core is not uniquely defined where more than one paper has
influences on citations other than pure impact or quality. The first, and h citations. The h-index ignores all the other papers below h, and
most obvious, is simply the number of papers generating a particular it also ignores the actual number of citations received above h. The
total of citations. A journal or department publishing 100 papers per advantages are:
year would expect more citations than one publishing 20. For this rea-
son the main comparative indicator that has been used traditionally
• It combines both productivity and impact in a single measure that
is the mean cites per paper (cpp) or raw impact per paper (RIP) dur- is easily understood and very intuitive.
ing the time period of study. This was the basis of the Leiden (CWTS)
• It is easily calculated just knowing the number of citations ei-
“crown indicator” measure for evaluating research units suitably nor- ther from WoS, Scopus or Google Scholar. Indeed, all three now
malised against other factors (Waltman, van Eck, van Leeuwen, Visser, routinely calculate it.
& van Raan, 2010, 2011). We should note that this is the opposite of
• It can be applied at different levels – researcher, journal or depart-
total citations – it pays no attention at all to the number of papers, so ment.
a researcher with a cpp of 20 could have one paper, or one hundred
• It is objective and a good comparator within a discipline where
papers each with 20 citations. citation rates are similar.
These other factors include: the general disciplinary area – nat-
• It is robust to poor data since it ignores the lower down papers
ural science, social science or humanities; particular fields such as where the problems usually occur. This is particularly important
biomedicine (high) or mathematics (low); the type of paper (reviews if using GS.
are high); the degree of generality of the paper (i.e., of interest to a However, many limitations have been identified including some
large or small audience); reputational effects such as the journal, the that affect all citation based measures (e.g., the problem of different
author, or the institution; the language; the region or country (gen- scientific areas, and ensuring correctness of data), and a range of
erally the United States has the highest number of researchers and modifications have been suggested (Bornmann, Mutz, & Daniel, 2008).
therefore citations) as well as the actual content and quality of the
paper. • The first is that the metric is insensitive to the actual numbers of ci-
Another interesting issue is whether all citations should be worth tations received by the papers in the h-core. Thus two researchers
the same? There are three distinct factors here – the number of au- (or journals) with the same h-index could have dramatically dif-
thors of a paper, the number of references in the citing paper, and ferent actual numbers of citations. Egghe (2006) has suggested the
the quality of the citing journal. In terms of numbers of authors, the g-index as a way of compensating for this. “A set of papers has a
sciences generally have many collaborators within an experimental g-index of g if g is the highest rank such that the top g papers
or laboratory setting who all get credited. Comparing this with the have, together, at least g2 citations” (p. 132). The fundamental
situation of a single author who has done all the work, should not idea is that the h-core papers must have at least h2 citations be-
the citations coming to that paper be spread among the authors? The tween them although in practice they may have many more. At
extreme example mentioned above concerning the single paper an- first sight, the use of the square rather than the cube or any other
nouncing the Higgs Boson actually had a significant effect on the posi- power seems arbitrary but it is a nice choice since the definition
tion of several universities in the 2014 Times Higher World University can be re-written so that "the top g papers have an average number
Ranking (Holmes, 2014). The paper, with 2896 “authors” affiliated to of citations at least g", which is much more intuitively appealing.
228 institutions, had received 1631 citations within a year. All of the g is at least as large as h.
institutions received full credit for this and for some, who only had a • The more they have, the larger g will become and so it will to some
relatively small number of papers, it made a huge difference (Aziz & extent reflect the total number of citations. The disadvantage of
Rozing, 2013; Moed, 2000). this metric is that it is less intuitively obvious than the h-index.
The number of references in the citing paper can be a form Another alternative is the e-index proposed by Zheng (2009).
of normalisation (fractional counting of citations) (Leydesdorff &
Bornmann, 2011a) which will be discussed below. Taking into ac- There are several other proposals that measure statistics on the
count the quality of the citing journal gives rise to new indicators that papers in the h-core, for example:
will be discussed in the section on journals. • The a-index (Jin, 2006; Rousseau, 2006) which is the mean number
of citations of the papers in the h-core.
3.2.2. The h-index • The m-index (Bornmann et al., 2008) which is the median of the
We have seen that the total number of citations, as a metric, is papers in the h-core since the data is always highly skewed. Cur-
strongly affected by the number of papers but does not provide any rently Google Scholar Metrics12 implements a 5-year h-index and
information on this. At the opposite extreme, the cpp is totally in- 5-year m-index.
sensitive to productivity. In 2005, a new metric was proposed by • The r-index (Jin, 2007) which is the square root of the sum of
Hirsch (2005) that combined in a single, easy to understand, num- the citations of the h-core papers. This is because the a-index
ber both impact (citations) and productivity (papers). The h-index actually penalises better researchers as the number of citations are
has been hugely influential since then, generating an entire litera- divided by h, which will be bigger for better scientists. A further
ture of its own. Currently his paper has well over 4000 citations in development is the ar-index (Jin, Liang, Rousseau, & Egghe, 2007)
GS. In this section we will only be able to summarise the main ad- which is a variant of the r-index also taking into account the age
vantages and disadvantages, for more detailed reviews see Alonso, of the papers.
Cabrerizo, Herrera-Viedma, and Herrera (2009), Bornman and Daniel • The h-index is strictly increasing and strongly related to the time
(2005), Costas and Bordons (2007), Glänzel (2006) and for mathe- the publications have existed. This biases it against young re-
matical properties see Glänzel (2006) and Franceschini and Maisano searchers. It also continues increasing even after a researcher has
(2010). retired. Data on this is available from Liang (2006) who investi-
The h index is defined as: “a scientist has index h if h of his or her gated the actual sequence of h values over time for the top scien-
Np papers have at least h citations each and the other (Np – h) papers tists included in Hirsch’s sample. A proposed way round this is to
have <= h citations each” (p. 16569).
So h represents the top h papers, all of which have at least h
citations. This one number thus combines both number of citations 12
http://scholar.google.co.uk/citations?view_op=top_venues&hl=en.
 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19 7

consider the h-rate (Burrell, 2007), that is the h-index at time t di- Overall, the h-index may be somewhat crude in compressing infor-
vided by the years since the researcher’s first publication. This was mation about a researcher into a single number, and it should always
also proposed by Hirsch as the m parameter in his original paper. be used for evaluation purposes in combination with other measures
Values of 2 or 3 indicate scientists who are both highly productive or peer judgement but it has clearly become well-established in prac-
and well cited. tice being available in all the citation databases.
• The h-index does not discriminate well since it only employs inte- Another approach is the use of percentile measures which we will
ger values. Given that most researchers may well have h-indexes cover in the next section.
between 10 and 30, many will share the same value. Guns and
Rousseau (2009) have investigated real and rational variants of 4. Normalisation methods
both g and h.
• As with all citation-based indicators, they need to be normalised In considering the factors that affect the number of citations that
in some way to citation rates of the field. Iglesias and Pecharromán papers receive, there are many to do with the individual paper – con-
(2007) collected, from WoS, the mean citations per paper in each tent, type of paper, quality, author, or institution (Mingers & Xu, 2010)
year from 1995 to 2005 for 21 different scientific fields. The totals – but underlying those there are clear disciplinary differences that are
ranged from under 2.5 for computer science and mathematics to hugely significant. As mentioned above, Iglesias and Pecharromán
over 24 for molecular biology. From this data they constructed a (2007) found that mean citation rates in molecular biology were ten
table of normalisation factors to be applied to the h-index depend- times those in computer science. The problem is not just between
ing on the field and also the total number of papers published by disciplines but also within disciplines such as business and manage-
the researcher. A similar issue concerns the number of authors. ment which encompass different types of research fields. Mingers and
The sciences tend to have more authors per paper than the social Leydesdorff (2014) found that management and strategy papers av-
sciences and humanities and this generates more papers and more eraged nearly four times as many citations as public administration.
citations. Batista, Campiteli, Kinouchi, and Martinez (2006) devel- This means that comparisons between researchers, journals or insti-
oped the hI-index as the h-index divided by the mean number of tutions across fields will not be meaningful without some form of nor-
authors of the h-core papers. They also claim that this accounts to malisation. It is also important to normalise for time period because
some extent for the citation differences between disciplines. Pub- the number of citations always increases over time (Leydesdorff,
lish or Perish also corrects for authors by dividing the citations Bornmann, Mutz, & Opthof, 2011; Waltman & van Eck, 2013b).
for each paper by the number of authors before calculating the
hI, norm-index. This metric has been further normalised to take 4.1. Field classification normalisation
into account the career length of the author (Harzing, Alakangas,
& Adams, 2014). The most well established methodology for evaluating research
• The h-index is dependent on or limited by the total number of centres was developed by the Centre for Science and Technology Stud-
publications and this is a disadvantage for researchers who are ies (CWTS) at Leiden University and is known as the crown indicator
highly cited but for a small number of publications (Costas & or Leiden Ranking Methodology (LRM) (van Raan, 2005c). Essentially,
Bordons, 2007). For example, Aguillo13 has compiled a list of the this method compares the number of citations received by the publi-
most highly cited researchers in GS according to the h-index (382 cations of a research unit over a particular time period with that which
with h’s of 100 or more). A notable absentee is Thomas Kuhn, would be expected, on a world-wide basis across the appropriate field
one of the most influential researchers of the last 50 years with and for the appropriate publication date. In this way, it normalises the
his concept of a scientific paradigm. His book (Kuhn, 1962) alone citation rates for the department to rates for its whole field. Typically,
has (14/11/14) 74,000 citations which, if the table were ranked in top departments may have citation rates that are three or four times
terms of total citations would put him in the top 100. His actual the field average. Leiden also produces a ranking of world universi-
total citations are around 115,000 citations putting him in the top ties based on bibliometric methods that will be discussed elsewhere
20. However, his h-index is only 64. This example shows how dif- (Waltman et al., 2012).
ferent metrics can lead to quite extreme results – on the h-index This is the traditional “crown indicator”, but this approach to nor-
he is nowhere; on total citations, in the top 20; and on cites per malisation has been criticised (Leydesdorff & Opthof, 2011; Lundberg,
paper probably first! 2007; Opthof & Leydesdorff, 2010) and an alternative has been used in
several cases (Cambell et al., 2008; Rehn & Kronman, 2008; Van Veller,
There have been many comparisons of the h-index with other in-
Gerritsma, Van der Togt, Leon, & Van Zeist, 2009). This has generated
dicators. Hirsch himself performed an empirical test of the accuracy
considerable debate in the literature (Bornmann, 2010; Bornmann &
of indicators in predicting the future success of researchers and con-
Mutz, 2011; Moed, 2010a; van Raan, van Leeuwen, Visser, van Eck, &
cluded, perhaps unsurprisingly, that the h-index was most accurate
Waltman, 2011; Waltman et al., 2010, 2011). The main criticism con-
(Hirsch, 2007). This was in contrast to other studies such as Bornmann
cerns the order of calculation in the indicator but the use of a mean
and Daniel (2007), Lehmann, Jackson, and Lautrup (2006), van Raan
when citation distributions are highly skewed is also a concern. It is
(2005a). Generally, such comparisons show that the h-index is highly
argued that, mathematically, it is wrong to sum the actual and ex-
correlated with other bibliometric indicators, but more so with mea-
pected numbers of citations separately and then divide them. Rather,
sures of productivity such as number of papers and total number of
the division should be performed first, for each paper, and then these
citations, rather than with citations per paper which is more a mea-
ratios should be averaged. In the latter case you get a proper statistic
sure of pure impact (Alonso et al., 2009; Costas & Bordons, 2007;
rather than merely a quotient. It might be thought that this is purely a
Todeschini, 2011).
technical issue, but it can affect the results significantly. In particular,
There have been several studies of the use of the h-index
the older CWTS method tends to weight more highly cited publica-
in business and management fields such as information systems
tions from fields with high citation numbers whereas the new one
(Oppenheim, 2007; Truex III, Cuellar, & Takeda, 2009), management
weights them equally. Also, the older method is not consistent in its
science (Mingers, 2008b; Mingers et al., 2012), consumer research
ranking of institutions when both improve equally in terms of pub-
(Saad, 2006), marketing (Moussa & Touzani, 2010) and business
lications and citations. Eventually this was accepted by CWTS, and
(Harzing & Van der Wal, 2009).
Waltman et al. (2010, 2011) (from CTWS) have produced both theo-
retical and empirical comparisons of the two methods and concluded
13
http://www.webometrics.info/en/node/58. that the newer one is theoretically preferably but does not make much
8 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19

difference in practice. The new method is called the “mean normalised normalising methods with the new CWTS crown indicator (MNCS)
citation score” (MNCS). Gingras et al. (2011) commented that the “al- and concluded that the source normalisation methods were prefer-
ternative” method was not alternative but in fact the correct way to able to the field classification approach, and that of them, the audi-
normalise, and had been in use elsewhere for fifteen years. ence factor and revised SNIP were best. This was especially noticeable
for interdisciplinary journals. The fractional counting method did not
4.2. Source normalisation fully eliminate disciplinary differences (Radicchi & Castellano, 2012)
and also did not account for citation age.
The normalisation method just discussed normalised citations
against other citations, but an alternative approach was suggested, 4.3. Percentile-based approaches
initially by Zitt and Small (2008) in their “audience factor”, which
considers the sources of citations, that is the reference lists of cit- We have already mentioned that there is a general statistical
ing papers, rather than citations themselves. This general approach problem with metrics that are based on the mean number of cita-
is gaining popularity and is also known as the “citing-side approach” tions, which is that citations distributions are always highly skewed
(Zitt, 2011), source normalisation (Moed, 2010c) (SNIP), fractional (Seglen, 1992) and this invalidates the mean as a measure of central
counting of citations (Leydesdorff & Opthof, 2010) and a priori nor- tendency; the median is better. There is also the problem of ratios of
malisation (Glänzel, Schubert, Thijs, & Debackere, 2011). means discussed above. A non-parametric alternative based on per-
The essential difference in this approach is that the reference set centiles (an extension of the median) has been suggested for research
of journals is not defined in advance, according to WoS or Scopus cat- groups (Bornmann & Mutz, 2011), individual scientists (Leydesdorff,
egories, but rather is defined at the time specifically for the collection Bornmann, Mutz et al., 2011) and journals (Leydesdorff & Bornmann,
of papers being evaluated (whether that is papers from a journal, de- 2011b). This is also used by the US National Science board in their
partment, or individual). It consists of all the papers, in the given time Science and Engineering Indicators.
window, that cite papers in the target set. Each collection of papers The method works as follows:
will, therefore, have its own unique reference set and it will be the
lists of references from those papers that will be used for normalisa- 1. For each paper to be evaluated, a reference set of papers published
tion. This approach has obvious advantages – it avoids the use of WoS in the same year, of the same type and belonging to the same WoS
categories which are ad hoc and outdated (Leydesdorff & Bornmann, category is determined.
2015; Mingers & Leydesdorff, 2014) and it allows for journals that are 2. These are rank ordered and split into percentile rank (PR) classes,
interdisciplinary and that would therefore be referenced by journals for example the top 1 percent (99th percentile), 5 percent,
from a range of fields. 10 percent, 25 percent, 50 percent and below 50 percent. For each
Having determined the reference set of papers, the methods then PR, the minimum number of citations necessary to get into the
differ in how they employ the number of references in calculating a class is noted.15 Based on its citations, the paper is then assigned
metric. The audience factor (Zitt, 2011; Zitt & Small, 2008) works at to one of the classes. This particular classification is known as 6PR.
the level of a citing journal. It calculates a weight for citations from 3. The procedure is repeated for all the target papers and the results
that journal based on the ratio between the average number of active are then summated, giving the overall percentage of papers in each
references14 in all journals to the average number of references in of the PR classes. The resulting distributions can be statistically
the citing journal. This ratio will be larger for journals that have few tested against both the field reference values and against other
references compared to the average because they are in less dense competitor journals or departments.16
citation fields. Citations to the target (cited) papers are then weighted The particular categories used above are only one possible set
using the calculated weights which should equalise for the citation (Bornmann, Leydesdorff, & Mutz, 2013) – others in use are [10 per-
density of the citing journals. cent, 90 percent] and [0.01 percent, 0.1 percent, 1 percent, 10 percent,
Fractional counting of citations (Leydesdorff & Bornmann, 2011a; 20 percent, 50 percent] (used in ISI Essential Science Indicators) and
Leydesdorff & Opthof, 2010; Leydesdorff, Radicchi, Bornmann, the full 100 percentiles (100PR) (Bornmann et al., 2013b; Leydesdorff,
Castellano, & Nooy, 2013; Small & Sweeney, 1985; Zitt & Bassecoulard, Bornmann, Mutz et al., 2011). This approach provides a lot of infor-
1994) begins at the level of an individual citation and the paper which mation about the proportions of papers at different levels, but it is
produced it. Instead of counting each citation as one, it counts it as a still useful to be able to summarise performance in a single value. The
fraction of the number of references in the citing paper. This if a ci- suggested method is to calculate a mean of the ranks weighted by the
tation comes from a paper with m references, the citation will have a proportion of papers in each. The minimum is 1, if all papers are in
value of 1/m. It is then legitimate to add all these fractionated citations the lowest rank; the maximum is 6 if they are all in the top percentile.
to give the total citation value for the cited paper. An advantage of The field average will be 1.91 – (0.01, 0.04, 0.05, 0.15, 0.25, 0.50) × (6,
this approach is that statistical significance tests can be performed on 5, 4, 3, 2, 1) – so a value above that is better than the field average.
the results. One issue is whether all references are included (which A variation of this metric has been developed as an alternative to the
Leydesdorff et al. do) or whether only the active references should journal impact factor (JIF) called I3 (Leydesdorff, 2012; Leydesdorff &
be counted. The third method is essentially that which underlies the Bornmann, 2011b). Instead of multiplying the percentile ranks by the
SNIP indicator for journals (Moed, 2010b) which will be discussed in proportion of papers in each class, they are multiplied by the actual
Section 5. In contrast to fractional counting, it forms a ratio of the numbers of papers in each class thus giving a measure that combines
mean number of citations to the journal to the mean number of refer- productivity with citation impact. In the original, the 100PR classifi-
ences in the citing journals. A later version of SNIP (Waltman, van Eck, cation was used but other ones are equally possible.
van Leeuwen, & Visser, 2013) used the harmonic mean to calculate The main drawback of this method is that it relies on the field def-
the average number of references and in this form it is essentially initions in WoS or another database which are unreliable, especially
the same as fractional counting except for an additional factor to take for interdisciplinary journals. It might be possible to combine it with
account of papers with no active citations. some form of source normalisation (Colliander, 2014).
Some empirical reviews of these approaches have been carried out.
Waltman and van Eck (2013a, 2013b) compared the three source-
15
There are several technical problems to be dealt with in operationalising these
classes (Bornmann, Leydesdorff, & Mutz, 2013; Bornmann et al., 2013b).
14 16
An “active” reference is one that is to a paper included in the database (e.g., WoS) Using Dunn’s test or the Mann–Whitney U test (Leydesdorff, Bornmann, Mutz et
within the time window. Other references are then ignored as “non-source references”. al., 2011).
 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19 9

5. Indicators of journal quality: the impact factor and other previous year so that the IF of this journal is boosted (DeMaria
metrics et al., 2008).
• If used for assessing individual researchers or papers the JIF is un-
So far, we have considered the impact of individual papers or representative (Oswald, 2007). As Fig. 1 shows, the distribution of
researchers, but of equal importance is the impact of journals in terms citations within a journal is highly skewed and so the mean value
of library’s decisions about which journals to take (less important in will be distorted by a few highly cited papers, and not represent
the age of e-journals), authors’ decisions about where to submit their the significant number that may never be cited at all.
papers, and in subsequent judgements of the quality of the paper.
Indeed journal ranking lists such as the UK Association of Business In response to criticisms of the JIF, several more sophisticated
Schools’ (ABS) has a huge effect on research behaviour (Mingers & metrics have been developed, although the price for sophistication is
Willmott, 2013). Until recently, the journal impact factor (JIF) has been complexity of calculation and a lack of intuitiveness in what it means
the pre-eminent measure. This was originally created by Garfield and (see Table 3).
Sher (1963) as a simple way of choosing journals for their SCI but, once The first metrics we will consider take into account not just the
it was routinely produced in WoS (who have copyright to producing quantity of citations but also their quality in terms of the prestige
it), it became a standard. Garfield recognised its limitations and also of the citing journal. They are based on iterative algorithms over a
recommended a metric called the “cited half-life” which is a measure network, like Googles’s PageRank, that initially assign all journals an
of how long citations last for. Specifically, it is the median age of equal amount of prestige and then iterate the solution based on the
papers cited in a particular year, so a journal that has a cited half- number of citations (the links) between the journals (nodes) until a
life of 5 years means that 50 percent of the citations are to papers steady state is reached. The first such was developed by Pinsky and
published in the last 5 years. Narin (1976) but that had calculation problems. Since then, Page,
JIF is simply the mean citations per paper for a journal over a two Brin, Motwani, and Winograd (1999) and Ma, Guan, and Zhao (2008)
year period. For example, the 2014 JIF is the number of citations in have an algorithm based directly on PageRank but adapted to cita-
2014 to papers published in a journal in 2012 and 2013, divided by tions; Bergstrom (2007) has developed the Eigenfactor which is im-
the number of such papers. WoS also published a 5-year JIF because plemented in WoS; and Gonzalez-Pereira et al. (2010) have developed
in many disciplines 2 years is too short a time period. It is generally SCImago Journal Rank (SJR) which is implemented in Scopus.
agreed that the JIF has few benefits for evaluating research, but many The Eigenfactor is based on the notion of a researcher taking a ran-
deficiencies (Brumback, 2008; Cameron, 2005; Seglen, 1997; Vanclay, dom walk following citations from one paper to the next, measuring
2012). Even Garfield (1998) has warned about its over-use.17 the relative frequency of occurrence of each journal as a measure of
prestige. It explicitly excludes journal self-citations unlike most other
• JIF depends heavily on the research field. As we have already seen, metrics. Its values tend to be very small, for example the largest in
there are large differences in the publishing and citing habits of the management field is Management Science with a value of 0.03
different disciplines and this is reflected in huge differences in while the 20th is 0.008 which is not very meaningful. The Eigenfac-
JIF values. Looking at the WoS journal citation reports 2013, in tor measures the total number of citations and so is affected by the
the area of cell biology the top journal has a JIF of 36.5 and the total number of papers published by a journal. A related metric
20th one of 9.8. Nature has a JIF of 42.4. In contrast, in the field of is the article influence score (AI) which is the Eigenfactor divided
management, the top journal (Academy of Management Review) by the proportion of papers in the database belonging to a particu-
is 7.8 and the 20th is only 2.9. Many journals have JIFs of less lar journal over 5 years. It can therefore be equated to a 5-year JIF.
than 1. Thus, it is not appropriate to compare JIFs across fields A value of 1.0 shows that the journal has average influence; values
(even within business and management) without some form of greater than 1.0 show greater influence. We can see that in cell biology
additional normalisation. the largest AI is 22.1 compared with 6.6 in management. Fersht (2009)
• The 2-year window. This is a very short time period for many and Davies (2008) argue empirically, that the Eigenfactor gives essen-
disciplines, especially given the lead time between submitting a tially the same information as total citations as it is size-dependent,
paper and having it published which may itself be 2 years. In but West, Bergstrom, and Bergstrom (2010) dispute this. It is certainly
management, many journals have a cited half-life of over 10 years the case that the rankings of journals with the Eigenfactor, which is
while in cell biology it is typically less than 6. The 5-year JIF is not normalised for the number of publications, are significantly dif-
better in this respect (Campanario, 2011). ferent to those based on total citations, JIF or AI, which are all quite
• There is a lack of transparency in the way the JIF is calculated and similar (Leydesdorff, 2009).
this casts doubt on the results. Brumback (2008) studied medical The SJR works in a similar way to the Eigenfactor but includes
journals and could not reproduce the appropriate figures. It is within it a size normalisation factor and so is more akin to the article
highly dependent on which types of papers are included in the influence score. Each journal is a node and each directed connection
denominator. In 2007, the editors of three prestigious medical is a normalised value of the number of citations from one journal
journals published a paper questioning the data (Rossner, Van to another over a three year window. It is normalised by the total
Epps, & Hill, 2007). Pislyakov (2009) has also found differences number of citations in the citing journal for the year in question. It
between JIFs calculated in WoS and Scopus for economics resulting works in two phases:
from different journal coverage.
• It is possible for journals to deliberately distort the results by, for 1. An un-normalised value of journal prestige is calculated itera-
example, publishing many review articles which are more highly tively until a steady state is reached. The value of prestige actually
cited; publishing short reports or book reviews that get cited but includes three components: A fixed amount for being included
are not included in the count of papers; or pressuring authors to in the database (Scopus); an amount dependent on the number
gratuitously reference excessive papers from the journal (Wilhite of papers the journal produces; a citation amount dependent on
& Fong, 2012). The Journal of the American College of Cardiology, the number of citations received, and the prestige of the sources.
for example, publishes each year an overview of highlights in its However, there are a number of arbitrary weighting constants in
the calculation.
2. The value from 1, which is size-dependent, is then normalised by
17
There was a special issue of Scientometrics (92, 2, 2012) devoted to it and also a the number of published articles and adjusted to give an “easy-to-
compilation of 40 papers published in Scientometrics (Braun, 2007). use” value.
10 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19

Table 3
Characteristics of metrics for measuring journal impact.

Metric Description Advantages Disadvantages Maximum values for: Normalisation of:

(a) cell biology No. of Field Prestige

(b) management papers

Impact factor (JIF) Mean citations per paper Well-known, easy to Not normalised to From WoS Y N N
and cited half-life over a 2 or 5 year window. calculate and discipline; short time (a) 36.5
(WoS) Normalised to number of understand span; concerns about (b) 7.8
papers. Counts citations data and
equally manipulation
Eigenfactor and Based on PageRank, The AI is normalised Very small values, From WoS N N Y
article influence measures citations in to number of difficult to interpret, Eigenfactor: Y N Y
score (AI) (WoS) terms of the prestige of papers. A value of not normalised (a)0.599
citing journal. Not 1.0 shows average (b)0.03
normalised to discipline influence across all AI:
or number of papers. journals (a) 22.2
Correlated with total (b) 6.56
citations. Ignores
self-citations. AI is
normalised to number of
papers, so is like a JIF
5-year window
SJR and SJR2 Based on citation prestige Normalised number Complex calculations Not known Y N Y
(Scopus) but also includes a size of papers but not and not easy to
normalisation factor. to field so interpret. Not field
SJR2 also allows for the comparable to JIF. normalised
closeness of the citing Most sophisticated
journal. 3-year window indicator
h-index (Scimago The h papers of a journal Easy to calculate Not normalised to From Google Metrics N N N
website and that have at least h and understand. number of papers or h5:
Google Metrics) citations. Can have any Robust to poor field. (a) 223
window – Google metrics data Not pure impact but (b) 72
uses 5-year includes volumes h5 median:
(a) 343
(b)122
SNIP Citations per paper Normalises both to Does not consider Not known Y Y N
Revised SNIP normalised to the relative number of papers citation prestige.
(Scopus) database citation and field Complex and difficult
potential, that is the mean to check.
number of references in Revised version is
the papers that cite the sensitive to variability
journal of number of
references
I3 Combines the distribution of Normalises across Needs reference sets Not known N Y N
citation percentiles with fields. Does not use based on pre-defined
respect to a reference set the mean but is categories such as
with the number of based on WoS
papers in each percentile percentiles which
class is better for
skewed data

Gonzales-Pereira et al. (2010) carried out extensive empirical com- the relatedness of the two journals is also taken into account. An extra
parisons with a 3-year JIF (on Scopus data). The main conclusions term is added based on the cosine of the angle between the co-citation
were that the two were highly correlated, but the SJR showed that vectors of the journals so that the citations from a journal in a highly
some journals with high JIFs and lower SJRs were indeed gaining related area count for more. It is claimed that this also goes some way
citations from less prestigious sources. This was seen most clearly towards reducing the disparity of scores between subjects. However,
in the computer science field where the top ten journals, based on it also makes the indicator even more complex, hard to compute, and
the two metrics, were entirely different except for the number one, less understandable.
which was clearly a massive outlier (Briefings in Bioinformatics). Val- The h-index can also be used to measure the impact of journals as
ues for the JIF are significantly higher than for SJR. Falagas, Kouranos, it can be applied to any collection of cited papers (Braun, Glänzel,
Arencibia-Jorge, and Karageorgopoulos (2008) also compared the SJR & Schubert, 2006; Schubert & Glänzel, 2007; Xu, Liu, & Mingers,
favourably with the JIF. 2015). Studies have been carried out in several disciplines: marketing
There are several limitations of these 2nd generation measures: (Moussa & Touzani, 2010), economics (Harzing & Van der Wal, 2009),
the values for “prestige” are difficult to interpret as they are not a information systems (Cuellar, Takeda, & Truex, 2008) and business
mean citation value but only make sense in comparison with oth- (Mingers et al., 2012). The advantages and disadvantages of the h-
ers; they are still not normalised for subject areas (Lancho-Barrantes, index for journals are the same as the h-index generally, but it is
Guerrero-Bote, & Moya-Anegón, 2010); and the subject areas them- particularly the case that it is not normalised for different disciplines,
selves are open to disagreement (Mingers & Leydesdorff, 2014). and it is also strongly affected by the number of papers published. So
A further development of the SJR indicator has been produced a journal that publishes a small number of highly cited papers will be
(Guerrero-Bote & Moya-Anegón, 2012) with the refinement that, in disadvantaged in comparison with one publishing many papers, even
weighting the citations according to the prestige of the citing journal, if not so highly cited. Google Metrics (part of Google Scholar) uses a
 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19 11

5-year h-index and also shows the median number of citations for bers of papers but is normalised through the use of percentiles (see
those papers in the h core to allow for differences between journals Section 4.3 for more explanation).
with the same h-index. It has been critiqued by Delgado-López-Cózar
and Cabezas-Clavijo (2012). 6. Visualising and mapping science
Another recently developed metric that is implemented in Scopus
but not WoS is SNIP – source normalised impact per paper (Moed, In addition to its use as an instrument for the evaluation of impact,
2010b). This normalises for different fields based on the citing-side citations can also be considered as an operationalisation of a core
form of normalisation discussed above, that is, rather than normalis- process in scholarly communication, namely, referencing. Citations
ing with respected to the citations that a journal receives, it normalises refer to texts other than the one that contains the cited references,
with respect to the number of references in the citing journals. The and thus induce a dynamic vision of the sciences developing as net-
method proceeds in three stages: works of relations (Price, 1965). The development of co-citation anal-
1 First the raw impact per paper (RIP) is calculated for the journal. ysis (Marshakova, 1973; Small, 1973) and co-word analysis (Callon
This is essentially a 3 year JIF – the total citations from year n to et al., 1983) were achievements of the 1970s and 1980s. Aggregated
papers in the preceding 3 years is divided by the number of citable journal–journal citations are available on a yearly basis in the Jour-
papers. nal Citation Reports of the Science Citation Index since 1975. During
2 Then the database citation potential for the journal (DCP) is cal- the mid-1980s several research teams began to use this data for vi-
culated. This is done by finding all the papers in year n that cite sualisation purposes using multidimensional scaling and other such
papers in the journal over the preceding 10 years, and then calcu- techniques (Doreian & Fararo, 1985; Leydesdorff, 1986; Tijssen, de
lating the arithmetic mean of the number of references (to papers Leeuw, & van Raan, 1987). The advent of graphical user-interfaces in
in the database – Scopus) in these papers. Windows during the second half of the 1990s stimulated the further
3 The DCP is then relativised (RDCP). The DCP is calculated for all development of network analysis and visualisation programs such as
journals in the database and the median value is found. Then Pajek (de Nooy et al., 2011) that enable users to visualise large net-
RDCPj = DCPj /Median DCP. Thus a field that has many references works. Using large computer facilities, Boyack, Klavans, and Börner
will have an RDCP above 1. (2005) first mapped “the backbone” of all the sciences (De Moya-
4 Finally, SNIPj = RIPj /RDCPj Anegón et al., 2007).
Bollen et al. (2009) developed maps based on clickstream data;
The result is that journals in fields that have a high citation po- Rosvall and Bergstrom (2010) proposed to use alluvial maps for show-
tential will have their RIP reduced, and vice versa for fields with ing the dynamics of science. Rafols, Porter, and Leydesdorff (2010)
low citation potential. This is an innovative measure both because first proposed to use these “global” maps as backgrounds for overlays
it normalises for both number of publications and field, and be- that inform the user about the position of specific sets of documents,
cause the set of reference journals are specific to each journal rather analogously to overlaying institutional address information on geo-
than being defined beforehand somewhat arbitrarily. Moed presents graphical maps like Google Maps. More recently, these techniques
empirical evidence from the sciences that the subject normalisa- have been further refined, using both journal (Leydesdorff, Rafols
tion does work even at the level of pairs of journals in the same et al., 2013) and patent data (Kay, Newman, Youtie, Porter, & Rafols,
field. Also, because it only uses references to papers within the 2014; Leydesdorff, Alkemade, Heimeriks, & Hoekstra, 2015).
database, it corrects for coverage differences – a journal with low Nowadays, scientometric tools for the visualisation are increas-
database coverage will have a lower DCP and thus a higher value of ingly available on the internet. Some of them enable the user di-
SNIP. rectly to map input downloaded from Web of Science or Scopus.
A modified version of SNIP has recently been introduced (Waltman VOSviewer18 (Van Eck & Waltman, 2010) can generate, for example,
et al., 2013) to overcome certain technical limitations, and also in re- co-word and co-citation maps from this data.
sponse to criticism from Leydesdorff and Opthof (2010) and Moed
(2010c) who favour a fractional citation approach. The modified ver-
6.1. Visualisation techniques
sion involves two main changes: (i) the mean number of references
(DCP), but not the RIP, is now calculated using the harmonic mean
The systems view of multidimensional scaling (MDS) is determin-
rather than the arithmetic mean. (ii) The relativisation of the DCP to
istic, whereas the graph-analytic approach can also begin with a ran-
the overall median DCP is now omitted entirely, now SNIP = RIP/
dom or arbitrary choice of a starting point. Using MDS, the network is
DCP.
first conceptualised as a multi-dimensional space that is then reduced
Mingers (2014) has pointed out two problems with the revised
stepwise to lower dimensionality. At each step, the stress increases.
SNIP. First, because the value is no longer relativised it does not bear
Kruskall’s stress function is formulated as follows:
any particular relation to either the RIP for a journal, or the average 
number of citations/references in the database which makes it harder
i=j (xi − xj  − dij )2
to interpret. Second, the harmonic mean, unlike the arithmetic, is S= 2
(1)
sensitive to the variability of values. The less even the numbers of i=j dij
references, the lower will be the harmonic mean and this can make a In this formula ||xi − xj || is equal to the distance on the map, while
significant difference to the value of SNIP which seems inappropriate. the distance measure dij can be, for example, the Euclidean distance
There is also a more general problem with these sophisticated metrics in the data under study. One can use MDS to illustrate factor-analytic
that work across a whole database, and that is that the results cannot results in tables, but in this case the Pearson correlation is used as the
be easily replicated as most researchers do not have sufficient access similarity criterion.
to the databases (Leydesdorff, 2013). Spring-embedded or force-based algorithms can be considered as
Two other alternatives to the JIF have been suggested (Leydesdorff, a generalization of MDS, but were inspired by developments in graph
2012) – fractional counting of citations, which is similar in princi- theory during the 1980s. Kamada and Kawai (1989) were the first to
ple to SNIP, and the use of non-parametric statistics such as per- reformulate the problem of achieving target distances in a network
centiles which avoids using means which are inappropriate with in terms of energy optimisation. They formulated the ensuing stress
highly skewed data. A specific metric, based on percentiles, called I3
has been proposed by Leydesdorff & Bornmann (2011b) which com-
bines relative citation impact with productivity in terms of the num- 18
http://www.vosviewer.com.
12 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19

Fig. 2. Cosine-normalised map of the 58 title words which occur ten or more times in the 505 documents published in EJOR during 2013. (cosine > 0.1; modularity Q = 0.548 using
Blondel et al. (2008); Kamada and Kawai (1989) used for the layout; see http://www.leydesdorff.net/software/ti).

in the graphical representation as follows: can then be considered as a distance in the vector space (Leydesdorff
1 & Rafols, 2011).
S= sij with sij = 2
(xi − xj  − dij )2 (2) Newman and Girvan (2004) developed an algorithm in graph the-
i=j dij ory that searches for (latent) community structures in networks of
Eq. (2) differs from Eq. (1) by taking the square root in Eq. (1), observable relations. An objective function for the decomposition is
and because of the weighing of each term in the numerator of Eq. (2) recursively minimised and thus a “modularity” Q can be measured
with 1/dij 2 . This weight is crucial for the quality of the layout, but (and normalised between zero and one). Blondel et al. (2008) im-
defies normalization with dj2 in the denominator of Eq. (1); hence proved community-finding for large networks; this routine is imple-
the difference between the two stress values. Note that 1 is a ratio of mented in Pajek and Gephi, whereas Newman and Girvan’s original
sums while 2 is a sum of ratios (see discussion above). routine can be found in the Sci2 Toolset for “the science of science”.19
The ensuing difference at the conceptual level is that spring- VOSviewer provides its own algorithms for the mapping and the de-
embedding is a graph-theoretical concept developed for the topology composition.
of a network. The weighting is achieved for each individual link. MDS
operates on the multivariate space as a system, and hence refers to a
different topology. In the multivariate space, two points can be close 6.2. Local and global maps
to each other without entertaining a relationship. For example, they
can be close or distanced in terms of the correlation between their To illustrate some of these possibilities, we analysed the 505 doc-
patterns of relationships. uments published in the European Journal of Operational Research in
In the network topology, Euclidean distances and geodesics (short- 2013.20 Among the 1555 non-trivial words in the titles of these doc-
est distances) are conceptually more meaningful than correlation- uments, 58 words occur more than ten times and form a large com-
based measures. In the vector space, correlation analysis (factor ponent. A semantic map of these terms is shown in Fig. 2.
analysis, etc.) is appropriate for analysing the main dimensions of In Fig. 2 we can see some sensible groupings – for exam-
a system. The cosines of the angles among the vectors, for example, ple transportation/scheduling, optimisation/programming, decision
build on the notion of a multi-dimensional space. In bibliometrics, analysis, performance measurement and a fifth around manage-
Ahlgren, Jarneving, and Rousseau (2003) have argued convincingly in ment/application areas.
favour of the cosine as a non-parametric similarity measure because of
the skewedness of the citation distributions and the abundant zeros in
citation matrices. Technically, one can also input a cosine-normalised 19
https://sci2.cns.iu.edu/user/index.php.
matrix into a spring-embedded algorithm. The value of (1 – cosine) 20
Using a routine available at http://www.leydesdorff.net/software/ti.
 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19 13

QS,22 Times Higher23 and Shanghai24 ) including one from Leiden25

based purely on bibliometrics (Waltman et al., 2012) while on a micro
scale personal employment and promotion is shaped by journal and
citation data. Much of this is based on the citation metrics that we
have discussed above.
The traditional method of research evaluation was peer review
(Abramo et al., 2011; Bornmann, 2011; Moed, 2007). However, this
has many drawbacks – it is very time consuming and costly (Abramo
& D’Angelo, 2011), subject to many biases and distortions (Horrobin,
1990; Moxham & Anderson, 1992), generally quite opaque (panel
members in the 2008 UK RAE were ordered to destroy all notes for
fear of litigation) (Reale, Barbara, & Costantini, 2007) and limited in
the extent to which it actually provides wide-ranging and detailed
information (Butler & McAllister, 2009). The UK did investigate using
bibliometrics in 2008 (HEFCE, 2009), used them to a limited extent
in 2014, and are expected to employ them more fully in 2020. In
contrast, bibliometrics has the potential to provide a cheaper, more
objective and more informative mode of analysis, although there is
Fig. 3. 613 journals cited in 505 documents published in EJOR during 2013,
overlaid on the global map of science in terms of journal-journal citation rela-
general agreement that bibliometrics should only be used in com-
tions. (Rao–Stirling diversity is 0.1187; Leydesdorff, Rafols & Chen (2013); see at bination with some form of transparent peer review (Moed, 2007;
http://www.leydesdorff.net/journals12). van Raan, 2005b). Abramo and D’Angelo (2011) compared informed
peer review (including the UK RAE) with bibliometrics on the natural
and formal sciences in Italy and concluded that bibliometrics were
Fig. 3 shows the 613 journals that are most highly cited in the clearly superior across a range of criteria – accuracy, robustness, va-
same 505 EJOR papers (12,172 citations between them) but overlaid lidity, functionality, time and cost. They recognised that there were
on to a global map of science (Leydesdorff, Rafols et al., 2013). These problems in the social sciences and humanities where citation data is
cited sources can, for example, be considered as an operationalisation often not available.
of the knowledge base on which these articles draw. It can be seen The effective use of bibliometrics has a number of requirements,
that, apart from the main area around OR and management, there not all of which are currently in place.
is significant citation to the environmental sciences, chemical engi- First, one needs robust and comprehensive data. As we have al-
neering, and biomedicine. Rao–Stirling diversity — a measure for the ready seen, the main databases are reliable but their coverage is lim-
interdisciplinarity of this knowledge base (Rao, 1982) — however, is ited especially in the humanities and social sciences and they need to
low (0.1187). In other words, citation within the specialty prevails. enlarge their scope to cover all forms of research outputs (Leydesdorff,
Fig. 4 shows a local map of the field of OR based on the 29 journals 2008). Google Scholar is more comprehensive, but unreliable and
most highly cited in papers published in Operations Research in 2013. non-transparent. At this time, full bibliometric evaluation is feasible
In this map three groupings have emerged – the central area of OR, in science and some areas of social science, but not in the humani-
including transportation; the lower left of particularly mathematical ties or some areas of technology (Archambault et al., 2006; Nederhof,
journals; and the upper region of economics and finance journals 2006; van Leeuwen, 2006). Abramo and D’Angelo (2011) suggest that
which includes Management Science. nations should routinely collect data on all the publications published
In summary, the visualisations enable us to represent the cur- within its institutions so that it is scrutinised and available on demand
rent state of the field (Fig. 2), its knowledge base (Fig. 3), and rather than having to be collected anew each time a research evalua-
its relevant environments (Fig. 4). Second-order visualisation pro- tion occurs (Ossenblok, Engels, & Sivertsen, 2012; Sivertsen & Larsen,
grams available at the internet such as VOSviewer and CitNetEx- 2012).
plorer21 enable the user to automatically generate several of these Second, one needs suitable metrics that measure what is impor-
visualizations from data downloaded from WoS or Scopus. One can tant in as unbiased way as possible. These should not be crude ones
also envisage making movies from this data. These networks evolve such as simple counts of citations or papers, the h-index (although
over time and the diagrams can be animated – see, for exam- this has its advantages) or journal impact factors but more sophisti-
ple: http://www.leydesdorff.net/journals/nanotech/ or other ones at cated ones that take into account the differences in citation practices
http://www.leydesdorff/visone for an overview and instruction. across different disciplines as has been discussed in Section 4. This is
currently an area of much debate with a range of possibilities (Gingras,
2014). The traditional crown indicator (now MNCS) is subject to crit-
icisms concerning the use of the mean on highly cited data and also
7. Evaluation and policy on the use of WoS field categories (Ruiz-Castillo & Waltman, 2014).
There are source normalised alternatives such as SNIP or fractional
As we said in Section 1, scientometrics has come to prominence counting (Aksnes, Schneider, & Gunnarsson, 2012) and metrics that
because of its use in the evaluation and management of research include the prestige of the citing journals such as SJR. There are also
performance, whether at the level of the researcher, research group, moves towards non-parametric statistics based on percentiles. One
institution or journal (Bornmann & Leydesdorff, 2014). Many coun- dilemma is that the more sophisticated the metrics become, the less
tries, especially the UK, Australia, New Zealand and Italy, carry out transparent and harder to replicate they are.
regular reviews of university performance affecting both the distri-
bution of research funds and the generation of league tables. On a
macro scale, world university league tables have proliferated (e.g.,
22
http://www.topuniversities.com/.
23
http://www.timeshighereducation.co.uk/world-university-rankings/.
24
http://www.shanghairanking.com/World-University-Rankings-2014/UK.html.
21 25
http://www.citnetexplorer.nl/Home. http://www.leidenranking.com/.
14 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19

Fig. 4. Local map of the 29 journals cited in articles of Operations Research in 2013 (1 percent level; cosine > 0.2; Kamada & Kawai, 1989; Blondel et al., 2008; Q = 0.213).

A third area for consideration is inter- or trans-disciplinary work, and so simple measures such as the h-index or the JIF are used indis-
and work that is more practical and practitioner oriented. How would criminately without due attention being paid to their limitations and
this be affected by a move towards bibliometrics? There is cur- biases. This also reminds us that there are ethical issues in the use
rently little research in this area (Larivière & Gingras, 2010) although of bibliometrics for research evaluation and Furner (2014) has devel-
Rafols, Leydesdorff, O’Hare, Nightingale, and Stirling (2012) found a oped a framework for evaluation that includes ethical dimensions.
systematic bias in research evaluation against interdisciplinary re-
search in the field of business and management. Indeed, bibliomet- 8. Future developments
rics is still at the stage of establishing reliable and feasible methods
for defining and measuring interdisciplinarity (Wagner et al., 2011). 8.1. Alternative metrics
Huutoniemi, Klein, Bruun, and Hukkinen (2010) developed a typology
and indicators to be applied to research proposals, and potentially re- Although citations still form the core of scientometrics, the dra-
search papers as well; Leydesdorff and Rafols (2011) have developed matic rise of social media has opened up many more channels for
citation-based metrics to measure the interdisciplinarity of journals; recording the impact of academic research (Bornmann, 2014; Konkiel
and Silva, Rodrigues, Oliveira, and da Costa (2013) evaluated the rel- & Scherer, 2013; Priem, 2014; Roemer & Borchardt, 2012). These go
ative interdisciplinarity of science fields using entropy measures. under the name of “altmetrics” both as a field, and as particular alter-
Fourth, we must recognise, and try to minimise, the fact that the native metrics.26 One of the interesting characteristics of altmetrics
act of measuring inevitably changes the behaviour of the people being is that it throws light on the impacts of scholarly work on the general
measured. So, citation-based metrics will lead to practices, legitimate public rather than just the academic community. The Public Library
and illegitimate, to increase citations; an emphasis on 4∗ journals of Science (PLoS) (Lin & Fenner, 2013) has produced a classification of
leads to a lack of innovation and a reinforcement of the status quo. types of impacts which goes from least significant to most significant:
For example, Moed (2008) detected significant patterns of response
• Viewed: institutional repositories, publishers, PLoS,
to UK research assessment metrics, with an increase in total publi-
Academia.com, ResearchGate. Perneger (2004) found a weak
cations after 1992 when numbers of papers were required; a shift
correlation with citations.
to journals with higher citations after 1996 when quality was em-
• Downloaded/Saved: as viewed plus CiteUlike, Mendelay.
phasised; and then an increase in the apparent number of research
• Discussed: Wikipedia, Facebook, Twitter, Natureblogs,27 Science-
active staff through greater collaboration during 1997–2000. Michels
Seeker,28 general research blogs. Eysenbach (2011) suggested a
and Schmoch (2014) found that German researchers changed their
“twimpact factor” based on the number of tweets.
behaviour to aim for more US-based high impact journals in order to
• Recommended: F1000Prime.29
increase their citations.
Fifth, we must be aware that often problems are caused not by
26
the data or metrics themselves, but by their inappropriate use either http://altmetrics.org/, http://www.altmetric.com/.
27
by academics or by administrators (Bornmann & Leydesdorff, 2014; http://blogs.nature.com/.
28
http://scienceseeker.org/.
van Raan, 2005b). There is often a desire for “quick and dirty” results 29
http://f1000.com/prime.
 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19 15

• Cited: Wikipedia CrossRef, WoS, Scopus, Google Scholar. References

Altmetrics is still in its infancy and the majority of papers would Aad, G., Abajyan, T., Abbott, B., Abdallah, J., Abdel Khalek, S., et al. (2012). Ob-
servation of a new particle in the search for the Standard Model Higgs bo-
have little social networking presence. There are also a number of
son with the ATLAS detector at the LHC. Physics Letters B, 716(1), 1–29.
problems: (i) altmetrics can be gamed by “buying” likes or tweets; doi:10.1016/j.physletb.2012.08.020.
(ii) there is little by way of theory about how and why altmetrics are Abramo, G., & D’Angelo, C. (2011). Evaluating research: from informed peer review to
generated (this is also true of traditional citations); (iii) a high score bibliometric. Scientometrics, 87(3), 499–514.
Abramo, G., D’Angelo, C., & Di Costa, F. (2011). National research assessment exer-
may not mean that the paper is especially good, just on a controversial cises: A comparison of peer review and bibliometric rankings. Scientometrics, 89,
or fashionable topic; and (iv) because social media is relatively new 929–941.
it will under-represent older papers. Adriaanse, L., & Rensleigh, C. (2013). Web of Science, Scopus and Google Scholar. The
Electronic Library, 31(6), 727–744. doi:10.1108/EL-12-2011-0174.
Ahlgren, P., Jarneving, B., & Rousseau, R. (2003). Requirements for a cocitation simi-
larity measure, with special reference to Pearson’s correlation coefficient. Journal
8.2. The shape of the discipline of the American Society for Information Science and Technology, 54(6), 550–560.
doi:10.1002/asi.10242.
Aksnes, D. W., Schneider, J. W., & Gunnarsson, M. (2012). Ranking national research sys-
Citations refer to texts other than the one that contains the cited
tems by citation indicators. A comparative analysis using whole and fractionalised
references, and thus induce a dynamic vision of the sciences develop- counting methods. Journal of Informetrics, 6(1), 36–43.
ing as networks of relations (Price, 1965). In the scientometric litera- Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E., & Herrera, F. (2009). h-Index: A review
focused in its variants, computation and standardization for different scientific
ture, this has led to the call for “a theory of citation” (Cozzens, 1989;
fields. Journal of Informetrics, 3(4), 273–289. doi:10.1016/j.joi.2009.04.001.
Cronin, 1998; Garfield, 1979; Leydesdorff, 1998; Nicolaisen, 2007). Amara, N., & Landry, R. (2012). Counting citations in the field of business and man-
Wouters (1998) noted that in science and technology studies (STS), agement: why use Google Scholar rather than the Web of Science. Scientometrics,
citations are studied as references in the context of “citing” practices, 93(3), 553–581. doi:10.1007/s11192-012-0729-2.
Archambault, É., Vignola-Gagné, É., Côté, G., Larivière, V., & Gingras, Y. (2006). Bench-
whereas the citation index inverts the directionality and studies “cit- marking scientific output in the social sciences and humanities: The limits of ex-
edness” as a measure of impact. From the perspective of STS, the isting databases. Scientometrics, 68(3), 329–342.
citation index thus would generate a semiotic artifact (Luukkonen, Aziz, N., & Rozing, M. (2013). Profit (p)-index: The degree to which authors profit from
co-authors. PLoS One, 8(4), e59814.
1997). Bar-Ilan, J. (2008). Informetrics at the beginning of the 21st century—A review. Journal
References can have different functions in texts, such as legiti- of Informetrics, 2(1), 1–52.
mating research agendas, warranting knowledge claims, black-boxing Batista, P., Campiteli, M., Kinouchi, O., & Martinez, A. (2006). Is it possible to
compare researchers with different scientific interests? Scientometrics, 68(1),
discussions, or be perfunctory. In and among texts, references can also 179–189.
be compared with the co-occurrences and co-absences of words in a Baumgartner, S., & Leydesdorff, L. (2014). Group-based trajectory modeling (GBTM)
network model of science (Braam, Moed, & van Raan, 1991a, 1991b) of citations in scholarly literature: Dynamic qualities of “transient” and “sticky"
knowledge claims. Journal of the American Society for Information Science and Tech-
A network theory of science was formulated by Hesse (1980, p. 83)
nology, 65(4), 797–811.
as “an account that was first explicit in Duhem and more recently Bergstrom, C. (2007). Measuring the value and prestige of scholarly journals. College &
reinforced in Quine. Neither in Duhem nor in Quine, however, is it Research Libraries News, 68(5), 3146.
Bhaskar, R. (1978). A realist theory of science. Hemel Hempstead: Harvester.
quite clear that the netlike interrelations between more observable
Björneborn, L., & Ingwersen, P. (2004). Toward a basic framework for webometrics.
predicates and their laws are in principle just as subject to modifi- Journal of the American Society for Information Science and Technology, 55(14), 1216–
cations from the rest of the network as are those that are relatively 1227. doi:10.1002/asi.20077.
theoretical.” A network can be visualised, but can also be formalised Blondel, V., Guillaume, J.-L., Lambiotte, R., & Lefebvre, L. (2008). Fast unfolding of com-
munities in large networks. Journal of Statistical Mechanics: Theory and Experiment,
as a matrix. The eigenvectors of the matrix span the latent dimensions 2008(10), P10008.
of the network. Bollen, J., Van de Sompel, H., Hagberg, A., Bettencourt, L., Chute, R., Rodriguez, M., et al.
There is thus a bifurcation within the discipline of scientomet- (2009). Clickstream data yields high-resolution maps of science. PLoS One, 4(3),
e4803.
rics. On the one hand, and by far the dominant partner, we have the Bornman, L., & Daniel, H.-D. (2005). Does the h-index for ranking of scientists really
relatively positivistic, quantitative analysis of citations as they have work? Scientometrics, 65(3), 391–392.
happened, after the fact so to speak. And on the other, we have the Bornmann, L. (2010). Towards an ideal method of measuring research performance:
Some comments to the Opthof and Leydesdorff (2010) paper. Journal of Informetrics,
sociological, and often constructivist theorising about the generation
4(3), 441–443.
of citations – a theory of citing behaviour. Clearly the two sides are, Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and
and need to be linked. The citing behaviour, as a set of generative Technology, 45(1), 197–245.
Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An
mechanisms (Bhaskar, 1978), produces the citation events but, at the
overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8,
same time, analyses of the patterns of citations as “demi-regularities” 895–903.
(Lawson, 1997) can provide insights into the processes of scientific Bornmann, L., & Daniel, H.-D. (2007). What do we know about the h index? Journal
communication which can stimulate or validate theories of behaviour. of the American Society for Information Science and Technology, 58(9), 1381–1385.
doi:10.1002/asi.20609.
Another interesting approach is to consider the overall process Bornmann, L., & Leydesdorff, L. (2014). Scientometrics in a changing research landscape.
as a social communication system. One could use Luhmann’s (1995, EMBO Reports, 15(12), 1228–1232.
1996) theory of society as being based on autopoietic communica- Bornmann, L., Leydesdorff, L., & Mutz, R. (2013). The use of percentiles and percentile
rank classes in the analysis of bibliometric data: Opportunities and limits. Journal
tion (Leydesdorff, 2007; Mingers, 2002). Different functional subsys- of Informetrics, 7(1), 158–165. doi:10.1016/j.joi.2012.10.001.
tems within society, e.g., science, generate their own organisation- Bornmann, L., Leydesdorff, L., & Wang, J. (2013a). How to improve the prediction based
ally closed networks of recursive communications. A communicative on citation impact percentiles for years shortly after the publication date? Journal
of Informetrics, 8(1), 175–180.
event consists of a unity of information, utterance and understanding
Bornmann, L., Leydesdorff, L., & Wang, J. (2013b). Which percentile-based approach
between senders and receivers. Within the scientometrics context, should be preferred for calculating normalized citation impact values? An empirical
the paper, its content and its publication would be the information comparison of five approaches including a newly developed citation-rank approach
(P100). Journal of Informetrics, 7(4), 933–944. doi:10.1016/j.joi.2013.09.003.
and utterance, and the future references to it in other papers would
Bornmann, L., & Mutz, R. (2011). Further steps towards an ideal method of mea-
be the understanding that it generates. Such communication systems suring citation performance: The avoidance of citation (ratio) averages in field-
operate at their own emergent level distinct from the individual sci- normalization. Journal of Informetrics, 5(1), 228–230.
entists who underlie them, and generate their own cognitive distinc- Bornmann, L., Mutz, R., & Daniel, H. D. (2008). Are there better indices for evaluation
purposes than the h index? A comparison of nine different variants of the h index
tions that can be revealed by the visualisation procedures discussed using data from biomedicine. Journal of the American Society for Information Science
above. and Technology, 59(5), 830–837. doi:10.1002/asi.20806.
16 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19

Boyack, K., Klavans, R., & Börner, K. (2005). Mapping the backbone of science. Sciento- Egghe, L. (2006). Theory and practice of the g-index. Scientometrics, 69(1), 131–
metrics, 64(3), 351–374. 152.
Braam, R., Moed, H., & van Raan, A. (1991a). Mapping of science by combined co- Egghe, L. (2010). The Hirsch index and related impact measures. Annual Review of
citation and word analysis I: Structural aspects. Journal of the American Society for Information Science and Technology, 44(1), 65–114.
Information Science, 42(4), 233–251. Egghe, L., & Rousseau, R. (1990). Introduction to Informetrics.
Braam, R., Moed, H., & van Raan, A. (1991b). Mapping of science by combined co- Egghe, L., & Rousseau, R. (Eds.). (1988). Informetrics 87/88. Amsterdam: Elsevier.
citation and word analysis. II: Dynamical aspects. Journal of the American Society Elkana, Y., Lederberg, J., Merton, R., Thackray, A., & Zuckerman, H. (Eds.). (1978). Toward
for Information Science, 42(4), 252–266. a metric of science: The advent of science indicators. New York: Wiley.
Bradford, S. (1934). Sources of information on specific subjects. Engineering, 137(3550), Eysenbach, G. (2011). Can tweets predict citations? Metrics of social impact based
85–86. on twitter and correlation with traditional metrics of scientific impact. Journal of
Braun, T. (Ed.). (2007). The impact factor of scientific and scholarly journals: Its use and Medical Internet Research, 13(4), e123. doi:10.2196/jmir.2012.
misuse in research evaluation. Hungary: Akedemiai Kiado Zrt. Falagas, M. E., Kouranos, V. D., Arencibia-Jorge, R., & Karageorgopoulos, D. E. (2008).
Braun, T., Glänzel, W., & Schubert, A. (1985). Scientometric indicators: A 32-country com- Comparison of SCImago journal rank indicator with journal impact factor. The FASEB
parison of publication productivity and citation impact. Singapore: World Scientific Journal, 22(8), 2623–2628.
Publishing. Ferlie, E., Ashburner, L., Fitzgerald, L., & Pettigrew, A. M. (1996). The new public man-
Braun, T., Glänzel, W., & Schubert, A. (2006). A Hirsch-type index for journals. Sciento- agement in action. Oxford: Oxford U niversity P ress.
metrics, 69(1), 169–173. doi:10.1007/s11192-006-0147-4. Fersht, A. (2009). The most influential journals: Impact factor and eigenfactor. Proceed-
Brumback, R. (2008). Worshipping false idols: The impact factor dilemma. Journal of ings of the National Academy of Sciences of the United States of America, 106(17),
Child Neurology, 23, 365–367. 6883–6884.
Burrell, Q. (2001). Stochastic modelling of the first-citation distribution. Scientometrics, Franceschet, M. (2010). A comparison of bibliometric indicators for computer science
52(1), 3–12. scholars and journals on Web of Science and Google Scholar. Scientometrics, 83(1),
Burrell, Q. (2002). The nth-citation distribution and obsolescence. Scientometrics, 53(3), 243–258. doi:10.1007/s11192-009-0021-2.
309–323. Franceschini, F., & Maisano, D. A. (2010). Analysis of the Hirsch index’s opera-
Burrell, Q. (2003). Predicting future citation behaviour. Journal of the American Society tional properties. European Journal of Operational Research, 203(2), 494–504.
for Information Science, 54(5), 372–378. doi:10.1016/j.ejor.2009.08.001.
Burrell, Q. (2007). Hirsch index or Hirsch rate? Some thoughts arising from Liang’s data. Furner, J. (2014). The ethics of evaluative bibliometrics. In B. Cronin, & C. Sugimoto
Scientometrics, 73(1), 19–28. (Eds.), Beyond bibliometrics: Harnessing multidimensional indicators of scholarly im-
Butler, L., & McAllister, I. (2009). Metrics or peer review? Evaluating the 2001 UK pact (pp. 85–107). London: MIT Press.
Research Assessment Exercise in political science. Political Studies Review, 7(1), García-Pérez, M. A. (2010). Accuracy and completeness of publication and citation
3–17. records in the Web of Science, PsycINFO, and Google Scholar: A case study for
Callon, M., Courtial, J.-P., Turner, W. A., & Bauin, S. (1983). From translations to prob- the computation of h indices in psychology. Journal of the American Society for
lematic networks: An introduction to co-word analysis. Social Science Information, Information Science and Technology, 61(10), 2070–2085. doi:10.1002/asi.21372.
22(2), 191–235. doi:10.1177/053901883022002003. Garfield, E. (1955). Citation indexes for science: A new dimension in documentation
Cambell, D., Archambaulte, E., & Cote, G. (2008). Benchmarking of Cana- through association of ideas. Science, 122(3159), 108–111.
dian genomics: 1996–2007. Retrieved from http://www.science-metrix.com/pdf/ Garfield, E. (1979). Citation indexing: Its theory and application in science, technology and
SM.Benchmarking_Genomics_Canada.pdf. humanities. New York: Wiley.
Cameron, B. (2005). Trends in the usage of ISI bibliometric data: Uses, abuses and Garfield, E. (1998). The impact factor and using it correctly. Der Unfallchirug, 48(2), 413.
implications. Libraries and the Academy, 5, 105–125. Garfield, E., Sher, I., & Thorpie, R. (1964). The use of citation data in writing the history of
Campanario, J. M. (2011). Empirical study of journal impact factors obtained using the science. Philadelphia: Institute for Scientific Information.
classical two-year citation window versus a five-year citation window. Scientomet- Garfield, E., & Sher, I. H. (1963). New factors in the evaluation of scientific lit-
rics, 87(1), 189–204. erature through citation indexing. American Documentation, 14(3), 195–201.
Chapman, A. (1989). Assessing research: Citation-count shortcomings. The Psychologist: doi:10.1002/asi.5090140304.
Bulletin of the British Psychological Society, 8(8), 339–341. Gingras, Y. (2014). Criteria for evaluating indicators. In B. Cronin, & C. Sugimoto (Eds.),
Colliander, C. (2014). A novel approach to citation normalization: A similarity-based Beyond bibliometrics: Harnessing multidimensional indicators of scholarly impact
method for creating reference sets. Journal of the Association for Information Science (pp. 110–125). London: MIT Press.
and Technology, 66(3), 489–500 Online. doi:10.1002/asi.23193. Gingras, Y., & Larivière, V. (2011). There are neither “king” nor “crown” in scientomet-
Costas, R., & Bordons, M. (2007). The h-index: Advantages, limitations and its relation rics: Comments on a supposed “alternative” method of normalization. Journal of
with other bibliometric indicators at the micro level. Journal of Informetrics, 1, Informetrics, 5(1), 226–227. doi:10.1016/j.joi.2010.10.005.
193–203. Glänzel, W. (2006). On the h-index – A mathematical approach to a new measure of
Cozzens, S. (1989). What do citations count? The rhetoric-first model. Scientometrics, publication activity and citation impact. Scientometrics, 67(2), 315–321.
15, 437–447. Glänzel, W., & Schoepflin, U. (1995). A bibliometric study of ageing and reception
Crane, D. (1972). Invisible colleges. Chicago: University of Chicago Press. processes of scientific literature. Journal of Information Science, 21, 37–53.
Cronin, B. (1998). Metatheorizing citation. Scientometrics, 43(1), 45–55. Glänzel, W., & Schubert, A. (1985). Price distribution: An exact formulation of Price’s
Cronin, B. (2001). Hyperauthorship: A postmodern perversion or evidence of a struc- “square root law”. Scientometrics, 7(3–6), 211–219.
tural shift in scholarly communication practices? Journal of the American Society for Glänzel, W., Schubert, A., Thijs, B., & Debackere, K. (2011). A priori vs. a posteriori
Information Science and Technology, 52(7), 558–569. normalisation of citation indicators. The case of journal ranking. Scientometrics,
Cuellar, M.J., Takeda, H., & Truex, D.P. (2008). The Hirsch family of bibliometric indices 87(2), 415–424.
as an improved measure of IS academic journal impact. In AMCIS 2008 Proceedings. González-Pereira, B., Guerrero-Bote, V. P., & Moya-Anegón, F. (2010). A new approach to
Paper 295. http://aisel.aisnet.org/amcis2008/295 the metric of journals’ scientific prestige: The SJR indicator. Journal of Informetrics,
Davis, P. M. (2008). Eigenfactor: Does the principle of repeated improvement result 4(3), 379–391. doi:10.1016/j.joi.2010.03.002.
in better estimates than raw citation counts? Journal of the American Society for Guerrero-Bote, V. P., & Moya-Anegón, F. (2012). A further step forward in measuring
Information Science and Technology, 59(13), 2186–2188. journals’ scientific prestige: The SJR2 indicator. Journal of Informetrics, 6(4), 674–
Day, R. (2014). The data – It is me!. In B. Cronin, & C. Sugimoto (Eds.), Beyond bibliomet- 688. doi:10.1016/j.joi.2012.07.001.
rics: Harnessing multidimensional indicators of scholarly impact (pp. 47–66). London: Guns, R., & Rousseau, D. (2009). Real and rational variants of the h-index and g-index.
MIT Press. Journal of Informatrics, 3, 64–71.
de Bellis, N. (2014). History and evolution of (biblio)metrics. In B. Cronin, & C. Sugimoto Harzing, A.-W. (2007). Publish or perish, from http://www.harzing.com/pop.htm.
(Eds.), Beyond bibliometrics: Harnessing multidimensional indicators of scholarly im- Harzing, A.-W., Alakangas, S., & Adams, D. (2014). hIa: An individual annual h-index
pact (pp. 23–44). London: MIT Press. to accommodate disciplinary and career length differences. Scientometrics, 99(3),
De Moya-Anegón, F., Vargas-Quesada, B., Chinchilla-Rodríguez, Z., Corera-Álvarez, E., 811–821. doi:10.1007/s11192-013-1208-0.
Munoz-Fernández, F., & Herrero-Solana, V. (2007). Visualizing the marrow of sci- Harzing, A.-W., & van der Wal, R. (2008). Google Scholar as a new source for citation
ence. Journal of the American Society for Information Science and Technology, 58(14), analysis. Ethics in Science and Environmental Politics, 8, 61–73.
2167–2179. Harzing, A.-W., & Van der Wal, R. (2009). A Google Scholar h-index for journals: An
de Nooy, W., Mrvar, A., & Batgelj, V. (2011). Exploratory social network analysis with alternative metric to measure journal impact in economics and business? Journal
Pajek (2nd ed.). New York, NY: Cambridge University Press. of the American Society for Information Science and Technology, 60(1), 41–46.
Delgado-López-Cózar, E., & Cabezas-Clavijo, Á. (2012). Google Scholar Metrics: An un- HEFCE. (2008). Counting what is measured or measuring what counts. HEFCE.
reliable tool for assessing scientific journals. El profesional de la información, 21(4), HEFCE (2009). Report on the pilot exercise to develop bibliometric indicators for the Re-
419–427. search Excellence Framework. London: HEFCE.
DeMaria, A., Bax, J., Ben-Yehuda, O., Clopton, P., Feld, G. K., Ginsburg, G., & Lima, J. (2008). Hesse, M. (1980). Revolutions and reconstructions in the philosophy of science. London:
Highlights of the year in JACC 2007. Journal of the American College of Cardiology, Harvester Press.
51(4), 490–512. Hirsch, J. (2005). An index to quantify an individual’s scientific research output. Pro-
Doreian, P., & Fararo, T. (1985). Structural equivalence in a journal network. Journal of ceedings of the National Academy of Sciences of the United States of America, 102(46),
the American Society for Information Science, 36, 28–37. 16569–16572.
Egghe, L. (2000). A heuristic study of the first citation distribution. Scientometrics, 48, Hirsch, J. (2007). Does the h-index have predictive power? Proceedings of the National
345–359. Academy of Sciences of the United States of America, 104(49), 19193–19198.
Egghe, L. (2005). Power laws in the information production process: Lotkaian informetrics. Holmes, R. (2014). How to win citations and rise in the rankings University Ranking
Amsterdam: Elsevier. Watch (Vol. 2014).
 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19 17

Horrobin, D. (1990). The philosophical basis of peer review and the suppres- Leydesdorff, L., & Milojević, S. (2015). Scientometrics. In J. D. Wright, M. Lynch, &
sion of innovation. Journal of the American Medical Association, 263, 1438– et al. (Eds.), The International encyclopedia of social and behavioral sciences, 2nd
1441. Edition (Section 8.5: Science and Technology Studies, Subsection 85030). Oxford,
Huutoniemi, K., Klein, J. T., Bruun, H., & Hukkinen, J. (2010). Analyzing in- UK, etc: Elsevier.
terdisciplinarity: Typology and indicators. Research Policy, 39(1), 79–88. Leydesdorff, L., & Opthof, T. (2010). Scopus’s source normalized impact per paper (SNIP)
doi:10.1016/j.respol.2009.09.011. versus a journal impact factor based on fractional counting of citations. Journal of
Iglesias, J., & Pecharromán, C. (2007). Scaling the h-index for different scientific ISI the American Society for Information Science and Technology, 61(11), 2365–2369.
fields. Scientometrics, 73(3), 303–320. doi:10.1007/s11192-007-1805-x. doi:10.1002/asi.21371.
Irvine, J., Martin, B., Peacock, T., & Turner, R. (1985). Charting the decline of British Leydesdorff, L., & Opthof, T. (2011). Remaining problems with the “New Crown Indica-
science. Nature, 316, 587–590. tor” (MNCS) of the CWTS. Journal of Informetrics, 5(1), 224–225.
Jacso, P. (2005). As we may search – Comparison of major features of the Web of Sci- Leydesdorff, L., Radicchi, F., Bornmann, L., Castellano, C., & Nooy, W. (2013). Field-
ence, Scopus, and Google Scholar citation-based and citation-enhanced databases. normalized impact factors (IFs): A comparison of rescaling and fractionally counted
Current Science, 89(9), 1537–1547. IFs. Journal of the American Society for Information Science and Technology, 64(11),
Jacso, P. (2008). Savvy searching: Google Scholar revisited. Online Information Review, 2299–2309.
32(1), 102–114. Leydesdorff, L., & Rafols, I. (2011). Indicators of the interdisciplinarity of jour-
Jin, B. (2006). The h-index: An evaluation indicator proposed by scientist. Science Focus, nals: Diversity, centrality, and citations. Journal of Informetrics, 5(1), 87–100.
1(1), 8–9. doi:10.1016/j.joi.2010.09.002.
Jin, B. (2007). The AR-index: Complementing the h-index. ISSI Newsletter, 3(1), 6. Leydesdorff, L., Rafols, I., & Chen, C. (2013). Interactive overlays of journals and the
Jin, B., Liang, L., Rousseau, R., & Egghe, L. (2007). The R- and AR-indices: Complementing measurement of interdisciplinarity on the basis of aggregated journal-journal cita-
the h-index. Chinese Science Bulletin, 52(6), 855–863. tions. Journal of the American Society for Information Science and Technology, 64(12),
Kamada, T., & Kawai, S. (1989). An algorithm for drawing general undirected graphs. 2573–2586.
Information Processing Letters, 31(1), 7–15. Liang, L. (2006). h-Index sequence and h-index matrix: Constructions and applications.
Kay, L., Newman, N., Youtie, J., Porter, A., & Rafols, I. (2014). Patent overlay mapping: Scientometrics, 69(1), 153–159.
Visualizing technological distance. Journal of the Association for Information Science Lin, J., & Fenner, M. (2013). Altmetrics in evolution: Defining and redefining
and Technology, 65(12), 2432–2443. the ontology of article-level metrics. Information Standards Quarterly, 25(2).
Konkiel, S., & Scherer, D. (2013). New opportunities for repositories in the age of altmet- doi:10.3789/isqv25no2.2013.04.
rics. Bulletin of the American Society for Information Science and Technology, 39(4), Lotka, A. (1926). Statistics: The frequency distribution of scientific productivity. Journal
22–26. of the Washington Academy of Sciences, 16(12), 317–325.
Kuhn, T. (1962). The structure of scientific revolutions. Chicago: Chicag o University Press. Luhmann, N. (1995). Social systems (J. Bednarz & D. Baecker, Trans.). Stanford: Stanford
Lancho-Barrantes, B. S., Guerrero-Bote, V. P., & Moya-Anegón, F. (2010). What lies University Press.
behind the averages and significance of citation indicators in different disciplines? Luhmann, N. (1996). On the scientific context of the concept of communication. Social
Journal of Information Science, 36(3), 371–382. Science Information, 35(2), 257–267.
Larivière, V., Archambault, É., Gingras, Y., & Vignola-Gagné, É. (2006). The place of serials Lundberg, J. (2007). Lifting the crown—Citation z-score. Journal of Informetrics, 1(2),
in referencing practices: Comparing natural sciences and engineering with social 145–154.
sciences and humanities. Journal of the American Society for Information Science and Luukkonen, T. (1997). Why has Latour’s theory of citations been ignored by the biblio-
Technology, 57(8), 997–1004. metric community? Discussion of sociological interpretations of citation analysis.
Larivière, V., & Gingras, Y. (2010). On the relationship between interdisciplinarity and Scientometrics, 38(1), 27–37. doi:10.1007/bf02461121.
scientific impact. Journal of the American Society for Information Science and Tech- Ma, N., Guan, J., & Zhao, Y. (2008). Bringing PageRank to the citation analysis. Information
nology, 61(1), 126–131. Processing and Management, 44, 800–810.
Lawson, T. (1997). Economics and reality. London: Routledge. Mahdi, S., D’Este, P., & Neely, A. (2008). Citation counts: Are they good predictors of RAE
Lehmann, S., Jackson, A. D., & Lautrup, B. E. (2006). Measures for mea- scores?. London: AIM Research.
sures.. Nature, 444(7122), 1003–1004 http://www.nature.com/nature/journal/ Marshakova, I. (1973). Bibliographic coupling system based on references. Nauchno-
v444/n7122/suppinfo/4441003a_S1.html. doi:10.1038/4441003a. Tekhnicheskaya Informatsiya Seriya, 2(6), 3–8.
Leydesdorff, L. (1986). The development of frames of references. Scientometrics, 9(3–4), Meade, N., & Islam, T. (1998). Technological forecasting – Model selection, model sta-
103–125. bility and combining models. Management Science, 44(8), 1115–1130.
Leydesdorff, L. (1998). Theories of citation? Scientometrics, 43(1), 5–25. Meho, L., & Rogers, Y. (2008). Citation counting, citation ranking, and h-index of human-
doi:10.1023/B:SCIE.0000027678.31097.e0. computer interaction researchers: A comparison of Scopus and Web of Science.
Leydesdorff, L. (2007). Scientific communication and cognitive codification: Social sys- Journal American Society for Information Science and Technology, 59(11), 1711–
tems theory and the sociology of scientific knowledge. European Journal of Social 1726.
Theory, 10(3), 375–388. doi:10.1177/1368431007080701. Meho, L., & Yang, K. (2007). Impact of data sources on citation counts and rankings of
Leydesdorff, L. (2008). Caveats for the use of citation indicators in research and journal LIS faculty: Web of Science, Scopus and Google Scholar. Journal American Society
evaluation. Journal of the American Society for Information Science and Technology, for Information Science and Technology, 58(13), 2105–2125.
59(2), 278–287. Merton, R. (1968). The Matthew effect in science. Science, 159(3810), 56–63.
Leydesdorff, L. (2009). How are new citation-based journal indicators adding to the Merton, R. (1988). The Matthew effect in science II: Cumulative advantage and the
bibliometric toolbox? Journal of the American Society for Information Science and symbolism of intellectual property. ISIS, 79, 606–623.
Technology, 60(7), 1327–1336. Michels, C., & Acsmoch, U. (2014). Impact of bibliometric studies on publication be-
Leydesdorff, L. (2012). Alternatives to the journal impact factor: I3 and the top-10% (or haviour of authors. Scientometrics, 98, 369–385.
top-25%?) of the most highly cited papers. Scientometrics, 92, 355–365. Mingers, J. (2002). Can social systems be autopoietic? Assessing Luhmann’s social the-
Leydesdorff, L. (2013). The revised SNIP indicator of Elsevier’s Scopus. Journal of Infor- ory. Sociological Review, 50(2), 278–299.
metrics, 7, 859–860. Mingers, J. (2008a). Exploring the dynamics of journal citations: Mod-
Leydesdorff, L., Alkemade, F., Heimeriks, G., & Hoekstra, R. (2015). Patents as instru- elling with S-curves. Journal Operational Research Society, 59(8), 1013–1025.
ments for exploring innovation dynamics: geographic and technological perspec- doi:10.1057/palgrave.jors.2602428.
tives on “photovoltaic cells”. Scientometrics, 102(1), 629–651. doi:10.1007/s11192- Mingers, J. (2008b). Measuring the research contribution of management academics
014-1447-8. using the Hirsch-index. Journal Operational Research Society, 60(8), 1143–1153.
Leydesdorff, L., & Bornmann, L. (2011a). How fractional counting of citations affects the Mingers, J. (2014). Problems with SNIP. Journal of Informetrics, 8(4), 890–894.
impact factor: Normalization in terms of differences in citation potentials among doi:10.1016/j.joi.2014.09.004.
fields of science. Journal of the American Society for Information Science and Technol- Mingers, J., & Burrell, Q. (2006). Modelling citation behavior in Management Science
ogy, 62(2), 217–229. doi:10.1002/asi.21450. journals. Information Processing & Management, 42(6), 1451–1464.
Leydesdorff, L., & Bornmann, L. (2011b). Integrated impact indicators compared with Mingers, J., & Leydesdorff, L. (2014). Identifying research fields within business and
impact factors: An alternative research design with policy implications. Journal of management: A journal cross-citation analysis. Journal of the Operational Research
the American Society for Information Science and Technology, 62(11), 2133–2146. Society, Online. doi:10.1057/jors.2014.113.
doi:10.1002/asi.21609. Mingers, J., & Lipitakis, E. (2010). Counting the citations: A comparison of Web of Science
Leydesdorff, L., & Bornmann, L. (2015). The operationalization of “fields” as WoS subject and Google Scholar in the field of management. Scientometrics, 85(2), 613–625.
categories (WCs) in evaluative bibliometrics: The cases of “library and information Mingers, J., Macri, F., & Petrovici, D. (2012). Using the h-index to measure the quality
science” and “science & technology studies”. Journal of the Association for Informa- of journals in the field of Business and Management. Information Processing &
tion Science and Technology; early view. Management, 48(2), 234–241. doi:10.1016/j.ipm.2011.03.009.
Leydesdorff, L., Bornmann, L., Mutz, R., & Opthof, T. (2011). Turning the tables on Mingers, J., & Willmott, H. (2013). Taylorizing business school research: On the “one
citation analysis one more time: Principles for comparing sets of documents. Journal best way” performative effects of journal ranking lists. Human Relations, 66(8),
of the American Society for Information Science and Technology, 62(7), 1370–1381. 1051–1073.
doi:10.1002/asi.21534. Mingers, J., & Xu, F. (2010). The drivers of citations in management science journals.
Leydesdorff, L., Bornmann, L., Opthof, T., & Mutz, R. (2011). Normalizing the measure- European Journal of Operational Research, 205(2), 422–430.
ment of citation performance: Principles for comparing sets of documents. Preprint Moed, H. (2000). Bibliometric indicators reflect publication and management strategies.
arXiv: 1101.3863.2011. Retrieved from arXiv website. http://arxiv.org/ Scientometrics, 47(2), 323–346.
Leydesdorff, L., & Felt, U. (2012). “Books” and “book chapters” in the book citation index Moed, H. (2007). The future of research evaluation rests with an intelligent combination
(BKCI) and science citation index (SCI, SoSCI, A&HCI). Proceedings of the American of advanced metrics and transparent peer review. Science and Public Policy, 34(8),
Society for Information Science and Technology, 49(1), 1–7. 575–583.
18 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19

Moed, H. (2008). UK Research Assessment Exercises: Informed judgements on research Ruiz-Castillo, J., & Waltman, L. (2014). Field-normalized citation impact indicators us-
quality or quantity? Scientometrics, 74(1), 153–161. ing algorithmically constructed classification systems of science. Retrieved from
Moed, H. (2010a). CWTS crown indicator measures citation impact of a research group’s http://orff.uc3m.es/handle/10016/18385.
publication oeuvre. Journal of Informetrics, 4(3), 436–438. Saad, G. (2006). Exploring the h-index at the author and journal levels using biblio-
Moed, H. (2010b). Measuring contextual citation impact of scientific journals. Journal metric data of productive consumer scholars and business-related journals respec-
of Informetrics, 4(3), 265–277. tively. Scientometrics, 69(1), 117–120.
Moed, H. (2010c). The Source-Normalized Impact per Paper (SNIP) is a valid and sophis- Schubert, A., & Glänzel, W. (2007). A systematic analysis of Hirsch-type indices for
ticated indicator of journal citation impact. Preprint arXiv:1005.4906. Retrieved journals. Journal of Informetrics, 1(3), 179–184. doi:10.1016/j.joi.2006.12.002.
from arXiv website http://arXiv.org Seglen, P. (1997). Why the impact factor of journals should not be used for evaluating
Moed, H., & Visser, M. (2008). Appraisal of citation data sources. Leiden: Centre for research. British Medical Journal, 314(7079), 498–502.
Science and Technology Studies, Leiden University. Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Infor-
Moed, H., Visser, M., & Buter, R. (2008). Development of bibliometric indicators of research mation Science, 43(9), 628–638.
quality. Centre for Science and Technology Studies, Leiden University. Senge, P. (1990). The fifth discipline: The art and practice of the learning organization.
Moussa, S., & Touzani, M. (2010). Ranking marketing journals using the Google Scholar- London: Century Books.
based hg-index. Journal of Informetrics, 4, 107–117. Sichel, H. (1985). A bibliometric distribution which really works. Journal of the American
Moxham, H., & Anderson, J. (1992). Peer review: A view from the inside. Science and Society for Information Science, 36, 314–321.
Technology Policy, 5(1), 7–15. Silva, F. N., Rodrigues, F. A., Oliveira, O. N., Jr., & da Fontoura Costa, L. (2013). Quantifying
Nacke, O. (1979). Informetrie: Ein neuer Name fur eine neue Disziplin. Nachrichten fur the interdisciplinarity of scientific journals and fields. Journal of Informetrics, 7(2),
Dokumentation, 30(6), 219–226. 469–477. doi:10.1016/j.joi.2013.01.007.
Nalimov, V., & Mulcjenko, B. (1971). Measurement of science: Study of the development Sivertsen, G., & Larsen, B. (2012). Comprehensive bibliographic coverage of the social
of science as an information process. Washington , DC: Foreign Technology Division. sciences and humanities in a citation index: An empirical analysis of the potential.
Nederhof, A. (2006). Bibliometric monitoring of research performance in the social Scientometrics, 91(2), 567–575.
sciences and humanities: A review. Scientometrics, 66(1), 81–100. Small, H. (1973). Co-citation in the scientific literature: A new measure of the rela-
Newman, M., & Girvan, M. (2004). Finding and evaluating community structure in tionship between two documents. Journal of the American Society for Information
networks. Physical Review E, 69(2), 026113. Science, 24(4), 265–269.
Nicolaisen, J. (2007). Citation analysis. Annual Review of Information Science and Tech- Small, H., & Sweeney, E. (1985). Clustering the Science Citation Index using cocitations:
nology, 41(1), 609–641. doi:10.1002/aris.2007.1440410120. A comparison of methods. Scientometrics, 7, 391–409.
Oppenheim, C. (2007). Using the h-index to rank influential British researchers in Thelwall, M., & Vaughan, L. (2004). Webometrics: An introduction to the special issue.
information science and librarianship. Journal of the American Society for Information Journal of the American Society for Information Science and Technology, 55(14), 1213–
Science and Technology, 58(2), 297–301. 1215.
Opthof, T., & Leydesdorff, L. (2010). Caveats for the journal and field normalizations in Thelwall, M., Vaughan, L., & Björneborn, L. (2005). Webometrics. Annual Review of
the CWTS (“Leiden”) evaluations of research performance. Journal of Informetrics, Information Science and Technology, 39(1), 81–135.
4(3), 423–430. Tijssen, R., de Leeuw, J., & van Raan, A. (1987). Quasi-correspondence analysis on square
Ossenblok, T., Engels, T., & Sivertsen, G. (2012). The representation of the social sciences scientometric transaction matrices. Scientometrics, 11(5–6), 347–361.
and humanities in the Web of Science-a comparison of publication patterns and Todeschini, R. (2011). The j-index: A new bibliometric index and mulitvariate compar-
incentive structures in Flanders and Norway (2005–9). Research Evaluation, 21(4), isons between other bibliometric indices. Scientometrics, 87, 621–639.
280–290. Truex III, D., Cuellar, M., & Takeda, H. (2009). Assessing scholarly influence: Using the
Oswald, A. (2007). An examination of the reliability of prestigious scholarly journals: Hirsch indices to reframe the discourse. Journal of the Association for Information
Evidence and implications for decision-makers. Economica, 74(293), 21–31. Systems, 10(7), 560–594.
Otlet, P. (1934). Traité de documentation: le livre sur le livre, théorie et pratique. Brussels: Van Eck, N., & Waltman, L. (2010). Software survey: VOSviewer, a computer program
Editions Mundaneum. for bibliometric mapping. Scientometrics, 84(2), 523–538.
Page, L., Brin, S., Motwani, R., & Winograd, T. (1999). The PageRank citation ranking: van Leeuwen, T. (2006). The application of bibliometric analyses in the evaluation of
Bringing order to the web. social science research. Who benefits from it, and why it is still feasible. Sciento-
Perneger, T.V. (2004). Relation between online “hit counts” and subsequent citations: metrics, 66(1), 133–154. doi:10.1007/s11192-006-0010-7.
prospective study of research papers in the BMJ (Vol. 329). van Raan, A. (2005a). Comparison of the Hirsch-index with standard bibliometric indi-
Pinski, G., & Narin, F. (1976). Citation influence for journal aggregates of scientific publi- cators and with peer judgement for 147 chemistry research groups. Scientometrics,
cations: Theory, with application to the literature of physics. Information Processing 67(3), 491–502.
& Management, 12(5), 297–312. van Raan, A. (2005b). Fatal attraction: Conceptual and methodological problems in the
Pislyakov, V. (2009). Comparing two “thermometers”: Impact factors of 20 leading ranking of universities by bibliometric methods. Scientometrics, 62(1), 133–143.
economic journals according to Journal Citation Reports and Scopus. Scientometrics, van Raan, A. (2005c). Measuring science: Capita selectaof current main issues. In
79(3), 541–550. H. Moed, W. Glenzel, & U. Schmoch (Eds.). Handbook of quantitative science and
Price, D.J.d.S. (1963). Little Science, big science. New York: Columbia University Press. technology research: 19-50. New York: Springer.
Price, D. J. d. S. (1965). Networks of scientific papers. Science, 149(3683), 510–515. van Raan, A., van Leeuwen, T., Visser, M., van Eck, N., & Waltman, L. (2011). Rivals for
Price, D. J. d. S. (1976). A general theory of bibliometric and other cumulative advantage the crown: Reply to Opthof and Leydesdorff. Journal of Informetrics, 4(3), 431–435.
processes. Journal of the American Society for Information Science, 27, 292–306. van Raan, A. J. (2004). Sleeping beauties in science. Scientometrics, 59, 467–472.
Priem, J. (2014). Altmetrics. In B. Cronin, & C. Sugimoto (Eds.), Beyond bibliometrics: Van Veller, M., Gerritsma, W., Van der Togt, P., Leon, C., & Van Zeist, C. (2009). Bibliomet-
Harnessing multidimensional indicators of scholarly impact (pp. 263–288). London: ric analyses on repository contents for the evaluation of research at Wageningen
MIT Press. UR. In A. Katsirikou, & C. Skiadas (Eds.), Qualitative and quantitative methods in
Pritchard, A. (1969). Statistical bibliography or bibliometrics? Journal of Documentation, libraries: Theory and applications (pp. 19–26). World Scientific.
25(4), 348–349. Vanclay, J. (2012). Impact factor: outdated artefact or stepping-stone to journal certi-
Radicchi, F., & Castellano, C. (2012). Testing the fairness of citation indicators for com- fication? Scientometrics, 92(2), 211–238.
parison across scientific domains: The case of fractional citation counts. Journal of Wagner, C. S., Roessner, J. D., Bobb, K., Klein, J. T., Boyack, K. W., Keyton, J., et al.
Informetrics, 6(1), 121–130. doi:10.1016/j.joi.2011.09.002. (2011). Approaches to understanding and measuring interdisciplinary scientific
Rafols, I., Leydesdorff, L., O’Hare, A., Nightingale, P., & Stirling, A. (2012). How journal research (IDR): A review of the literature. Journal of Informetrics, 5(1), 14–26.
rankings can suppress interdisciplinary research: A comparison between innova- doi:10.1016/j.joi.2010.06.004.
tion studies and business & management. Research Policy, 41(7), 1262–1282. Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E., Tijssen, R. J., Eck, N. J., et al. (2012).
Rafols, I., Porter, A. L., & Leydesdorff, L. (2010). Science overlay maps: A new tool The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation.
for research policy and library management. Journal of the American Society for Journal of the American Society for Information Science and Technology, 63(12), 2419–
Information Science and Technology, 61(9), 1871–1887. doi:10.1002/asi.21368. 2432.
Rao, C. (1982). Diversity and dissimilarity coefficients: A unified approach. Theoretical Waltman, L., & van Eck, N. (2013a). Source normalized indicators of citation impact: An
Population Biology, 21(1), 24–43. overview of different approaches and an empirical comparison. Scientometrics, 96,
Reale, E., Barbara, A., & Costantini, A. (2007). Peer review for the evaluation of academic 699–716.
research: Lessons from the Italian experience. Research Evaluation, 16(3), 216–228. Waltman, L., & van Eck, N. (2013b). A systematic empirical comparison of different
Rehn, C., & Kronman, U. (2008). Bibliometric handbook for Karolinska Insti- approaches for normalizing citation impact indicators. Journal of Informetrics, 7(4),
tutet. Retrieved from http://ki.se/content/1/c6/01/79/31/bibliometric_handbook_ 833–849.
karolinska.institutet_v_1.05.pdf. Waltman, L., van Eck, N., van Leeuwen, T., & Visser, M. (2013). Some modifica-
Roemer, R. C., & Borchardt, R. (2012). From bibliometrics to altmetrics: A changing tions to the SNIP journal impact indicator. Journal of Informetrics, 7(2), 272–285.
scholarly landscape. College & Research Libraries News, 73(10), 596–600. doi:10.1007/s11192-011-0354-5.
Rossner, M., Van Epps, H., & Hill, E. (2007). Show me the data. The Journal of Cell Biology, Waltman, L., van Eck, N., van Leeuwen, T., Visser, M., & van Raan, A. (2010). Towards a
179(6), 1091–1092. new crown indicator: Some theoretical considerations. Journal of Informetrics, 5(1),
Rosvall, M., & Bergstrom, C. (2010). Mapping change in large networks. PLoS One, 5(1), 37–47. doi:10.1016/j.joi.2010.08.001.
e8694. Waltman, L., van Eck, N., van Leeuwen, T., Visser, M., & van Raan, A. (2011). To-
Rousseau, R. (2006). New developments related to the Hirsch index. Science Focus, 1(4), wards a new crown indicator: an empirical analysis. Scientometrics, 87(3), 467–481.
23–25. doi:10.1007/s11192-011-0354-5.
Rousseau, R. (2014). Library science: Forgotten founder of bibliometrics. Nature, Weinstock, N. (1971). Citation indexes. In A. Kent (Ed.), Encyclopedia of library and
510(7504), 218-218. information science (pp. 16–41). New York: Marcel Dekker.
 J. Mingers, L. Leydesdorff / European Journal of Operational Research 246 (2015) 1–19 19

West, J., Bergstrom, T., & Bergstrom, C. (2010). Big Macs and Eigenfactor scores: Don’t Zheng, C. (2009). The e-index, complementing the h-index for excess citations. PLoS
let correlation coefficients fool you. Journal of the American Society for Information One, 4(5), e5429.
Science and Technology, 61(9), 1800–1807. doi:10.1002/asi.21374. Zipf, G. (1936). The psycho-biology of language: An introduction to dynamic philology.
Wilhite, A., & Fong, E. (2012). Coercive citation in academic publishing. Science, London: Routledge.
335(6068), 542–543. Zitt, M. (2011). Behind citing-side normalization of citations: Some properties of the
Wilson, C. (1999). Informetrics. Annual Review of Information Science and Technology, journal impact factor. Scientometrics, 89(1), 329–344. doi:10.1007/s11192-011-
34, 107–247. 0441-7.
Wouters, P. (1998). The signs of science. Scientometrics, 41(1), 225–241. Zitt, M., & Bassecoulard, E. (1994). Development of a method for detection and trend
Wouters, P. (2014). The citation: From culture to infrastructure. In B. Cronin, & C. Sug- analysis of research fronts built by lexical or cocitation analysis. Scientometrics, 30,
imoto (Eds.), Beyond bibliometrics: Harnessing multidimensional indicators of schol- 333–351.
arly impact (pp. 47–66). London: MIT Press. Zitt, M., & Small, H. (2008). Modifying the journal impact factor by fractional cita-
Xu, F., Liu, W. B., & Mingers, J. (2015). New journal classification methods based tion weighting: The audience factor. Journal of the American Society for Information
on the global h-index. Information Processing & Management, 51(2), 50–61. Science and Technology, 59(11), 1856–1860. doi:10.1002/asi.20880.
doi:10.1016/j.ipm.2014.10.011.