Signal Processing: Image Communication 27 (2012) 209–219

Contents lists available at SciVerse ScienceDirect

Signal Processing: Image Communication

journal homepage: www.elsevier.com/locate/image

Machine learning to design full-reference image quality

assessment algorithm
Christophe Charrier n, Olivier Lézoray, Gilles Lebrun
Université de Caen Basse-Normandie, GREYC UMR CNRS 6072, Equipe Image, ENSICAEN, 6 Bd. Maréchal Juin, F-14050 Caen, France

a r t i c l e i n f o abstract

Article history: A crucial step in image compression is the evaluation of its performance, and more
Received 13 April 2011 precisely, available ways to measure the quality of compressed images. In this paper, a
Accepted 3 January 2012 machine learning expert, providing a quality score is proposed. This quality measure
Available online 3 February 2012
is based on a learned classification process in order to respect human observers.
Keywords: The proposed method namely Machine Learning-based Image Quality Measure
FR-IQA algorithm (MLIQM) first classifies the quality using multi-Support Vector Machine (SVM) classi-
Classification fication according to the quality scale recommended by the ITU. This quality scale
Theory of evidence contains 5 ranks ordered from 1 (the worst quality) to 5 (the best quality). To evaluate
SVM classification
the quality of images, a feature vector containing visual attributes describing images
SVM regression
content is constructed. Then, a classification process is performed to provide the final
quality class of the considered image. Finally, once a quality class is associated to the
considered image, a specific SVM regression is performed to score its quality. Obtained
results are compared to the one obtained applying classical Full-Reference Image
Quality Assessment (FR-IQA) algorithms to judge the efficiency of the proposed method.
& 2012 Elsevier B.V. All rights reserved.

1. Introduction same quality. This implies that efficient performance mea-

sures are available. The way to evaluate this performance is
Nowadays, it has become an ordinary thing for anybody a crucial step regarding image compression, and more
to take photos with digital cameras, to upload images on precisely available ways to measure the quality of com-
computers and to use softwares to apply many image pressed images. There is a very rich literature on image
processing algorithms on these images (compression, quality criteria, generally dedicated to specific applications
deblurring, denoising, etc.). This is a simple and representa- (optics, detector, compression, restoration, etc.). Quality
tive example of the growing of digital media that is every- evaluation can be divided into two main topics: (1) objec-
where in our world. Many Tera-bytes thus transit on the tive and (2) subjective evaluation.
Internet. In order to reduce the amount of transmitted data, The first topic gives place to two families of criteria:
one typical applied processing on an image is compression, (1) basic and (2) Human Visual System (HVS)-based
so that little data is to be further transmitted. There are criteria. The first family corresponds to the traditional
many compression schemes, such as the so-called JPEG that criteria known as mathematical measures, because they
is the most famous and commonly used. Very recently, result from geometry (concept of distance) or from signal
Google has launched their own compression scheme: WebP processing (signal to noise ratio). The second family of
[1]. It is supposed to offer lesser bit rates than JPEG for the criteria takes into account the characteristics of the
human visual system by a weighting of images’ errors.
This second topic relates to psychophysical experiments
n
Corresponding author. allowing to add a subjective dimension to the quality
E-mail address: [email protected] (C. Charrier). evaluation process. Due to the time-expensive aspect of

0923-5965/$ - see front matter & 2012 Elsevier B.V. All rights reserved.
doi:10.1016/j.image.2012.01.002
210 C. Charrier et al. / Signal Processing: Image Communication 27 (2012) 209–219

this last topic, Image Quality Assessment (IQA) algorithms [5,6]. Since it is not natural for human beings to score the
have been intensively investigated to quantify the quality quality of an image, they prefer to give a semantic descrip-
of a compressed image. tion of what they are watching. This semantic description is
IQA algorithms can be divided into three main topics: usually feeling description: ‘‘it is beautiful’’, ‘‘it is bad’’
(1) Full-reference (FR) IQA methods, (2) Reduced-refer- and so on.
ences (RR) IQA techniques and (3) No-reference (NR) Previous works tried to apply a machine learning-based
IQA algorithms. approach, mainly based on standard back propagation
FR-IQA algorithms refer to algorithms that require the neural network to predict the quality score of a test image
presence of a reference signal for the prediction of the [7–9]. E.g., Bouzerdoum et al. [7] propose a FR-IQA algo-
quality of a test signal while RR-IQA techniques refer to rithm based on a neural network approach. The chosen
algorithms that only require partial information about the neural network is a standard back propagation neural net-
reference signal in order to predict the quality of a test work. Its input layer consists of as many neurons as
signal. NR-IQA solutions refer to algorithm for which the parameters in the input vector. The network has two hidden
reference signal is not available. The two first classes of IQA layers of six neurons each, and one output neuron. The
algorithms can be considered as similarity measures since characteristic vector to be input into the neural network is
the main goal of those methods is to judge how two images chosen to be composed of several elements based on the
are visually close. RR-IQA algorithms provide a solution that Wang et al.’s [10] features. These include the image mean
lies between full-reference and no-reference models. and the image standard-deviation of both the reference and
The usually applied scheme to design an IQA algorithm the test image, the covariance and the MSE between the
consists in performing (1) a color space transformation to reference and the test image. More recently, Narwaria and
obtain decorrelated color coordinates and (2) a decomposi- Lin [11] propose an IQA algorithm based on support vector
tion of these new coordinates towards perceptual channels. regression. The input features are the singular vectors out of
An error is then estimated for each of these channels. A final singular value decomposition. Yet, the proposed approaches
quality score is obtained by pooling these errors in both do not account for the intrinsic classification process of the
spatial and frequency domains. The most common way to quality judgment of human beings.
perform this pooling is to use the Minkowski error metric. All IQA algorithms perform well (in terms of high
Some studies [2] have shown that this summation does not correlation with human ratings) for very poor or very
perform well. The same final value can be computed for two good quality images but in between there are big differ-
different degraded images even if the visual quality of the ences between algorithms. Firstly, one can assume that
two images is drastically different [3]. This is due to the fact for medium quality images, predicted scores do not reflect
that the implicit assumption of this metric is based on the very well human ratings and predicted scores are not as
independence of all signal samples. It is yet commonly good as they should be. In a second interpretation, one
assumed that this is not true when one uses perceptual can assume that an IQA algorithm using the same sensi-
channels. This explains why the Minkowski metric might tivity across the quality continuum would not be able to
fail to generate a good final score. The use of such a metric is refine its prediction for medium quality images. It should
not necessarily the best way to score the quality of a test be better to develop a quality metric that can modulate its
image. Actually, in the recommendations given by the ITU sensitivity with respect to image quality. One way to
[4], the human observers have to choose a quality class from do so is to classify image quality with respect to quality
an integer scale from 0 to 100. Those scores characterize the classes and from the obtained classification, to modelize
quality of the reconstructed images in semantic terms the distribution of each class in order to design a quality
{excellent, very good, good, bad, very bad}. That function the sensitivity of which will differ from others.
way, the human observers make then neither more nor less In this paper, the modelization of the judgment of
than one classification, and the given score could be inter- human beings by a machine learning expert to design a
preted as a confidence of the observer in its judgment. In FR-IQA algorithm is proposed. Fig. 1 displays the general
addition, when a human being judges the quality of an scheme of the Machine Learning-based Image Quality
image, many internal psychophysical scales come into play Measure (MLIQM) used to predict the quality of a test

Fig. 1. General scheme of the proposed method to obtain the final quality score of a test image.
 C. Charrier et al. / Signal Processing: Image Communication 27 (2012) 209–219 211

image. After computing a feature vector including several image quality. Seshadrinathan and Bovik [14] studied the
local quality features, a SVM multi-class classification relationship between the structural similarity (SSIM) [10]
process is performed to provide the final quality class and VIF [15] frameworks and older metrics, i.e. the MSE
C i , 8i 2 ½1; :::; 5 of the test image. Those five correspond and HVS-based quality metrics. They concluded that SSIM
to the quality classes as advocated by recommendation and VIF are closely related to the older IQA metrics under
ITU-R BT.500-11 [4]. Finally, from this classification, a certain natural scene modeling assumptions. This also was
SVM regression process is applied to score the quality of recently studied by Horé and Ziou who defined a bijective
the test image as follows. Each quality class is associated relation between SSIM and PSNR yielding predictions of
with a score range of length 1: the first class is associated SSIM values from PSNR (and inversely) [16]. The global
to the range [0,1[, the second one to the range [1,2[and so conclusion of all those comparison studies is that no IQA
on until the fifth quality class that is associated to the algorithm has been shown to definitively outperform all
range [4,5]. For each class C i , 8i 2 ½1; :::; 5, a regression others for all possible degradations, although owing to the
function is designed in order to score the quality of a inclusion of both scene models and perceptual models, the
degraded image within the associated range. Thus, the MS-SSIM index outperforms many with statistical signifi-
quality of a degraded image is scored between 0 and 5. cance. Thus, factors embedded in the MS-SSIM index will
This way, the proposed IQA method yields a sensitivity serve a spatial criterion as described in Section 2.1.
adaptation to quality image in order to counterbalance Wang et al. [17] have shown that natural images are
medium prediction of usually used IQA techniques. highly structured, in the sense that their pixels exhibit
The paper is structured as follows. In Section 2, we briefly strong dependencies, and these dependencies carry impor-
present the set of features used to describe the quality of tant information about the visual scene. Structural informa-
images. Section 3 details how classification and regression tion is located on visible edges of the image. These edges
are performed with Support Vector Machines. Section 4 correspond to spatial frequency that infers in a positive or
presents obtained results. The last section concludes. negative way with the other frequencies to produce spatial
structures of the image. Thus, spatial-frequency factors are
2. The selected full-reference features computed to take into account the structural information.

When a human being judges the quality of an image,

2.1. Spatial criteria (13 features)
the employed internal scales are supposed to be scales
of comparison for which a ‘‘conscious’’ reference image is
The first selected criteria in our study concern the
required. This is to put in relation with the inherent
factors integrated in the MS-SSIM metric proposed by
conscience of each human being. The conscience is related
Wang and Bovik [18]. These criteria allow us to measure
to what Freud calls ‘‘the perception–consciousness sys-
(1) the luminance distortion, (2) the contrast distortion
tem’’ [12]. It concerns a peripheral function of the psychic
and (3) the structure comparison. Those criteria are
apparatus which receives information from the external
computed considering only the achromatic information.
world and those coming from the memories and the
The authors proposed to represent an image as a vector in
internal feelings of pleasure or displeasure. The immediate
an image space. In that case, any image distortion can be
character of this perceptive function involves an inability for
interpreted as adding a distortion vector to the reference
the conscience of keeping a lasting trace of this information.
image vector. In this space, the two vectors that represent
It sends them to the preconscious, a place of a first setting in
luminance and contrast changes span a plane that is
memory. The consciousness perceives and transmits signif-
adapted to the reference image vector. The image distor-
icant qualities. Freud employs a formula like ‘‘index of
tion corresponding to a rotation of such a plane by an
perception, of quality, of reality’’ to describe the content of
angle can be interpreted as the structural change.
the operations of the perception–conscience system.
The luminance comparison between an original image I
Thus, to design the input features vector of the
and its degraded version J is defined as
classification process, only derived full-reference charac-
teristics are employed. A scalar is then generated for each 2mI mJ þ C 1
lðI, JÞ ¼ ð1Þ
trial feature. The whole set of computed scalars forms the m2I þ m2J þ C 1
feature vector associated to an image. This vector will be
classified to designate the associated class of quality. where mI and mJ , respectively, represent the mean intensity
This section describes the used set of features to of the images I and J, and C1 is a constant for avoiding
measure degradations of the distorted image. This cate- instability when m2I þ m2J  0. A common choice for the
gory of criteria is obtained measuring the dissimilarity stabilizing constant is C 1 ¼ ðK 1 LÞ2 , where L is the theoretical
between an original image and its degraded version. dynamic range of the image’s pixels and K 1 ¼ 0:01.
Many criteria have been developed considering either The contrast distortion measure is defined to have a
the definition domain of the image (e.g. color space), the similar form:
frequency domain (e.g. Fourier transform, DCT) or the 2sI sJ þ C 2
spatio-frequency space (e.g. wavelet transform). cðI, JÞ ¼ ð2Þ
s2I þ s2J þ C 2
Sheik et al. [13] compared 10 recent IQA algorithms
and determined which had particularly high levels of where C2 is a non-negative constant commonly defined
performance. They concluded that more can be done to as C 2 ¼ ðK 2 LÞ2 (K 2 ¼ 0:03), and sI (resp. sJ ) represents the
reduce the gap between machine and human evaluation of standard deviation.
212 C. Charrier et al. / Signal Processing: Image Communication 27 (2012) 209–219

The structure comparison is performed after lumi-

nance subtraction and contrast normalization. The struc-
ture comparison function is defined as
sI, J þ C 3
sðI, JÞ ¼ ð3Þ
sI sJ þ C 3
where C3 is a non-negative constant defined as C 3 ¼ C 2 =2,
P
and sIJ ¼ 1=ðN1Þ N i ¼ 1 ðIi mi ÞðJ i mJ Þ. Substituting C3 by
C 2 =2 in (3)
2sI, J þC 2
sðI, JÞ ¼ ð4Þ
2sI sJ þ C 2

To obtain a multi-scale index, a low-pass filter is applied

to the reference (I) and the distorted images (J). Next a
downsampling of the filtered images by a factor of 2 is
performed. Considering the initial design of the MS-SSIM
index that consists of computing the factors cðÞ and sðÞ at
five different scales, and the luminance lðÞ at the coarser
level, 11 distortion maps are generated. Each of them is
then pooled in a single scalar distortion score, providing
11 factors that are integrated in the feature vector.
Since previous criteria only concern the achromatic
axis, two local descriptors dedicated to chromatic infor-
mation are computed [19]. Those descriptors are not
punctually defined in the image but with respect to the
mean value of the local neighborhood of the pixel. The
two used features are (1) a local chrominance distortion
feature measuring the sensitivity of an observer to color
degradation within a uniform area and (2) a local colori-
Fig. 2. Illustration of the (a) three levels steerable pyramidal decom-
metric dispersion feature that measures the spatio-colori- position of a synthetic image containing a white disk centered on a dark
metric dispersion in each one of the two color images. The background and (b) the associated Fourier transform magnitude of the
calculation of these two descriptors is performed in an four used filters.
antagonist Luminance–Chrominance color space, namely
the CIE Lab color space [20]. These two criteria are also The coefficients induced by the decomposition are
included in the feature vector. next squared to obtain local energy measures. As men-
tioned in [25], those coefficients are normalized to take
2.2. Spatial-frequency criteria (12 features) into account the dynamic limited range of the mechan-
isms in the Human Visual System.
The aim of such features is to model, as well as Let aðx,y, f , yÞ be an original coefficient issued from the
possible, HVS-characteristics such as contrast masking decomposition process located at the position (x,y) in fre-
effects, the luminance variation sensitivity and so on. quency band y and orientation band y. The associated
There are many models to estimate the visibility of errors squared and normalized sensor output rðx,y, f , yÞ is defined as
by simulating the relevant functional properties of the
ðaðx,y, f , yÞÞ2
HVS. All these models perform decomposition of the input rðx,y, f , yÞ ¼ k P 2
ð5Þ
signal into a set of channels, each of them being selec- f20;45,90;135 ðaðx,y, f , fÞÞ þ s2
tively sensitive to a restricted range of spatial frequencies This procedure leads to normalized sensors having a
and orientations, in order to account for the spatial- limited dynamic range. Each sensor is able to discriminate
frequency sensitivity of the HVS. Decompositions mainly contrast differences over a narrow range of contrasts. This
differ from number radial bands, orientations and band- is why the use of multiple contrast bands (with different
width [21–23]. k’s and s’s) is required to discriminate contrast changes
Among all existing decompositions, the steerable pyr- over the full range of contrast.
amid transform [24] is used in this paper to quantify The final stage computes the simple squared error norm
contrast masking effects. The decomposition consists of between the sensor outputs from the reference image
many spatial frequency levels, which are further divided r 0 ðx,y,f , yÞ and the degraded images r 1 ðx,y,f , yÞ for each
into a set of orientation bands. The basis function is frequency band t and orientation band y:
directional derivative operators. In this paper, three levels  2
with four orientation bands with bandwidths of 45 X 
 
degrees 0, 45, 90, 135 plus one isotropic lowpass filter Drð f , yÞ ¼  r0 ðx,y,f , yÞr1 ðx,y,f , yÞ ð6Þ
 x,y 
are used. Fig. 2 presents an example decomposition of a
synthetic image and the associated Fourier transform From this step, 12 scores are available and integrated
magnitude of the four used filters. within the feature vector.
 C. Charrier et al. / Signal Processing: Image Communication 27 (2012) 209–219 213

3. SVM classification and regression 3.1. SVM model selection

From all existing classification schemes, a Support Kernel function choice is critical for the design of a
Vector Machine (SVM)-based technique has been selected machine learning expert. Radial Basic Function (RBF)
due to high classification rates obtained in previous works kernel function is commonly used with SVM. The main
[26], and to their high generalization abilities. The SVMs reason is that RBFs works like a similarity measure between
were developed by Vapnik [27] and are based on the two examples.
structural risk minimization principle from the statistical In this paper, the common One-Versus-One (OO)
learning theory. SVMs express predictions in terms of a decomposition scheme is used to create 10 binary classi-
linear combination of kernel functions centered on a fiers. Let t i,j , 8i 2 ½1; 5, j 2 ½2; 5 be a binary problem with
subset of the training data, known as support vectors (SV). t i,j 2 f þ1,1g. Number 5 represents the final quality
Given the training data S ¼ fðxi ,yi Þgi ¼ f1,...,mg , xi 2 Rn , classes according to the ones recommended by the ITU.
yi 2 f1, þ 1g, SVM maps the input vector x into a high- Let hi ðÞ (Eq. (9)) be the SVM decision function obtained by
dimensional feature space H through some non-linear training it on the ith binary problem. Table 1 gives binary
mapping functions f : Rn -H, and builds an optimal problems transformation used in the OO scheme.
separating hyperplane in that space. The mapping opera- The binary problem transformation is the first part of a
tion fðÞ is performed by a kernel function Kð,Þ which combination scheme. A final decision must be taken from
defines an inner product in H. The separating hyperplane all binary decision functions. Since the SVMs are binary
given by a SVM is w  fðxÞ þ b ¼ 0. The optimal hyperplane classifiers, the resolution of a multi-class problem is
is characterized by the maximal distance to the closest achieved through a combination of binary problems in
training data. The margin is inversely proportional to the order to define a multi-class decision function D. Several
norm of w. Thus computing this hyperplane is equivalent combination schemes of binary classifiers exist [31,32].
to minimize the following optimization problem: One interesting way to achieve this combination is the
! use of the theory of evidence [33,34] since the confidence
1 Xm
Vðw,b, xÞ ¼ JwJ þC2
xi ð7Þ one has in classifier can be taken into account for the final
2 i¼1 assignment decision.

where the constraint 8m i ¼ 1 : yi ½w  fðxi Þ þ b Z 1xi , xi Z 0

requires that all training examples are correctly classified 3.2. The combination of binary classifiers
up to some slack x and C is a parameter allowing trading-
off between training errors and model complexity. This Once the multi-class classifier has been decomposed in
optimization is a convex quadratic programming pro- 10 binary classifiers, one needs to take a decision about the
blem. Its whole dual [27] is to maximize the following final quality class assignment of the input vector. This
optimization problem: assignment is done using the theory of evidence framework
(also known as the Dempster–Shafer theory or the belief
X
m
1 X m
functions theory) [35,33]. Indeed, each of the binary classi-
WðaÞ ¼ ai  a a y y Kðxi ,xj Þ ð8Þ
i¼1
2 i, j ¼ 1 i j i j fiers can be considered as an information source that can be
imprecise and uncertain. Combining these different sources
Pm
subject to 8m i ¼ 1 : 0 r ai r C, i ¼ 1 yi ai ¼ 0. The optimal using the theory of evidence yields to process uncertain
solution an specifies the coefficients for the optimal hyper- information to take the final assignment decision.
P
plane wn ¼ m i ¼ 1 ai yi fðxi Þ and defines the subset SV of all
n
Conceptually, the final decision is taken with respect
support vectors (SV). An example xi of the training set is a to the confidence we have on the results of each binary
SV if ani Z 0 in the optimal solution. The support vectors classifier. The confidence index can be provided in many
subset gives the binary decision function h: different ways: a recognition rate, a likelihood probability,
X an a posteriori probability and so on. Yet, SVMs do not
hðxÞ ¼ signðf ðxÞÞ with f ðxÞ ¼ ani yi Kðxi ,xÞ þ bn ð9Þ
directly provide such a measure.
i2SV
In this paper, an a posteriori probability is computed
n
where the threshold b is computed via the unbounded from the output of the SVM and will serve as confidence
support vectors [27] (i.e., 0 o ani o C). An efficient algo- index. Instead of estimating the class-conditional densi-
rithm SMO (Sequential Minimal Optimization) [28] and ties pðf 9yÞ, a parametric model is used to fit the a posteriori
many refinements [29,30] were proposed to solve dual pðy ¼ 19f Þ where f represents the uncalibrated output
problem. value of SVMs. Platt [28] has proposed a method to compute

Table 1
Binary problems transformation used in a One-Versus-One combination scheme.

Class t 5;4 t 5;3 t 5;2 t 5;1 t 4;3 t 4;2 t 4;1 t 3;2 t3;1 t 2;1

5 þ1 þ1 þ1 þ1 – – – – – –
4 1 – – – þ1 þ1 þ1 – – –
3 – 1 – – 1 – – þ1 þ1 –
2 – – 1 – – 1 – 1 – þ1
1 – – – 1 – – 1 – 1 1
214 C. Charrier et al. / Signal Processing: Image Communication 27 (2012) 209–219

the a posteriori probabilities from the obtained SVM para- normalized:

meters. The suggested formula is based on a parametric X
K¼ m1 ðBÞm2 ðCÞ ð14Þ
form of a sigmoid as B\C ¼ |

1
pðy ¼ 19f Þ ¼ ð10Þ One notes that Dempster’s combination, also known as
1þ expðEf þ FÞ
orthogonal sum and written as m ¼ m1  m2 , is commu-
where parameters E and F are fit using maximum likelihood tative and associative.
estimation from a training set ðf i ,yi Þ. Those parameters are When performing Dempster’s combination, it is crucial
found by minimizing the negative log likelihood of the to take into account the value of K, which is the normal-
training data, which is a cross-entropy error function ization term of the combination: the higher the value, the
defined as more incoherent the combination. When k¼1 one reaches
X a complete opposition and the data fusion is impossible.
min t i logðpi Þ þ ð1t i Þlogð1pi Þ ð11Þ Several solutions have been developed to deal with this
i
conflict term. For example Smets and Kruse [36] proposed
where t i ¼ ðyi þ1Þ=2 represents the target probabilities from to avoid the normalization step, since they considered the
a new training set ðf i ,t i Þ, and pi ¼ 1=ð1þ expðEf i þ FÞÞ. This conflict can only come from a bad definition of O. In that
sigmoid model is equivalent to assume that the SVM out- case, K represents the mass associated to one or more
puts are proportional to the log odds of a positive example. new hypotheses that have not been initially taken into
account.
After performing the combination, the decision asso-
3.2.1. Elements of theory of evidence ciated to the most ‘‘probable’’ element O has to be
Let O ¼ fo1 ; . . . ; oN g be the set of N final classes quantified. Among the existing rules of decision, the most
possible for the quality of an image, called the frame of commonly used is the maximum of the pignistic prob-
discernment. In our study, N ¼5 and O corresponds to ability. This decision rule, introduced by Smets [37], uses
the five final classes ðol Þ1 r l r 5 representing the five the pignistic transformation that allows one to distribute
quality classes {excellent, very good, good, bad, the mass associated to a subset of O over each one of its
very bad} [4]. Instead of narrowing its measures to the elements:
set O (as performed by the theory of probability con- X mðAÞ
strained by its additivity axiom), the theory of evidence BetPðol ,mÞ ¼ , 8ol 2 O, 81 rl r 5 ð15Þ
o 2A D O 9A9
extends on the power set O, labeled as 2O , the set of the l

2N subsets of O. Then, a mass function m is defined and where 9A9 is the cardinal of A. The decision is executed
represents the belief allowed to the different states of the from the highest value of the elements of O.
system, at a given moment. This function is also known as
the initial mass function mðÞ defined from 2O in [0,1] and
3.2.2. Mass function design
corroborating
X One of the main drawbacks of the theory of evidence is
mðAÞ ¼ 1 and mð|Þ ¼ 0 ð12Þ the design of mass functions: the quality of the fusion
ADO process depends on the quality of the mass function.
The design of this mass function is deeply linked to the
where mðAÞ quantifies the belief that the search class
application. Yet, there are three commonly used models:
belongs to the subset AD O (and to none other subset
(1) the distance-based model introduces by Denœux [38]
of A). Subsets A such as mðAÞ 4 0 are referred to as focal
and Denœux and Zoushal [39], (2) Shafer’s model [33]
elements. A represents either a singleton oj or a disjunc-
based on a likelihood function where the conditional a
tion of hypothesis. In the case where the set of hypothesis
priori probability function is supposed to be known and
is exhaustive and exclusive, the mass of the empty set is
(3) Appriou’s models [40] also based on likelihood func-
equal to 0. Such an assumption means that the solution
tions. In [40], the author proposed two models to manage
belongs to the frame of discernment.
the uncertain learning in the framework of evidence
In case of imperfect data (e.g., incomplete or uncertain
theory. Those models are consistent with the Bayesian
data), fusion is an interesting solution to obtain more
approach when the mass is only allocated to singletons.
relevant information. In that case, the combination can be
Among the three previous models, the one proposed
performed from the mass function in order to provide
by Denœux [38] has been retained in our study on account
combined masses synthesizing the knowledge of the
of its integration of both the distance to the neighbors and
different sources.
different criteria of neighborhood in its definition. Thus the
Two initial mass functions m1 and m2 representing the
mass mðfoj gÞ is defined as a decreasing function of the
information providing from two independent sources, can
distance d between the vector to classify and the barycenter
be combined according to Dempster’s [35] rule:
P of the class:
m1 ðBÞm2 ðCÞ (
mðAÞ ¼ B\C ¼ A , 8A 2 2O , Aa| ð13Þ 2
mðol Þ ¼ a expðgl d Þ
1K ð16Þ
mðOÞ ¼ 1mðol Þ
where K is known as the conflict factor and represents the
discrepancy between the two sources. It corresponds where 0 o a o 1 is the a posteriori probability computed
to the mass of the empty set if the masses are not from the binary SVM dedicated to the class ol . gl depends
 C. Charrier et al. / Signal Processing: Image Communication 27 (2012) 209–219 215

on the class ol and is computed by minimization of an error Mathematically, the problem becomes
criterion using the SEM (Stochastic Expectation Maximiza- !
tion) algorithm [41]. 1 1X m
min n JwJ2 þ C nE þ
n
ðxi þ xi Þ ð18Þ
The mass functions yield to take into account the w, E, xi ,xii 2 mi¼1
associated uncertainty to each one of the classifiers.
subject to 8m i ¼ 1 , yi f i r E þ xi , f i yi r E þ xi , xi , xi Z 0 and
n n
Thus, close classes are brought together in the same focal
element, and the final decision is taken only after combin- E Z 0. Schölkopf and Smola [43] have shown that n is an
ing the obtained results from other projections. upper bound of the fraction of margin errors and a lower
To construct such a focal element, the input vector is bound of the fraction of SV. Furthermore, they have
not associated to only one class from fo1 , o2 , o3 , o4 , o5 g, shown that, with probability 1, n equals to both fractions.
but to a subset of classes corresponding at most to O. To Thus, in situations where prior knowledge on these
generate such a subset, the affectation constraint has to fractions is available, n is much easier to adjust than E.
be loosened. One way to perform that is to generate In this paper, the RBF is chosen as kernel for n-SVR. For
an interval computed from the maximum value of the each quality class, a n-SVM is trained in order to estimate
a posteriori probabilities to generate the subset A such as function f as defined in Eq. (9) using the quality scores of
the training sets. In order to be coherent with the ITU
A ¼ fol 2 O=maxðpl Þdl rpl rmaxðpl Þg ð17Þ
scale, a numerical scale is assigned to each quality class.
where l 2 f1, . . . ,5g and dl is an ad hoc constant depending The range of the five quality scales is [0;5] and each
on the used classifier. quality scale has a numerical scale of length 1. Thus the
In that case, all the classes for which their probabilities quality class ‘‘very bad quality’’ is associated to the scale
are included within this new interval are considered as [0,1], the following one ‘‘bad quality’’ is associated to the
candidates for classification during the fusion process. scale ]1;2], and so on until the final quality class ‘‘excel-
lent’’ that is associated to the scale ]4;5]. Thus, no overlap
between scores obtained from different classes is possible.
3.3. SVM regression scheme Finally, one obtains five regression functions asso-
ciated to each quality class applying the One-Versus-All
Even if scoring the quality of an image is not natural approach. When a distorted image is first classified within
for human beings, it is quite necessary to obtain a scalar a quality class, the associated regression function yields to
quality score. The main reason is due to the fact that total score the quality of that image using a scalar number
order only exists in the real set R. depending on the associated quality class. When all the
SVMs can be applied not only to classification pro- score ranges for all five regression functions are consid-
blems but also to the case of regression. Our SVM-based ered, a continuous score scale from 0 to 5 is available to
classifier does not directly provide any quality score. In predict the quality of a candidate image.
order to provide such a quality score, we use the support
vector regression technique referred to as n-SVR [42]
which is commonly used to solve regression problems. 4. Experimental setup and performance measure
In particular n-SVR has the advantage of being able to
automatically adjust the width of the E-tube [42]. 4.1. Experimental setup
We first present the E-SVR and then present n-SVR
as an improvement [42,27]. Given the training data S ¼ 4.1.1. The used image databases
fðxi ,yi Þgi ¼ f1,...,mg , xi 2 Rn , yi 2 f1, þ1g. In E-SVR, x is first To judge the performance of the proposed approach,
mapped to z ¼ FðxÞ in feature space, then a linear function two different image databases are used: (1) the LIVE
f ðx,wÞ ¼ wT z þ b is constructed in such a way that it database release 2 [44] and (2) the TID2008 database [45].
deviates as less as possible from the training set according The LIVE database consists of five subsets of five types of
to a E-insensitive loss function: distortions: (1) JPEG2000 distortions (227 images), (2) JPEG
( distortions (233 images), (3) White noise distortions (174
0 if 9yf ðxÞ9 o E
9yf ðxÞ9E ¼ images) (4) Gaussian blur distortions (174 images) and
9yf ðxÞ9E otherwise
(5) Fast-fading Rayleigh channel distortions (which are
simulated with JPEG2000 compression followed by channel
while JwJ is as small as possible. This is equivalent to
bit-errors) (174 images). The subjective ratings (that will
minimize
! serve as groundtruth) in its Differential Mean Opinion Score
1 Xm
(DMOS) form are also available.
min JwJ2 þC
n
ðxi þ xi Þ
2 The TID2008 database contains 25 reference images
i¼1
and 1600 distorted images using 16 distortion types, as
subject to 8m i ¼ 1 , yi f i r E þ xi , f i yi r E þ xi , xi , xi Z 0 where
n n
described in Table 2. The MOS value of each image is
f i ¼ f ðxi ,wÞ and C is a user-defined constant. After training, provided too.
n
those nonzero xi ’s and xi ’s will be exactly equal to the The training and test sets design. To apply the MLIQM
difference between the corresponding yi and f i . classification process, two distinct sets have been gener-
A drawback of E-SVR is that E can be difficult to tune. ated from the trail databases: the training sets and the
n-SVR alleviated this problem trading off E against model test sets. Since five quality classes are used, ten OO-SVM
complexity and training error using parameter n 4 0. classifiers are designed.
216 C. Charrier et al. / Signal Processing: Image Communication 27 (2012) 209–219

Table 2 measures can be interpreted as prediction accuracy mea-

Description of the 17 degradation types within the TID2008 database. sures (Pearson and Kendall coefficients) and prediction
monotonicity measure (Spearman coefficient).
Degradation Type of distortion
#
4.2. Results
1 Additive Gaussian noise
2 Additive noise in color components is more intensive All three correlation coefficients (LCC, KROCC, SROCC)
than additive noise in the luminance component
3 Spatially correlated noise
have been computed between the predicted values and
4 Masked noise the subjective DMOSs considering the test set TestC1, the
5 High frequency noise entire LIVE database and the entire TID2008 database.
6 Impulse noise Since similar results have been obtained for the three
7 Quantization noise
correlation coefficients, only SROCC is reported.
8 Gaussian blur
9 Image denoising Fig. 3 presents SROCC values obtained between the
10 JPEG compression predicted values and the subjective DMOSs considering both
11 JPEG2000 compression the test set TestC1 and the entire LIVE database for all the five
12 JPEG transmission errors trial FR-IQA methods. Concerning the MLQIM algorithm, the
13 JPEG2000 transmission errors
14 Non-eccentricity pattern noise
displayed results are median values of SROCC. From the
15 Local block-wise distortions of different intensities correlation evaluation results, we see that the performance of
16 Mean shift (intensity shift) the MLIQM is significantly better than for the four tested FR-
17 Contrast change IQA algorithms when whole LIVE database is considered. For
most subsets of LIVE, the use of MLIQM provides consistent
improvement in the performance of IQA algorithms for
One training set (TrainC1) is generated from LIVE different correlation coefficients. Even if all improvements
database. This is composed of the degraded versions of are not significant (which is not really surprising since several
12 images of the LIVE image database, for all kinds of trial IQA measures achieve high performance on LIVE), this
degradation. The LIVE test set (TestC1) is composed of the consistency of improvement can be interpreted as an
degraded versions of the 13 remaining images. indicator of the validity of the proposed approach. A second
To complete n-SVM regression, five training sets (TrainR1, interpretation concerns the selected features. As they are of
TrainR2, y, TrainR5) are generated for each quality class, prime importance to reach high quality results for machine
following the same previous design process. This will result learning classification and regression, this improvement
in five regression functions design, i.e., one per quality class. tends to demonstrate that the used features are relevant
The parameters of both the SVM classification scheme to design SVM classification and regression-based NR-IQA
and the n-SVM regression scheme are determined using algorithm. Even if MLIMQ seems to be less performant for
a 10-fold cross-validation technique on the training sets. fast fading degradation (that uses JP2K), the difference of
In addition, a bootstrap process with 999 replicates is correlation coefficients with the best IQA method is not
used to quantify the performance of MLIQM. significantly different. Fig. 4 presents an example of pre-
As training is only applied on LIVE subsets (TrainC1, dicted quality scores by MLIQM on degraded images.
TrainR1, TrainR2, y, TrainR5), the entire TID2008 image These high obtained correlation coefficient values
database will serve as test set as well as the subset TestC1. were expected since the training sets used to train the
Performance measures. To measure the performance of SVM classifier and the SVM regression scheme were
the proposed approach, a comparison with usual state-of- generated from LIVE database.
the-art FR-IQA algorithms is performed. These FR-IQA Fig. 5 displays the performance of the trail IQA algo-
techniques are MS-SSIM [10], VSNR [46], VIF [15] and rithms with the TID2008 image database. No new training
PSNR. All these methods are computed using the lumi- phase has been performed. This means that shown results
nance component of the images. are obtained from the MLIQM technique trained on TrainC1
To provide quantitative performance evaluation, three and (TrainR1, y, TrainR5) sets for, respectively, the SVM
measures of correlation have been used: (1) Pearson, classification step and the SVM regression step. The pro-
(2) Kendall and (3) Spearman measures. To perform the posed approach yields to obtain high SROCC values for most
Pearson correlation measures (CC), a logistic function (as subsets of TID database. Except for degradations #5, #7,
adopted in the video quality experts group (VQEG) Phase I #12, #15, #16 and #17, MLIQM provides improvement of
FR-TV test [47]) was used to provide a non-linear map- performance. In addition, when all subsets are considered,
ping between the predicted values and subjective scores. the proposed scheme significantly outperforms the trial NR-
This function is a three-parameter logistic function: IQA algorithms, namely MS-SSIM, VSNR, VIF and PSNR.
Degradations #5 and #7, respectively, deal with high
b1
rðxÞ ¼ ð19Þ frequency noise and quantization noise. Considering the
1 þ expðb2 ðxb3 ÞÞ
first kind of artifact, the difference of correlation between
This nonlinearity is applied to the FR-IQA algorithm score, the best IQA algorithm (MS-SSIM) and the MLIQM approach
which gives a better fit for all data. Kendall (KROCC) and is not statistically significant. This is not true if the second
Spearman (SROCC) rank order correlation measures were degradation is highlighted. This degradation can be inter-
computed between the DMOS values and the predicted preted as a loss of color, which induces artificial structural
scores obtained using any trial FR-IQA algorithms. Those information (edges) for strong quantization. In that case,
 C. Charrier et al. / Signal Processing: Image Communication 27 (2012) 209–219 217

Fig. 3. Obtained correlation coefficients between the predicted DMOS values and the subjective DMOSs considering LIVE database test set.

Fig. 4. Example of results obtained computing the trial FR-IQA algorithms on an original image (churchandcapitol extracted from LIVE and its
degraded versions by applying JPEG (0.83865 bpp), JPEG2000 (0.194 bpp), Gaussian blur (s ¼ 1:565074) and a fast fading process (receiver SNR¼ 18.9).

structural dissimilarities are high and are perfectly captured block-wise distortions of different intensities) can be con-
using MS-SSIM index. The used entry features for MLIQM sidered as transmission errors since local blocks of the image
contain many other features that could blur the information are color degraded. As for degradation #12, a small correla-
provided by dedicated structural features. Yet, the correla- tion difference is noticeable between MS-SSIM and MLIQM.
tion difference between the two approaches (MS-SSIM and Degradations #16 and #15, respectively, concern a
MLIQM) is small. change of intensity and of contrast. They cannot be
Considering compression oriented degradations, except considered only as a degradation process, but also as a
for degradation #12 (JPEG transmission errors), MLIQM change of the naturalness of images. When analyzing the
yields an increase of SROCC values for compression- images corresponding to the considered degradation,
degraded images. In addition, degradation #15 (local visible differences between the reference image and the
218 C. Charrier et al. / Signal Processing: Image Communication 27 (2012) 209–219

Fig. 5. Obtained Spearman rank order correlation coefficient (SROCC) between the predicted DMOS values and the subjective DMOSs considering
TID2008 database as test set. The type of degradations is described in Table 2.

degraded versions are not necessarily great. Nevertheless, details can modify the final human judgment. The fact
for these degradations, a small difference of correlation is that humans are able to rank order the quality of images
between the best IQA algorithm and the MLQIM. can be modeled by a decision function. This function can be
Finally, considering the entire TID database, MLIQM formulated as a non-linear regression function. The belief
yields (1) a higher correlation rate and (2) a statistically any human being can formulate about his decision can be
significant difference with the other trial IQA schemes. In summarized by soft margin used to define the non-linear
addition, adding more elements associated to degradation decision function in the regression process. This is a new
for which MLIQM is less performant, the proposed approach framework to design FR-IQA algorithms.
should perform better (since 100 images for those degrada- Yet, even if the proposed scheme seems to be validated,
tions do not seem to reach a relevant training process). the obtained results are deeply linked to the extracted
The same final remark formulated for obtained results features: in this case, a mere adjustment of the used vector
on LIVE can be applied to TID: this consistency of of features and of the used kernel functions is required.
improvement for subsets as for the entire TID database The fact that correlation results obtained with MLIQM
can be considered as an indicator of the validity of the (which integrate MS-SSIM factors) are most of the time
proposed approach. better than those obtained with MS-SSIM yields to per-
The complexity of the proposed approach relies on the haps hypothesize that the original combination of the
training phase in order to design both the classification MS-SSIM factors is not necessarily optimal. Maybe some
process and the regression scheme. This phase can (and artifacts might not be well taken into account too.
should) be done offline, as a preprocessing stage. Actually,
both SVMs and n-SVRs training are of high complexity. 6. Conclusion
Once MLIQM is trained, during the online stage, its
complexity depends on the complexity of feature extrac- In this paper a new approach to design a FR-IQA
tion process, since the complexity associated to both algorithm is proposed. This approach is based on a
classification and regression stage can be neglected. Even classification process such as the human being is supposed
if this complexity is higher than simple IQA algorithms, it to proceed to judge the quality of an object. To apply the
is acceptable since MLIQIM provides very high correlations classification process, a vector of features has been gener-
obtained with respect to human judgments (and it out- ated. The selected features are chosen from full-reference
performs IQA algorithms for some degradation). image HVS-based features and full-reference image fea-
tures, for both of them a reference image is needed.
5. Discussion The compared techniques with the proposed LMIQM
method are four state-of-the-art FR-IQA methods. The
The proposed FR-IQA algorithm based on SVM classifi- obtained results show that LMIQM gives better results
cation and regression to compute the quality score of an and yields a significant improvement of the correlation
image seems to be a promising new way of design, since coefficients with human judgments.
whatever the used database (LIVE and TID), the consistency
of the correlation improvement is observed. This means
that the formulated hypothesis concerning the classifica- Acknowledgment
tion process used by human beings when watching scenes
is valid. Furthermore, this classification strategy can be This work is supported by the ANR project #ANR-08-
modeled by a high dimensional classifier since many SECU-007-04.
 C. Charrier et al. / Signal Processing: Image Communication 27 (2012) 209–219 219

References [24] E.P. Simoncelli, W.T. Freeman, The steerable pyramid: a flexible
architecture for multi-scale derivative computation, in: ICIP,
Washington, DC, 1995, pp. 444–447.
[1] Google, WebP compression scheme, /http://code.google.com/intl/ [25] P.C. Teo, D.J. Heeger, Perceptual image distortion, in: ICIP, vol. 2,
fr/speed/webp/S. 1994, pp. 982–986.
[2] Z. Wang, A.C. Bovik, E.P. Simoncelli, Structural approaches to image [26] G. Lebrun, C. Charrier, O. Lezoray, C. Meurie, H. Cardot, Fast pixel
quality assessment, in: Handbook of Image and Video Processing, classification by SVM using vector quantization, tabu search and
2nd ed., Academic Press, 2005, pp. 961–974. hybrid color space, in: The 11th International Conference on CAIP,
[3] Z. Wang, A.C. Bovik, Mean squared error: love it or leave it? A new Rocquencourt, France, 2005, pp. 685–692.
look at signal fidelity measures, IEEE Signal Processing Magazine 26
[27] V.N. Vapnik, Statistical Learning Theory, Wiley, New York, 1998.
(1) (2009) 98–117. [28] J. Platt, Fast training of support vector machines using sequential
[4] ITU-R Recommendation BT.500-11, Methodology for the Subjective
minimal optimization, in: Advances in Kernel Methods—Support
Assessment of the Quality of Television Pictures, Technical Report,
Vector Learning, MIT Press, 1999.
International Telecommunication Union, Geneva, Switzerland,
[29] R. Collobert, S. Bengio, SVMTorch: support vector machines for
2002.
large-scale regression problems, Journal of Machine Learning
[5] H. Hemminger, P. Mahler, Psychophysical saturation scales and
Research 1 (2001) 143–160.
the spectral sensitivity in human vision, Psychological Research
[30] C.-C. Chang, C.-J. Lin, LIBSVM: a library for support vector machines,
42 (1980) 207–212.
Software, 2001. Available at: /http://www.csie.ntu.edu.tw/  cjlin/
[6] R.N. Shepard, Psychological relations and psychophysical scales: on
libsvmS.
the status of direct psychophysical measurement, Journal of Math-
[31] C.-W. Hsu, C.-J. Lin, A comparison of methods for multiclass
ematical Psychology 24 (1) (1981) 21–57.
support vector machines, IEEE Transactions on Neural Networks
[7] A. Bouzerdoum, A. Havstad, A. Beghdadi, Image quality assessment
13 (3) (2002) 415–425.
using a neural network approach, in: Fourth IEEE International
[32] O. Lezoray, H. Cardot, Comparing combination rules of pairwise
Symposium on Signal Processing and Information Technology,
neural networks classifiers, Neural Processing Letters 27 (1) (2008)
2004, pp. 330–333.
43–56.
[8] P. Gastaldo, R. Zunino, I. Heynderickx, E. Vicario, Objective quality
[33] G. Shafer, A Mathematical Theory of Evidence, Princeton University
assessment of displayed images by using neural networks, Signal
Press, 1976.
Processing: Image Communication 20 (2005) 643–661.
[34] B. Quost, T. Denœux, M.-H. Masson, Pairwise classifier combination
[9] R.V. Babu, S. Suresh, A. Perkis, No-reference JPEG image quality
using belief functions, Pattern Recognition Letters 28 (5) (2007)
assessment using GAP-RBF, Signal Processing 87 (6) (2007)
644–653.
1493–1503.
[35] A. Dempster, Upper and lower probabilities induced by multi-
[10] Z. Wang, E.P. Simoncelli, A.C. Bovik, Multi-scale structural similar-
ity for image quality assessment, in: IEEE Asilomar Conference on valued mapping, Annals of Mathematical Statistics 38 (1967)
Signals Systems and Computers, 2003, pp. 1398–1402. 325–339.
[11] M. Narwaria, W. Lin, Objective image quality assessment based on [36] P. Smets, R. Kruse, The transferable belief model for belief repre-
support vector regression, IEEE Transactions on Neural Networks sentation, in: P.S.A. Motro (Ed.), Uncertainty Management in Infor-
21 (3) (2010) 515–519. mation Systems: From Needs to Solutions, Kluwer, Boston, 1997.
[12] D. Rapaport, M.M. Gill, The points of view and assumptions [37] P. Smets, Constructing the pignistic probability function in a
of metapsychology, The International Journal of Psychoanalysis context of uncertainty, in: Uncertainty in Artificial Intelligence,
40 (1959) 153–162. vol. 52, Elsevier Science Publishers, 1990, pp. 29–39.
[13] H.R. Sheikh, M.F. Sabir, A.C. Bovik, A statistical evaluation of recent [38] T. Denœux, A k-nearest neighbor classification rule based on
full reference image quality assessment algorithms, IEEE Transac- Dempster–Shafer theory, IEEE Transactions on Systems, Man and
tions on Image Processing 5 (11) (2006) 3441–3452. Cybernetics 25 (5) (1995) 804–813.
[14] K. Seshadrinathan, A.C. Bovik, Unifying analysis of full reference [39] T. Denœux, L.M. Zouhal, Handling possibilistic labels in pattern
image quality assessment, in: IEEE International Conference on classification using evidential reasoning, Fuzzy Sets and Systems
Image Processing (ICIP), 2008, pp. 1200–1203. 122 (2) (2001) 47–62.
[15] H.R. Sheikh, A.C. Bovik, Image information and visual quality, IEEE [40] A. Appriou, Probabilités et incertitude en fusion de données multi-
Transactions on Image Processing 15 (2) (2006) 430–444. senseurs, Revue Scientifique et Technique de la Défense 11 (1991)
[16] A. Horé, D. Ziou, Image quality metrics: Psnr vs. ssim, in: IEEE 27–40.
International Conference on Pattern Recognition (ICPR), Istanbul, [41] G. Celeux, J. Diebolt, The SEM algorithm: a probabilistic teacher
Turkey, 2010, pp. 2366–2369. algorithm derived from the EM algorithm for the mixture problem,
[17] Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality Computational Statistics Quarterly (2) (1985) 73–82.
assessment: from error measurement to structural similarity, IEEE [42] A.J. Smola, B. Scholkopf, A Tutorial on Support Vector Regression,
Transactions on Image Processing 13 (1) (2004) 1–14. Technical Report, NeuroCOLT Technical Report (NC2-TR-1998-030),
[18] Z. Wang, A.C. Bovik, A universal quality index, IEEE Transactions on Royal Holloway College, University of London, UK, 1998.
Image Processing 9 (3) (2002) 81–84. [43] B. Schölkopf, A.J. Smola, New Support Vector Algorithms, Technical
[19] A. Trémeau, C. Charrier, E. Favier, Quantitative description of image Report, NeuroCOLT Technical Report (NC2-TR-1998-031), Royal
distortions linked to compression schemes, in: Proceedings of the Holloway College, University of London, UK, 1998.
International Conference on the Quantitative Description of Mate- [44] Laboratory for Image & Video Engineering, University of Texas
rials Microstructure, Warsaw, 1997, qMAT’97. (Austin), LIVE Image Quality Assessment Database, /http://live.ece.
[20] M.W. Schwartz, W.B. Cowan, J.C. Beatty, An experimental compar- utexas.edu/research/QualityS.
ison of RGB, YIQ, L*a*b*, HSV and opponent color models, in: ACM [45] N. Ponomarenko, M. Carli, V. Lukin, K. Egiazarian, J. Astola,
Transactions on Graphics, vol. 6, 1987, pp. 123–158. F. Battisti, Color image database for evaluation of image quality
[21] A.B. Watson, The cortex transform: rapid computation of simulated metrics, in: International Workshop on Multimedia Signal Proces-
neural images, Computer Vision, Graphics and Image Processing sing, Australia, 2008, pp. 403–408.
39 (1987) 311–327. [46] D.M. Chandler, S.S. Hemami, VSNR: a wavelet-based visual signal-
[22] J. Lubin, The use of psychophysical data and models in the analysis to-noise ratio for natural images, IEEE Transactions on Image
of display system performance, in: Digital Images and Human Processing 16 (9) (2007) 2284–2298.
Vision, MIT Press, 1993, pp. 163–178. [47] VQEG, Final Report from the Video Quality Experts Group on the
[23] S. Daly, A visual model for optimizing the design of image proces- Validation of Objective Models of Video Quality Assessment,
sing algorithm, in: ICIP, vol. 2, 1994, pp. 16–20. Technical Report, 2000.