See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/26531994

Feature Extraction Methods for Real-Time Face Detection and Classification

Article in EURASIP Journal on Advances in Signal Processing · August 2005

DOI: 10.1155/ASP.2005.2061 · Source: DOAJ

CITATIONS READS
6 220

3 authors, including:

David Masip Jordi Vitrià

Universitat Oberta de Catalunya University of Barcelona
86 PUBLICATIONS 1,225 CITATIONS 256 PUBLICATIONS 3,633 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Jordi Vitrià on 01 June 2014.

The user has requested enhancement of the downloaded file.

EURASIP Journal on Applied Signal Processing 2005:13, 2061–2071

c 2005 David Masip et al.

Feature Extraction Methods for Real-Time Face

Detection and Classification

David Masip
Centre de Visió per Computador (CVC), Departamento de Informática, Universitat Autònoma de Barcelona, Bellaterra 08193, Spain
Email: [email protected]

Marco Bressan
Centre de Visió per Computador (CVC), Departamento de Informática, Universitat Autònoma de Barcelona, Bellaterra 08193, Spain
Email: [email protected]

Jordi Vitrià
Centre de Visió per Computador (CVC), Departamento de Informática, Universitat Autònoma de Barcelona, Bellaterra 08193, Spain
Email: [email protected]

Received 22 December 2003; Revised 29 November 2004

We propose a complete scheme for face detection and recognition. We have used a Bayesian classifier for face detection and a
nearest neighbor approach for face classification. To improve the performance of the classifier, a feature extraction algorithm
based on a modified nonparametric discriminant analysis has also been implemented. The complete scheme has been tested in a
real-time environment achieving encouraging results. We also show a new boosting scheme based on adapting the features to the
misclassified examples, achieving also interesting results.
Keywords and phrases: face detection, face recognition, boosting, feature extraction.

1. INTRODUCTION (1) Knowledge-based methods, where some rules or re-

lationships between features are encoded. Kotropoulos and
As computers become faster and faster, new applications
Pitas followed this approach using projection profiles to lo-
dealing with human faces become possible. Examples of this
cate the face [2]. We have used a similar technique to locate
applications are face recognition applied to surveillance sys-
the eyes once the face has been detected using our method.
tems, gesture analysis applied to user-friendly interfaces, or
(2) Feature-invariant approaches, where the idea is to de-
gender recognition applied to reactive marketing. We will
tect the facial features first, such as eyes, mouth, eye brows,
propose here a global face detection and recognition frame-
and group them into candidate faces [3].
work, which has achieved good results in an uncontrolled
(3) Template-matching methods, where there is a prede-
environment. Usually, working under uncontrolled condi-
fined face pattern that is correlated with the image. Point dis-
tions is one of the hardest problems of computer vision, for
tribution models (PDMs) have also been used for this pur-
example, in applications where illumination presents strong
pose [4].
changes, or where we have to deal with objects under un-
(4) Appearance-based methods, where the goal is to train
predictable movements. We have tested the system in a real
a classifier that learns the features of the faces from a train-
environment, with no restrictions on scale and illumination,
ing set with face and nonface images. Many classic techniques
achieving real-time satisfying performance.
such as principal component analysis [5], Gaussian mixture
According to [1] face detection schemes can be classified
models [6], neural networks [7], hidden Markov models [8],
into four different categories, although some methods can
support vector machines [9], and probabilistic models [6]
belong to more than one category.
have been applied in this approach.
Many methods of face recognition have also been pro-
This is an open access article distributed under the Creative Commons posed. Basically they can be divided into holistic template-
Attribution License, which permits unrestricted use, distribution, and matching-based systems, geometrical local-feature-based
reproduction in any medium, provided the original work is properly cited. schemes, and hybrid schemes [10].
2062 EURASIP Journal on Applied Signal Processing

(i) The holistic methods use the whole image as a raw In the next section, we introduce the face detection
input to the learning process. Examples of this techniques are method that is based on a classifier combination technique,
principal component analysis [5], independent component the AdaBoost algorithm. We also will suggest a modification
analysis [11], or support vector machines [12] applied to face of the algorithm. Then we will focus on the classification en-
recognition. gine for face recognition, and the discriminant analysis used
(ii) In the feature-based schemes, some structural fea- as feature extraction will be presented. In Section 4 we will
tures are extracted, such as eyes, mouth, and their local ap- show the performed experiments. First we have tested the de-
pearance, position, or relative relationship are used for train- tection scheme using images extracted from two public face
ing the classifier. The most successful technique is the elastic databases. Also the recognition layer has been tested in the
bunch graph matching presented in [13] where the authors real environment. We finalize the paper with the conclusions.
use Gabor wavelets to extract the basic features for the graph-
matching scheme.
2. FACE DETECTION
(iii) Hybrid methods try to use the best of the holistic and
feature-based approaches combining local features and the Today, the most promising approach for face detection is an
whole face to recognize. An example of hybrid methods is the appearance-based method that is based on the classification,
use of eigenfeatures [14], which extends the idea of eigenfaces using ensemble methods, of an overcomplete set of simple
to specific regions of the face such as mouth, nose, or eyes. image features. This approach was first proposed by Viola
Among the holistic methods, appearance-based methods and Jones [21] and developed by Lienhart [22]. The main
are the most successful. They are commonly implemented characteristics of this approach are a cascade architecture,
following these steps. an overcomplete set of rectangle features, and an algorithm
based on AdaBoost for constructing ensembles of rectangle
(1) Image preprocessing [15], where usually an illumina-
features in each classifier node. Much of the recent work on
tion correction is performed, followed by the localization of
face detection following Viola-Jones has explored alternative
some parts of the face for geometrical alignment that makes
boosting algorithms such as FloatBoost [23], GentleBoost
the feature-based approaches more accurate. Usually the cen-
[22], and asymmetric AdaBoost [24]. In the next section we
ter of the eyes are located, and faces are warped in such a way
present the AdaBoost approach, that also constitutes the cen-
that distance between eyes remains stable within subjects.
tral part of our detector.
(2) Feature extraction. Dimensionality reduction tech-
niques have shown important advantages in some pattern 2.1. Classifier combination and AdaBoost
recognition tasks and face processing is not an exception. Sometimes it is possible to improve the performance of a
Usually we achieve a compression of the input data, reduc- poor classifier by combining multiple instances of it, that will
ing the storage needs. In other cases there is also an im- be called weak classifiers, in a more powerful decision rule. In
provement of the classification results due to reduction of the the literature, we can find three different approaches to do it:
noise present in the most part of the natural images. Princi- boosting [25], bagging [26], and random subspace methods
pal component analysis is perhaps one of the most spread di- [27]. The difference between these algorithms is the way how
mensionality reduction techniques [5, 16]. The goal in PCA they combine the classifiers. The main idea of random sub-
is to find a linear projection that preserves the maximum space methods is to use only a subset of the features of each
amount of input data variance. Another approach is to take vector. Different classifiers are created using random sub-
into account the labels of the input data points, and try to spaces by sampling the original-data feature space. The final
find the linear projection that best discriminates the space in decision rule is usually a majority voting among the different
different classes. Fisher linear discriminant analysis is an ex- classifiers. The RSM method is specially useful when dealing
ample of this kind of techniques [17, 18]. Nevertheless, the with high-dimensional data and reduced training sets.
algorithm has some limitations, because the dimensionality Bagging combines the results of different classifiers by
of the resulting space after the projection is upper bounded sampling the training set. A new classifier is trained using
by the number of classes, and there is also a Gaussian as- each subset of the training vectors, and then every test vector
sumption in the input data. In this paper we will introduce is classified using all the classifiers, and a final decision rule
the use of nonparametric discriminant analysis [19] for face is performed (weighted majority voting, etc.). Bagging is of-
recognition, a technique that overcomes the drawbacks of ten useful when we have outliers in the training set, because
FLD, and a modification of the original NDA algorithm that they are isolated in some subsets and do not alter the global
increases its performance. results.
(3) Feature classification. Once the proper features are ex- The idea behind boosting is similar to bagging, but the
tracted, any classifier can be applied. The most common ones classifiers are trained in a serialized way. At each training
are the nearest-neighbor classifier using either Euclidean or step a new classifier is created. These classifiers are called
angle distance, and neural networks. weak classifiers. The training samples are classified using it,
Most of these methods have been successfully used in and a set of weights are modified according to the classifica-
artificial environments, but do not perform well in many tion results (the weights of the misclassified examples are in-
real-world situations as several independent tests have doc- creased). Then a new classifier is trained taking into account
umented [20]. these weights. The algorithm is iterated a certain number of
Feature Extraction for Face Classification 2063

Given the training samples X1 , X1 , . . . , Xn

(1) Initialize the weight vector to 1 ∀Wi=1,...,n .

(2) For s = 1, 2, . . . , S do:
(i) Resample the training data samples X according to their actual weights, obtaining the samples H.
(ii) Train a classifier Cs (H y ).
(iii) Classify the samples X using Cs .
(iv) Compute the probability of the error of classification taking into account the weights as follows:

1
s s
N
εs = Wξ, (1)
N i=1 i i

where

1, if Xi was wrongly classified in the step s,
ξis =
0,
(2)
otherwise.
(v) Compute ζs as
 
1 1 − εs
ζs = log . (3)
2 εs
(vi) If εs < 0.5set Ws+1
i = Wsi exp(ζs ξis ) for each training vector i, and then normalize the weights in a
N s+1
way that i=1 Wi = N.
Otherwise restart the algorithm.
(3) Finally the classifiers obtained are combined using a weighted majority voting using the coefficients ζs
that encode a measure of the error in each step. The final decision rule for each new test vector is


O(x) = ζs Ls > 0, (4)
s

where L is the label obtained in the classifier s, L ∈ (−1, 1).

Algorithm 1: General AdaBoost algorithm.

steps, and the final decision rule is a weighted combination (see Figure 1). To convert this ternary representation into a
of the intermediate classifiers. binary one, we have separated the filtered image into two
In this work, we have chosen as the boosting scheme a representations where we put to 1 the pixels where there is a
variant of the AdaBoost algorithm [25] (see Algorithm 1). ridge/valley and 0 otherwise. Both representations have been
In the literature we can find some variants of the original al- vectorized and concatenated (so at the end we have a binary
gorithm depending on the use of the weights in the learn- vector with double dimensionality).
ing process. One possible approach is to use a classifier that As we use a Bayesian classifier, for each target image Xi we
can accept weights as inputs, or simply multiply the train- decide that Xi is a face if p(Xi |CFaces ) > p(Xi |CNonFaces ) (see
ing vectors by their weights. Another approximation is to use the MAP rule [30]). To estimate the conditional probabilities
the weights to resample the training set in such a way that for the faces, we use
misclassified examples must have higher probability of being
D
chosen for the next step. We have used this scheme in our 
p X |CFaces = pdXd qd1−Xd , (5)
experiments.
d =1

2.1.1. Boosting using fixed features where Xd is dth pixel of the image, pd is the probability of
We propose a detection process that makes use of the boost- finding a 1 in the pixel d, and qd the probability of finding a
ing scheme of Algorithm 1 where the chosen weak classifier 0 in the pixel d (pd = 1 − qd ). This probabilities p and q can
is the naive Bayes classifier. In particular we have assumed a be estimated directly from the training samples by finding
Bernoulli distribution on the data [28]. In fact, this assump- the frequencies of the ones and zeros of the face vectors. In a
tion is justified because we have used binary data instead of similar way the conditional probabilities for the nonfaces are
the original images.
For each input image we have extracted a set of fixed fea- D
tures, using a detector of ridges and valleys (see [29] for more 
p X |CNonFaces = pdXd qd1−Xd , (6)
details) due to its robustness against changes in the illumina- d =1
tion. The final features have been obtained by thresholding
the response of the filters, assigning −1 to a pixel when it is where p and q are obtained as p and q but using the nonface
situated in a valley, 1 when it is on a ridge, and 0 otherwise instead of the face samples.
2064 EURASIP Journal on Applied Signal Processing

Figure 1: Example of ridges and valleys detection for a subset of face and nonface images.

Face detection requires high detection rates (higher than Usually we will use a number of bases R < D, to obtain an im-
90%) and very low false positive rates (lower than 0.0001%) portant dimensionality reduction, and a selection of the most
in order to be useful. In order to get this kind of false posi- useful features. In the case of face images it can be shown that
tive rates, Viola and Jones [21] proposed the use of boosted the nonnegativity constraints achieve bases which represent
classifiers in a cascade architecture where each classifier was specific parts of faces such as eyes, mouth, and so forth To
specialized in classifying a specific subset of nonfaces while find this set of bases and coefficients, the following update
keeping high detection rates. We have also followed this ap- rules should be iterated:
proach.

Qd Xid
Wi j ←− Wi j H jd ,
2.1.2. Boosting using adaptable features d
(WH)id

Usually the boosting algorithm uses a fixed set of features ex- Wi j

Wi j ←−  , (9)
tracted from the input data X. These features can be in the k Wk j
most simple case the pixel values of the image, or in other
Xid
H jd ←− H jd Wi j .
cases, the result of some kind of filtering (see [21]) or feature i
(WH) id
extraction, like the method proposed in the last section.
We also have considered a completely different approach. We should take into account that matrices W and H must be
The idea is to introduce in the boosting process the feature randomly initialized with positive values. Notice also the in-
extraction in such a way that at every fixed rate of boosting clusion of the diagonal matrix Q, which encodes the weights
steps, the features per each image are recomputed in a dy- of the samples. We use the same weights obtained at each step
namical way. To compute the new features, the weights of the of the AdaBoost algorithm to set the matrix Q at each step,
training vectors are used, so every time the features are more and focus the feature extraction to the most difficult exam-
focused in the more difficult examples. ples.
In our experiments, we have used a variant of the non- The advantage of this adaptive boosting approach is that
negative matrix factorization algorithm [31] to recompute at every boosting step it is more specialized in the wrong
the features and our classifier was a single-layer perceptron. classified examples, and the features that are being consid-
The modified algorithm uses a weighting matrix to focus the ered are learned, giving more importance to the most diffi-
algorithm in some specific samples (see [32] for more de- cult examples. This must lead the algorithm to a faster con-
tails). vergence, achieving the same or even better results than the
The basic NMF technique proposed by Lee and Seung classic boosting but in fewer steps.
[31] tries to find parts in the training objects, providing a
part-based set of bases. The key point of the NMF algorithm
is the use of the nonnegativity constraint applied to both the 3. FACE RECOGNITION
set of bases and coefficients that represent each vector in the In our scheme we will use a discriminant analysis to project
reduced space. This nonnegativity constraint provides always the data into a reduced space followed by a nearest-neighbor
sparse bases due to the fact that the weighted combination of classification. The dimensionality reduction achieved in the
the bases is always additive, so bases cannot have big regions feature extraction is essential in real-time classification prob-
active because once a big portion of an image is used as a lems, learning invariances in the samples, and yielding a
base, it cannot be removed (due to the constraint of additiv- compact representation that reduces the storage and com-
ity). putational needs. Prior to the feature extraction the proper
The general formulation for the NMF algorithm consists image normalization has been performed, in order to make
of given a set of N D-dimensional data points X (represent- the system more robust against changes in illumination and
ing positive-valued pixel images), and finding the nonnega- in geometric transformations.
tive bases W that satisfy
3.1. Image preprocessing


D
Xi j ≈ (W ∗ H)i j = Wid Hdj . (8) The normalization engine has two parts. One takes care
d =1 of illumination and a second one takes care of geometric
Feature Extraction for Face Classification 2065

(1) Given the matrix X containing data samples placed as N D-dimensional columns, the within-class scatter
SI matrix, and M maximum dimensions of discriminant space.
(2) Compute eigenvectors and eigenvalues for SI . Make Φ the matrix with the eigenvectors placed as
columns and Λ the diagonal matrix with only the nonzero eigenvalues in the diagonal. M I is the number
of nonzero eigenvalues.
(3) Whiten the data with respect to SI to obtain M I dimensional whitened data

Z = Λ−1/2 ΦT X. (7)
SE
(4) Compute on the whitened data.
(5) Compute eigenvectors and eigenvalues for SE and make Ψ the matrix with the eigenvectors placed as
columns and sorted by decreasing eigenvalue.
(6) Preserve only the first R = min{M I , M, rank(SE )} columns, Ψ = {ψ1 , . . . , ψR } (those corresponding to
the R largest eigenvalues).
(7) The resulting optimal transformation is W = ΨT Λ−1/2 ΦT and the projected data is, Y = WX = ΨT Z.

Algorithm 2: General algorithm for solving the discriminability optimization problem stated in (11).

normalization. Given a candidate face, illumination was nor- of sample vectors between different classes and within a class,
malized based on the local mean and variance.Geometric respectively. It has been shown (see [33, 34]) that the R × D
normalization was quite simplistic. Tilt was not taken into linear transform that satisfies
account. Gray value frequencies were obtained for different
unidimensional projections and this information was used 
W = arg Tmax tr WT SE W (11)
for obtaining the position of the eyes and face pose. Two W S W=I
I

peaks in the horizontal projection and one peak in the ver-

tical projection allow the localization of the pixels of the eyes optimizes the separability measure J. This problem has an
in the image. Yaw and normal face rotations were normal- analytical solution based on the eigenvectors of the scat-
ized taking advantage of the almost bilateral symmetry of the ter matrices. The algorithm presented in Algorithm 2 ob-
faces with respect to the eyes. tains this solution [34]. The most widely spread approach
Since this scheme results in a very poor approach, an for defining the within- and between-class scatter matrices is
additional threshold was introduced in order to decide the the one that makes use of only up to second-order statistics
goodness of the normalization. It was determined empiri- of the data. This was done in a classic paper by Fisher [17]
cally using the training samples and checking which is the and the technique is referred to as Fisher discriminant anal-
reasonable range of values where the central pixel of the eyes ysis (FLD). In FLD the within-class scatter matrix is usually
can be located. Actually, this threshold acts as an additional computed as a weighted sum of the class-conditional sample
layer of detection. Face candidates yielding values below this covariance matrices. If equiprobable priors are assumed for
threshold were disregarded. Otherwise, we store the position classes Ck , k = 1, . . . , K, then
of each eye in the original frame.
1

K
3.2. Face classification SI = Σk , (12)
K k=1
As we use NN as classification rule, a proper feature extrac-
tion algorithm is needed, and discriminant analysis can be
where Σk is the class-conditional covariance matrix, esti-
very useful for this task. In this section we will introduce the
mated from the sample set. The between-class scatter matrix
classic Fisher discriminant analysis to show later how its non-
is defined as
parametric extension can solve the main drawbacks of FLD:
Gaussian distribution assumption and reduced dimensional-
1

K
ity of the generated subspaces. We will also show our modifi-  T
SE = µ − µ0 µk − µ0 , (13)
cation of the classic NDA [19], which is expected to improve K k=1 k
the NN classification.
where µk is the class-conditional sample mean and µ0 is the
3.2.1. Fisher discriminant analysis unconditional (global) sample mean.
The goal of discriminant analysis is to find the features that Notice the rank of SE is K − 1, so the number of extracted
best separate the different classes. One of the most used cri- features is, at most, one less than the number of classes. Also
terions J to reach it is to maximize notice the parametric nature of the scatter matrix. The solu-
 tion provided by FLD is blind beyond second-order statistics.
J = tr SE SI , (10) So we cannot expect our method to accurately indicate which
features should be extracted to preserve any complex classifi-
where the matrices SE and SI , generally represent the scatter cation structure.
2066 EURASIP Journal on Applied Signal Processing

(a) (b)

Figure 2: First directions of NDA (solid line) and FLD (dashed line) projections, for two artificial datasets. Observe the results in (b) where
the FLD assumptions are not met.

3.2.2. Nonparametric discriminant analysis examine the relationship between NN and NDA. This results
in a modification of the within-class covariance matrix which
In [19] Fukunaga and Mantock present a nonparametric
we also introduce.
method for discriminant analysis in an attempt to overcome
Given a training sample x, the accuracy of the 1-NN rule
the limitations present in FLD. In nonparametric discrim-
can be directly computed by examining the ratio ∆E / ∆I .
inant analysis, the between-class scatter SE is of nonpara-
If this ratio is more than one, x will be correctly classified.
metric nature. This scatter matrix is generally full rank, thus
Given the M × R linear transform W, the projected distances
loosening the bound on extracted feature dimensionality.
Also, the nonparametric structure of this matrix inherently are defined as ∆E,I E,I
W = W∆ . Notice that this definition does
leads to extracted features that preserve relevant structures not exactly agree with the extra- and intraclass distances in
for classification. We briefly expose this technique, exten- projection space since, except for the orthonormal transfor-
sively detailed in [34]. mation case, we have no warranty on distance preservation.
In NDA, the between-class scatter matrix is obtained Equivalence of both definitions is asymptotically true. By the
from vectors locally pointing to another class. This is done as above remarks, it is expected that optimization of the follow-
follows. The extra-class nearest neighbor for a sample x ∈ Ck ing objective function should improve or, at least, not down-
is defined as xE = {x ∈ Ck / x − x ≤ z − x, ∀z ∈ Ck }. grade NN performance:
In the same fashion we can define the set of intraclass nearest 
 2
neighbors as xI = {x ∈ Lc / x − x ≤ z − x, ∀z ∈ Ck }. W = arg max E ∆EW  . (15)
From these neighbors, the extra-class differences are de- E{∆W 2 }=1
I

fined as ∆E = x − xE and the intraclass differences as ∆I =

x − xI . Notice that ∆E points locally to the nearest class (or This optimization problem can be interpreted as follows: find
classes) that does not contain the sample. The nonparametric the linear transform that maximizes the distance between
between-class scatter matrix is defined as (assuming uniform classes, preserving the expected distance among the members
priors) of a single class. Consider that

 2   
1
 E  E
N
E
S = ∆ ∆n
T
, (14) E ∆W  = E (W∆)T (W∆) = tr WT ∆∆T W , (16)
N n=1 n
where ∆ can be ∆I or ∆E . Substituting (16) in (17) we have
where ∆En is the extra-class difference for sample xn . that this last equation is a particular case of (11). Addition-
A parametric form is chosen for the within-class scat- ally, the formulas for the within- and between-class scatter
ter matrix SI , defined as in (12). Figure 2 illustrates the dif- matrices are directly extracted from this equation. In this
ferences between NDA and FLD in two artificial datasets, case, the between-class scatter matrix agrees with (14), but
one with Gaussian classes where results are similar, and one the within-class scatter matrix is now defined in a nonpara-
where FLD assumptions are not met. For the second case, metric fashion:
the bimodality of one of the classes displaces the class mean
introducing errors in the estimate of the parametric ver-
1
I IT
N
sion of SE . The nonparametric version is not affected by this Sw = ∆ ∆ . (17)
situation. We now make use of the introduced notation to N n=1 n n
Feature Extraction for Face Classification 2067

0.9

0.8

0.7
Classification accuracy

0.6

0.5

0.4
Figure 4: Examples of false negative and false positive images ob-
0.3
tained in the boosting scheme. As can be seen, the false positives are
0.2 images with the structure similar to a face (eyes, nose, and mouth).

0.1

0 instead of using adaptable features. Although the accuracies

0 10 20 30 40 50 60 70 80 90 100
obtained in the adaptive scheme are higher, the main goal
Subspace dimensionality
of this experiment was to design a real-time system, and the
NDA2 temporal performance of the adaptive scheme still was not
NDA enough to satisfy this real-time restriction.
LDA This real-time requisite has also justified the election of
PCA
a feature extraction algorithm (to reduce the amount of data
Figure 3: Recognition accuracy of different feature extraction algo- storage) in the verification step. Also our face detector is re-
rithms with different subspace dimensionalities. stricted to faces with small rotations (less than 10 degrees),
although the set up of the real application allows the capture
of enough frontal views of each person.
Given that we have an optimization problem of the form
given in (11) the algorithm presented in Algorithm 2 can 4.1. Face detection
also be applied to the optimization of our proposed objec-
4.1.1. Face detection using fixed features
tive function (15).
To compare the improvement of the modified NDA in The Bayesian classifier defined in Section 2 has been used in a
a face recognition problem, we have taken 10 samples (5 boosting scheme, where we have selected one feature at each
for training and 5 for testing) of each of the 115 subjects step. We have used 6500 training images, with 1500 faces ex-
from the AR Face Database [35]. We have compared differ- tracted from different public face databases (we have used the
ent feature extraction techniques for the same data set in the XM2VTS [37] and the first image of each subject from the
face recognition problem (using the 1-NN classifier). In the AR Face Database [35]) and 5000 nonface images. To test the
Figure 3, we show the classification accuracy as a function of performance of the algorithm, a huge set of 28 000 images
the number of features extracted using NDA, principal com- has been used (26 000 nonfaces and 2000 faces taken from
ponent analysis, Fisher discrimiant analysis, and the mod- the same databases). The global performance of the classi-
ified NDA (NDA2 in the figure). It can be seen how NDA fier in the first cascade level is 2.45% false rejection rate and
and the modified NDA outperform the other techniques, the only a 0.83% false positive rate. If we consider the full cas-
modified algorithm being slightly better in almost all the di- cade with 32 levels (as described in [38]), the final detection
mensions. Other comparative studies between NDA and the rate is close to 94%, while we obtain one false positive every
most used feature extraction techniques using different data 100 000 samples.
sets can be found in [28, 36], where it is shown that in low- In Figure 4, we show some examples of images wrongly
dimensional subspaces, NDA performs better than the other classified by the detector. Some twisted and blurry faces are
techniques. In high-dimensional subspaces in problems with confused, and also some natural images which present a
a large number of classes, both NDA and FLD have similar structure very similar to a face are also misclassified. Also
performance. Nevertheless, in this work we have chosen the we have tested the face detector using the CMU test database
modified NDA approach, given that it performs better in the [39] (with 483 labeled faces), obtaining 94.2% of detection
general case (low- and high-dimensional subspaces and in rate and just 82 false positives. Notice that these results are
low and large class problems). similar to the most common techniques used in the state of
the art, although our detector is more simple, which allows us
to use it in a real environment. A complete comparative study
4. EXPERIMENTS
can be found in [1], where it can be seen that the best tech-
We have designed a face verification application composed nique achieves 98% of detection rate, but also obtains a large
of two parts: the face detection in natural images, and the amount of false detections (12 758). Other techniques such
verification of the identity of the face. In the case of the face Fisher linear discriminant achieve just 74 false detections, al-
detector, we have opted for using a classic boosting scheme though the detection rate decreases (93.6%). Our purpose
2068 EURASIP Journal on Applied Signal Processing

100 100
99 99
98 98
97 97
96 96
Accuracy

Accuracy
95 95
94 94
93 93
92 92
91 91
90 90
0 50 100 150 200 250 300 0 50 100 150 200 250 300
Boosting steps Boosting steps

Using AdaBoost with fixed features Using AdaBoost with fixed features
Using AdaBoost with adaptable features Using AdaBoost with adaptable features

(a) (b)

Figure 5: Accuracies obtained in the face and nonface images as a function of the boosting steps. As can be seen, the adaptive feature learning
achieves better results.

achieves detection rates close to the other techniques, keep- As most face processing applications, ours consisted in three
ing a reasonable false detection rate and allowing a fast im- engines: detection, normalization, and recognition. Now we
plementation. will focus on the recognition stage; in Section 4.1 we have de-
scribed our detection scheme.
4.1.2. Face detection using adaptable features
Although we have discarded its use in the real-time applica- 4.2.1. Settings for the face detector
tion due to the computational constraints, we also show the Given a single frame of 576×768 pixels, the image was resized
results obtained with the adaptable boosting scheme to com- to 288 × 384 to avoid the effect of interlacing. The set of im-
pare the efficiency of both algorithms and highlight that there ages, candidates to be a face, is generated as follows: a set of
is an increase in the detection rates. In fact, the introduction sliding windows of 32 × 24 pixels has been generated for each
of the feature extraction into the boosting scheme is still an frame in such a way that all the possible subwindows from
interesting line of research and can be still optimized. the image are generated. The center of the sliding window
We have trained the boosting scheme using the same is considered every two pixels. The process is repeated at 4
1500 face images as above, and also the 5000 nonface images different scales, each time the image is rescaled by a factor of
extracted from random natural images. To test the perfor- 1.25. Four scales have been shown to be enough for our appli-
mance of the algorithm, the same 28 000 images have been cation, detecting faces up to 64×64 pixels. Detection was per-
used. We have achieved a 99.73% of global accuracy, with formed using the scheme proposed in Section 4.1.1 on each
only 1.07% of false rejection and 0.2% of false positive ratio. candidate image; 63 000 candidate subimages are generated
We have considered a correct detection that each subimage from each frame. Once the subimage is detected as a face, we
has at least a face covering 80% of its surface. In Figure 5, it take from the original image the square corresponding to the
can be seen how the use of the adaptable feature approach face, and we resize it to a 32 × 32 image (optimal size for face
can improve the results can be improved significantly as we recognition).
detect 1.4% more faces. In a similar way we are able to im- As a face image can appear in more than one sliding win-
prove the rejection of nonface images. On the other hand, dow or in multiple scales, a merging step has been added to
the computational needs of the adaptable version have also avoid multiple detections of the same face. We compute the
increased, and thus it takes 12 seconds to process a single im- overlap among sliding windows with a detected face in close
age in a Pentium 4 2.4 GHZ platform using Matlab 6.0 code. positions (overlapping more than 80% of their surface). We
keep the one that is closer to the mean of the overlapping
4.2. Face recognition in an uncontrolled environment windows (taking into account the center).
A second experiment was performed using the modified
NDA representation for real-time recognition in an uncon- 4.2.2. Feature extraction and face recognition
trolled environment. We now detail the design of this exper- Finally, for the recognition engine, we considered a scheme
iment, which attempts to simulate a surveillance situation. based on a linear projection using the NDA algorithm
Feature Extraction for Face Classification 2069

(a) (b)

Figure 6: Example of some faces used in the face recognition experiment, before and after normalization.

(a) (b)

Figure 7: Frames extracted from the human ID at distance application in the computer vision center.

followed by a nearest-neighbor classification. Recognition the gallery. The total number of faces for these 47 subjects
was performed on the 32 × 32 faces in the original frame. was 4176, approximately 88 faces per subject. Ten fold cross-
This is illustrated in Figure 6. In Figure 6a some detected validation on the faces was used to evaluate the performance
faces from a video sequence are shown. Figure 6b shows these of our classifier. For this experiment, classification accuracy
same faces after the normalization. was 96.83%. We also did a supervised cross-validation, so no
Given these images, we now detail the feature extraction samples from the same day were at the same time in train-
and classification stages. As a previous step, we have com- ing and test sets. Results were very similar, yielding a 95.9%
puted a principal component analysis projection matrix us- accuracy.
ing a huge face data set, and the data vectors have been pro- Recognition was also evaluated online, recording the
jected to a 224-dimensional subspace that preserves approx- recognition results and video to manually evaluate the online
imately the 97% of the variance, prior to learning the NDA classification accuracy. Recognition rate for approximately
representation. Then 128 NDA components were preserved. 2000 test images belonging to the 47 subjects in the gallery
Classification was performed using the 5 nearest neighbors’ was 92.2%. In this experiment, we also observed that the clas-
average voting. sifier would greatly benefit from temporal integration which,
All the parameters from this scheme (PCA and NDA at the moment, was not implemented. The frame rate of the
dimensionality, number of nearest neighbors, and classifier application with all three working engines and this gallery
combination policy) were set by cross-validating the training of the 47 subjects was approximately 15 fps. Also prototype
set. selection techniques applied to the NN classifier could be
applied to speed up the system. Figure 7 shows two frames
4.2.3. Experimental setup and results taken directly from the working application. A first frame
illustrates the environment in which the experiment took
A Sony EVI D-31 camera was installed looking at a staircase
place, and the second frame illustrates the recognizer at work.
in the Computer Vision Center. This camera was connected
to a VCR and four hours of recordings at peek hours were
gathered each day every other week, for a total lapse of 6
weeks. The face detector was applied to these videos and the 5. CONCLUSIONS
detected images were saved and manually labeled. From the
approximately 80 different people detected in all tapes, only In this paper, a real-time face recognition framework has
those 47 with more than 30 detected faces were included in been presented. Two different problems have been solved.
2070 EURASIP Journal on Applied Signal Processing

First the problem of the face detection in uncontrolled en- [9] E. Osuna, R. Freund, and F. Girosi, “Training support vector
vironments has been solved using a boosting scheme based machines: an application to face detection,” in Proc. IEEE Con-
on the naive Bayes classifier. As it has been shown, this ference on Computer Vision and Pattern Recognition (CVPR
’97), pp. 130–136, San Juan, Puerto Rico, USA, June 1997.
scheme achieves very low false positive ratios in the exper-
[10] W. Zhao, R. Chellappa, P. J. Phillips, and A. Rosenfeld, “Face
imental tests made with two public face databases. Then a recognition: a literature survey,” ACM Computing Surveys,
face classification scheme based on the NDA feature extrac- vol. 35, no. 4, pp. 399–458, 2003.
tion followed by a nearest-neighbor classification has been [11] M. S. Bartlett, M. H. Lades, and T. J. Sejnowski, “Independent
performed. The results show very good accuracies in the tests component representations for face recognition,” in Human
made in a real environment, and also we have achieved a very Vision and Electronic Imaging III, vol. 3299 of Proceedings of
good performance in terms of time, achieving rates close to SPIE, pp. 528–539, San Jose, Calif, USA, January 1998.
15 frames per second. NDA computational cost is similar to [12] Z. Li and X. Tang, “Bayesian face recognition using support
PCA and other linear projection techniques, while the accu- vector machine and face clustering,” in Proc. IEEE Computer
Society Conference on Computer Vision and Pattern Recognition
racies obtained have been higher in all the tests. (CVPR ’04), vol. 2, pp. 374–380, Washington, DC, USA, June–
We also show a new approach to boosting. We have in- July 2004.
troduced the feature learning into the boosting scheme by [13] L. Wiskott, J.-M. Fellous, N. Kruger, and C. von der Malsburg,
adapting the features used to the most difficult examples. The “Face recognition by elastic bunch graph matching,” IEEE
algorithm achieves excellent accuracies, but further work is Trans. Pattern Anal. Machine Intell., vol. 19, no. 7, pp. 775–
needed to speed up the algorithm in order to be used in a 779, 1997.
real-time environment. [14] A. Pentland, B. Moghaddam, and T. Starner, “View-based and
Another open consideration in the boosting scheme is modular eigenspaces for face recognition,” in Proc. IEEE Con-
ference on Computer Vision and Pattern Recognition (CVPR
the fact that we can take benefit from asymmetric classifiers ’94), pp. 84–91, Seattle, Wash, USA, June 1994.
in the boosting process. One desirable property in face de- [15] A. M. Martı́nez, “Recognizing imprecisely localized, partially
tection is not to lose faces in the detection process and this occluded, and expression variant faces from a single sample
property should be introduced in the internal boosting clas- per class,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 24,
sifier. no. 6, pp. 748–763, 2002.
[16] M. Turk and A. Pentland, “Eigenfaces for recognition,” Journal
of Cognitive Neuroscience, vol. 3, no. 1, pp. 71–86, 1991.
ACKNOWLEDGMENT [17] R. Fisher, “The use of multiple measurements in taxonomic
problems,” Annals of Eugenics, vol. 7, pp. 179–188, 1936.
This work is supported by MCYT Grant TIC2003-00654 and [18] P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman, “Eigen-
FP2000-4960, Ministerio de Ciencia y Tecnologia, Spain. faces vs. Fisherfaces: recognition using class specific linear
projection,” IEEE Trans. Pattern Anal. Machine Intell., vol. 19,
no. 7, pp. 711–720, 1997.
REFERENCES
[19] K. Fukunaga and J. Mantock, “Nonparametric discriminant
[1] M. Yang, D. J. Kriegman, and N. Ahuja, “Detecting faces in analysis,” IEEE Trans. Pattern Anal. Machine Intell., vol. 5,
images: a survey,” IEEE Trans. Pattern Anal. Machine Intell., no. 6, pp. 671–678, 1983.
vol. 24, no. 1, pp. 34–58, 2002. [20] P. J. Phillips, P. Grother, R. J. Micheals, D. M. Blackburn, E.
[2] C. Kotropoulos and I. Pitas, “Rule-based face detection in Tabassi, and J. M. Bone, “Frvt 2002: Evaluation report,” Tech.
frontal views,” in Proc. IEEE Int. Conf. Acoustics, Speech, Sig- Rep., March 2003, online, available: http://www.frvt.org/
nal Processing (ICASSP ’97), vol. 4, pp. 2537–2540, Munich, FRVT2002/documents.htm.
Germany, April 1997. [21] P. Viola and M. Jones, “Robust real-time object detection,” In-
[3] T. K. Leung, M. C. Burl, and P. Perona, “Finding faces in clut- ternational Journal of Computer Vision, 2002.
tered scenes using random labeled graph matching,” in Proc. [22] R. Lienhart, A. Kuranov, and V. Pisarevsky, “Empirical anal-
5th International Conference on Computer Vision (ICCV ’95), ysis of detection of boosted classifiers for rapid object detec-
pp. 637–644, Boston, Mass, USA, June 1995. tion,” Tech. Rep., Microsoft Research Lab, Intel Labs, Cam-
[4] A. Lanitis, C. Taylor, and T. Cootes, “An Automatic face bridge, UK, 2002.
identification system using flexible appearance models,” [23] H. S. Z. Li, Z. Q. Zhang, and H. J. Zhang, “Floatboost learn-
Image and Vision Computing, vol. 13, no. 5, pp. 393– ing for classification,” in NIPS 15, S. Thrun, S. Becker, and K.
401, 1995, online, available: http://citeseer.ist.psu.edu/article/. Obermayer, Eds., Vancouver, Canada, December 2002.
lanitis95automatic.html [24] P. Viola and M. Jones, “Fast and robust classification using
[5] M. Kirby and L. Sirovich, “Application of the Karhunen-Loeve asymmetric adaboost and a detector cascade,” in Advances in
procedure for the characterization of human faces,” IEEE Neural Information Processing Systems 14, T. G. Dietterich, S.
Trans. Pattern Anal. Machine Intell., vol. 12, no. 1, pp. 103– Becker, and Z. Ghahramani, Eds., pp. 1311–1318, MIT Press,
108, 1990. Cambridge, Mass, USA, 2002.
[6] B. Moghaddam and A. Pentland, “Probabilistic visual learning [25] Y. Freund and R. E. Schapire, “Experiments with a new boost-
for object representation,” IEEE Trans. Pattern Anal. Machine ing algorithm,” in Proc. 13th International Conference on Ma-
Intell., vol. 19, no. 7, pp. 696–710, 1997. chine Learning (ICML ’96), pp. 148–156, Bari, Italy, July 1996.
[7] K.-K. Sung and T. Poggio, “Example-based learning for view- [26] M. Skurichina and R. P. W. Duin, “Bagging for linear classi-
based human face detection,” IEEE Trans. Pattern Anal. Ma- fiers,” Tech. Rep., Delft University of Technology, Delft, The
chine Intell., vol. 20, no. 1, pp. 39–51, 1998. Netherlands, 2002.
[8] F. Samaria, Face recognition using hidden Markov models, [27] M. Skurichina and R. P. W. Duin, “Bagging, boosting and the
Ph.D. dissertation, University of Cambridge, Cambridge, UK, random subspace method for linear classifiers,” Pattern Anal-
1994. ysis and Applications, vol. 5, no. 2, pp. 121–135, 2002.
Feature Extraction for Face Classification 2071

[28] M. Bressan, Statistical independence for classification of high di- Jordi Vitrià received the Ph.D. degree from
mensional data, Ph.D. dissertation, Universitat Autonoma de the Autonomous University of Barcelona
Barcelona, Barcelona, Spain, 2003. (UAB), Barcelona, Spain, for his work
[29] A. M. Lopez, F. Lumbreras, J. Serrat, and J. J. Villanueva, on mathematical morphology in 1990. He
“Evaluation of methods for ridge and valley detection,” IEEE joined the Computer Science Department,
Trans. Pattern Anal. Machine Intell., vol. 21, no. 4, pp. 327– UAB, where he became an Associate Profes-
335, 1999. sor in 1991. His research interests include
[30] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, machine learning, pattern recognition, and
John Wiley & Sons, New York, NY, USA, 2nd edition, 2001. visual object recognition. He is the author
[31] D. D. Lee and H. S. Seung, “Learning the parts of objects by of more than 40 scientific publications and
non-negative matrix factorization,” Nature, vol. 401, no. 6755,
one book.
pp. 788–791, 1999.
[32] D. Guillamet, M. Bressan, and J. Vitrià, “A weighted non-
negative matrix factorization for local representations,” in
Proc. IEEE Computer Society Conference on Computer Vision
and Pattern Recognition (CVPR ’01), vol. 1, pp. 942–947,
Kauai, Hawaii, USA, December 2001.
[33] P. Devijver and J. Kittler, Pattern Recognition: A Statistical Ap-
proach, Prentice Hall, London, UK, 1982.
[34] K. Fukunaga, Introduction to Statistical Pattern Recognition,
Academic Press, Boston, Mass, USA, 2nd edition, 1990.
[35] A. M. Martı́nez and R. Benavente, “The AR face database,”
Tech. Rep. 24, Computer Vision Center, Barcelona, Spain,
1998.
[36] M. Bressan and J. Vitrià, “Nonparametric discriminant anal-
ysis and nearest neighbor classification,” Pattern Recognition
Letters, vol. 24, no. 15, pp. 2743–2749, 2003.
[37] J. Matas, M. Hamouz, K. Jonsson, et al., “Comparison of face
verification results on the XM2VTS database,” in Proc. Inter-
national Conference on Pattern Recognition (ICPR ’00), vol. 4,
pp. 858–863, Barcelona, Spain, September 2000.
[38] P. Viola and M. Jones, “Rapid object detection using a boosted
cascade of simple features,” in Proc. IEEE Computer Soci-
ety Conference on Computer Vision and Pattern Recognition
(CVPR ’01), vol. 1, pp. 1063–6919, Kauai, Hawaii, USA, De-
cember 2001.
[39] H. A. Rowley, S. Baluja, and T. Kanade, “Neural network-
based face detection,” IEEE Trans. Pattern Anal. Machine In-
tell., vol. 20, no. 1, pp. 23–38, 1998.

David Masip is a Ph.D. student in com-

puter science at the Computer Vision Cen-
ter (CVC), the Autonomous University of
Barcelona (UAB). His research interests
concern the development of algorithms of
feature extraction for pattern recognition
and face classification. He has a degree in
computer science and an M.S. degree in
computer vision from UAB.

Marco Bressan is a permanent researcher in

XRCE’s Image Processing Research Group
(IPC). He holds a B.A. degree in applied
mathematics from the University of Buenos
Aires (UBA), an M.S. degree in computer
vision from the Computer Vision Centre
in Barcelona (CVC), and a Ph.D. degree
in computer science and artificial intelli-
gence from the Autonomous University of
Barcelona (UAB). As a researcher, he has au-
thored more than 15 refereed publications. His main research in-
terests are in statistical learning and classification; image and video
semantic scene understanding; and object detection, recognition,
and enhancement, particularly when dealing with uncontrolled en-
vironments.
 Photographȱ©ȱTurismeȱdeȱBarcelonaȱ/ȱJ.ȱTrullàs

Preliminaryȱcallȱforȱpapers OrganizingȱCommittee
HonoraryȱChair
The 2011 European Signal Processing Conference (EUSIPCOȬ2011) is the MiguelȱA.ȱLagunasȱ(CTTC)
nineteenth in a series of conferences promoted by the European Association for GeneralȱChair
Signal Processing (EURASIP, www.eurasip.org). This year edition will take place AnaȱI.ȱPérezȬNeiraȱ(UPC)
in Barcelona, capital city of Catalonia (Spain), and will be jointly organized by the GeneralȱViceȬChair
Centre Tecnològic de Telecomunicacions de Catalunya (CTTC) and the CarlesȱAntónȬHaroȱ(CTTC)
Universitat Politècnica de Catalunya (UPC). TechnicalȱProgramȱChair
XavierȱMestreȱ(CTTC)
EUSIPCOȬ2011 will focus on key aspects of signal processing theory and
TechnicalȱProgramȱCo
Technical Program CoȬChairs
Chairs
applications
li ti as listed
li t d below.
b l A
Acceptance
t off submissions
b i i will
ill be
b based
b d on quality,
lit JavierȱHernandoȱ(UPC)
relevance and originality. Accepted papers will be published in the EUSIPCO MontserratȱPardàsȱ(UPC)
proceedings and presented during the conference. Paper submissions, proposals PlenaryȱTalks
for tutorials and proposals for special sessions are invited in, but not limited to, FerranȱMarquésȱ(UPC)
the following areas of interest. YoninaȱEldarȱ(Technion)
SpecialȱSessions
IgnacioȱSantamaríaȱ(Unversidadȱ
Areas of Interest deȱCantabria)
MatsȱBengtssonȱ(KTH)
• Audio and electroȬacoustics.
• Design, implementation, and applications of signal processing systems. Finances
MontserratȱNájarȱ(UPC)
Montserrat Nájar (UPC)
• Multimedia
l d signall processing andd coding.
d
Tutorials
• Image and multidimensional signal processing. DanielȱP.ȱPalomarȱ
• Signal detection and estimation. (HongȱKongȱUST)
• Sensor array and multiȬchannel signal processing. BeatriceȱPesquetȬPopescuȱ(ENST)
• Sensor fusion in networked systems. Publicityȱ
• Signal processing for communications. StephanȱPfletschingerȱ(CTTC)
MònicaȱNavarroȱ(CTTC)
• Medical imaging and image analysis.
Publications
• NonȬstationary, nonȬlinear and nonȬGaussian signal processing. AntonioȱPascualȱ(UPC)
CarlesȱFernándezȱ(CTTC)
Submissions IIndustrialȱLiaisonȱ&ȱExhibits
d i l Li i & E hibi
AngelikiȱAlexiouȱȱ
Procedures to submit a paper and proposals for special sessions and tutorials will (UniversityȱofȱPiraeus)
be detailed at www.eusipco2011.org. Submitted papers must be cameraȬready, no AlbertȱSitjàȱ(CTTC)
more than 5 pages long, and conforming to the standard specified on the InternationalȱLiaison
EUSIPCO 2011 web site. First authors who are registered students can participate JuȱLiuȱ(ShandongȱUniversityȬChina)
in the best student paper competition. JinhongȱYuanȱ(UNSWȬAustralia)
TamasȱSziranyiȱ(SZTAKIȱȬHungary)
RichȱSternȱ(CMUȬUSA)
ImportantȱDeadlines: RicardoȱL.ȱdeȱQueirozȱȱ(UNBȬBrazil)

P
Proposalsȱforȱspecialȱsessionsȱ
l f i l i 15 D 2010
15ȱDecȱ2010
Proposalsȱforȱtutorials 18ȱFeb 2011
Electronicȱsubmissionȱofȱfullȱpapers 21ȱFeb 2011
Notificationȱofȱacceptance 23ȱMay 2011
SubmissionȱofȱcameraȬreadyȱpapers 6ȱJun 2011

Webpage:ȱwww.eusipco2011.org

View publication stats