Articulo 4
Articulo 4
Home
JP Pienaar1 , RM Fisher1 2 and GP Hancke2
1 Department of Electrical, Electronic and Computer Engineering,
University of Pretoria, Pretoria,
2 Department of Computer Science
Abstract—Automation systems are gaining popularity around a system to impirically trust the source. Current designs of
the world. The use of these powerful technologies for home secu- systems don’t provide for a trust relationship to be created
rity has been proposed and some systems have been developed. between the data (or instructions) that the user can submit
Other implementations see the user taking a central role in
providing and receiving updates to the system. We propose a and the system that will respond to this data/commands. A
system making use of an Android based smartphone as the user single PIN is no longer consider to be secure enough and a
control point. Our Android application allows for dual factor range of authentication methods have been proposed [4]. The
(facial and secret pin) based authentication in order to protect system that we will propose will allow for a user to submit
the privacy of the user. The system successfully implements facial two methods of authentication in order to provide for a more
recognition on the limited resources of a smartphone by making
use of the Eigenfaces algorithm. The system we created was secure implementation.
designed for home automation but makes use of technologies Inside the application the issue of privacy has also become
that allow it to be applied within any environment. This opens a primary concern [5] [6]. The risk of external access to
the possibility for more research into dual factor authentication the system may discourage many users from making use of
and the architecture of our system provides a blue print for the a automation system. Automation is an area that needs to
implementation of home based automation systems. This system
with minimal modifications can be applied within an industrial careful consider security, especially if application involves
application. user data [7]. One technology that we believe could play a
Index Terms—Authentication, Facial Recognition, Biometrics, pivotal role in the creation of a home/industry automation
Internet-of-Things, User Centric Automation system is the smartphone. The powerful operating system
that has has been created and the advanced technology that
I. I NTRODUCTION is included will potentially mean that the smartphone device
Automation is rapidly becoming a mainstream part of the could act as the central technology that enables home or small
world in which we all live. People have begun to see the industrial automation systems [8]. These devices have also
major advantages that this technology can bring. Houses and rapidly become the primary computing device for many people
industrial buildings are becoming more and more focused [9]. The huge popularity of smartphone devices and their
towards being smarter than ever before. With the advances in relative capabilities could make them the central technology
the new technologies that are available and the new capabilities for a small automation system. A well defined interoperable
that have been created the automation of homes, buildings and automation system could be created to work independent of
working environments has become a reality. The expansion any specific details relating to the smartphone [10] [11].
to allowing the implementation to be able to react to certain Our proposed system allows for the simple and effective
situations automatically or with some user input is something use of an Android based smartphone to control a home
that could potentially have a huge effect on many peoples lives. in a home-based automation situation. We created both the
People have gained an interest in making use of technology Android application as well as the embedded system within
in order to make their lives easier. a home environment but designed the system to be capable
The main aim is to fast track the adoption of sensor of deploying within an industrial environment. The system
technology. Making use of a mobile phone based device was tested allowing control over home electrical devices
will hopefully allow for a rapid adoption rate, due to the (geysers, lights etc) in order to mimic a common embedded
popularity of these devices [1]. With this adoption more and environment. The Android application provides for secure
more people will become involved in the collection of sensor access by requiring two factor authentication. The use of
based data [2]. This data can then be used to allow for the biometric security features (facial recognition) as well as a
direct management of user centric systems, such as smart user defined secret, in the form of a five digit pin code, were
homes or within industrial applications [3]. This use of user used to control access to the application. The user must at
center/created sensor data is prejudiced by the inability of all times remain the primary focus of such systems and the
l-)))
security of the system needs to be highly well thought out
[12]. The system was designed to be as interoperable with
as many technologies as possible [13]. In theory any system
could be included with the current design allowing for a robust
and capable implementation that focuses on ensuring easy use
for the user.
1) Facial Recognition: The first authentication method will zero mean [16]. Using these normalized images distances on
be facial recognition based. The facial recognition has been the images are measure and compared to that of the input
implemented through the use of an Eigenface algorithm. The image. Although this algorithm is the simplest to implement
algorithm used to create the facial recognition system will be there are a number of drawbacks, the biggest of which is that it
discussed in the next section. Performing facial recognition is computationally expensive to match the image to its closest
on board a Android based smartphone, has been a possibility match. Another drawback is that this algorithm is that it does
for a number of Android Versions. Our system does not make not handle lighting variation well; it would need images of
use of the included API and has been created from known most of the lighting variations to work reliably. The correlation
approaches/algorithms and have not made any use of existing algorithm also requires large amounts of storage for all the
libraries. The Android application will be secured making use training images under the varying conditions [17].
of the facial recognition system designed. This is one piece The Fisherfaces algorithm relies on the principle of linear
of the two factor authentication that will be implemented in discriminant analysis (LDA) which is used to find a linear
order to secure the application. combination of features which can represent multiple classes
2) Pin Verification System: As a second level of authentica- of data. This works similar to the Eigenfaces method by
tion, a 5-digit user specified pin was used. Implementing this computing a covariance matrix of the training images. This al-
second method allows two different methods of identification gorithm compares between-class scatter as well as within-class
to be used, namely; who you are for the facial recognition scatter where Eigenfaces mostly compares only the between-
implementation and what you know for the pin verification class scatter [18] [19]. The within-class scatter is mostly due to
system. A 5-digit pin is used as with 5 digits, 2002 unique the lighting variation in the images. This results in Fisherfaces
passcodes can be selected. This number is calculated as shown performing much better under lighting variations as well as
below, where n is the number of different numbers to choose expression variation.
and r is the number of digits that can be selected. In the chosen Fisherfaces also functions as a dimensionality reduction
system 12345 and 54321 are considered to be the same, the method which reduces the amount of data needed to store the
order of the numbers is not considered. training images. However Fisherfaces is more computationally
(n + r − 1)! expensive than Eigenfaces, as more comparisons need to be
(1)
r!(n − 1)! computed. This makes the algorithm unfavourable for a low
Through the application of both of the authentication systems resource situation as is the case with many mobile devices.
to the application, a higher level of accuracy will be achieved. The use of a more computationally friendly algorithm will be
This will help with one of the key features of our applica- greatly favoured as the user is not going to be expected to wait
tion, which is to allow for the technology to be completely long periods of time for the facial recognition to be completed.
hidden from the user and still being able to provide the best C. Eigenfaces
experience. The application must ensure that the user has the The Eigenface algorithm makes use of principle component
easiest use of the entire system as possible. The technology analysis (PCA) which is a statistical procedure, it is used to
must be able to fall to the background. This ability will allow replace original values with a smaller set of values comprised
for the user to completely experience the capabilities of the of linear combinations of the original variables(the principal
technology and not have to worry about the intricate details components). By doing this the dimensionality of data can
as to how the system achieves its goals. be greatly reduced [20]. This is one large advantage of the
III. B IOMETRIC S ECURITY - FACIAL R ECOGNITION Eigenface algorithm, especially in mobile applications where
Facial recognition is an ever-growing field of study in the available resources of the mobile device are limited. The
security systems. Facial recognition uses a specific algorithm smaller data set is used to represent the variability of the origi-
to identify a person using his face through the use of an nal data with the use of less data. For PCA to be implemented
image or video feed. Various algorithms and techniques can the data needs to be similar in size. To accomplish this some
be considered in facial recognition systems, some of the better pre-processing in the form of image normalization will need
known and computationally powerful algorithms that will be to be implemented.
discussed here include Eigenfaces, Fisherfaces and correlation. The images are captured using the front facing camera
A. Options Available of a Samsung GT-I9300 smart phone. Implemented to limit
A number of facial recognition algorithms exist [14]. Each the effects of image orientation and background noise in the
of the algorithms have their own strengths and weaknesses images. These images are resized to a resolution of 128 × 128
[15]. The algorithms that we considered in the implementation as well as grayscaled, this forms the preprocessing step of the
of the authentication system were: Correlation, Fisherfaces and recognition process. The accuracy of the facial recognition
Eigenfaces. implementation is affected by the number of images in the
B. Correlation and Fisherfaces training set, as well as the threshold value used to determine
recognition. IV. S YSTEM O PERATION
The technique of applying correlation is possibly the sim-
plest of the recognition algorithms. This algorithm works by The facial recognition implemented on our Android based
normalizing the training images to have unit variance and a smartphone was the primary research concern for the system.
Once the embedded system was created we attempted to is calculated using the number of images that do not get
provide added security by using two factor authentication for recognized. During the course of this test the threshold value
access to the controlling application. This was in order to assist used in the classifier is kept constant. The number of images
in ensuring that the user feels safe and secure knowing that used in the training set ranges from 25 to 50 images with
only they have access to the private details an application of a change of 5 images each time. The total accuracy of the
this nature can provide. algorithm was then calculated from these values.
A. Facial Recognition Operation The next test was completed to calculate the most appropri-
The Eigenfaces algorithm requires the use of assorted ate option for the threshold value that is used. The number
training images. These images should be captured with similar of training images is kept at fifty for the duration of this
conditions and lighting. The images in the training set will be test. The threshold value is calculated as a percentage of the
represented as vectors of the various training faces. The vectors maximum weight difference between the testing image and the
contain the pixel intensities of the various images [21]. An Eigenfaces. The threshold value ranges between 45% and 1%
image with a resolution of N × N will be represented as an with a change of 1% between each test.
image vector with a size of N 2 × 1 The final test that was completed was to measure the
In order to eliminate common data between the images, performance of the algorithm against the number of training
and improve the distinguishability, the mean data is subtracted, images that were used. Performance was measured directly
providing for a set of more distinguishable photos. Subtraction by using execution time. The number of training images was
of the mean image results in a covariance matrix [22]: compared against the total execution time for the algorithm.
B. Other Component Tests
C = AAT (2)
The other components of the system were also tested. All of
where A is the matrix of the different image vectors after them successfully responded 100% of the time. As an example
subtraction of the mean. This matrix then has the size of
the pin authentication component achieved a 100% accuracy
N 2 × N 2 which, on a mobile device, is computationally
for the decryption of the stored pin and comparison to the
infeasible. The concept of PCA allows us to reduce the
entered pin only allowing access when they both matched.
dimensions of the covariance matrix by calculating C. By
Multiple tests were completed on the other devices; all of
making use of PCA the dimensions of the covariance matrix
which achieved perfect results.
are further reduced which results in a covariance matrix of
size m × M , with M being the original training set size. C. Results
Making use of this the eigenvectors can then be calculated 1) Number of Training Images Test: The result show that
and be represented as vi , where i ranges from 1 to M [23]. It with an increasing number of images in the training set,
can be seen from some matrix properties that by arranging the overall accuracy of the facial recognition algorithm is
the corresponding eigenvalues of the covariance matrix in increased. This algorithm is run on a mobile device that
descending order and only using some of the higher values, the has limited resources, so there has to be a trade-off between
more useful Eigenfaces can be used for recognition [24] [25]. the accuracy of the algorithm and the execution time of the
This in turn allows for a smaller data set, increasing efficiency. recognition process. An accuracy of 80% could be achieved
Images to be recognized can be represented by a combi- using 45 images in the training set as seen in Table I. The
nation of the various Eigenfaces. The mean of the testing set decision on the number of images to use for training purposes
is subtracted from the image to be recognized and multiplied is not an exact science and compromises need to be made
with the eigenvector weights that each Eigenface contributes between execution time and accuracy. It can be seen from the
to the image. results that more images does not always mean a higher degree
wk = uTk φk (3) of accuracy but generally speaking the more images used allow
where uT is the transposed matrix of the eigenvectors and φk for a greater degree of accuracy from the classifier. In order
is the data of an image after the mean was subtracted. These to circumvent this a more powerful device (phone) could be
weights contribute to an error value, which is compared to a used, this could allow for a higher accuracy and not impact
threshold to determine recognition [24]. the user as much. We should also consider the effect this may
V. T ESTS have on users with older models as they should not be directly
A number of tests were completed in order to determine the prevented from using our system, maybe a solution is to let
capabilities of the facial recognition system. the user decide which to favour, speed or accuracy.
2) Threshold Value: It can be seen that in choosing a
A. Facial Recognition Experimental Approach threshold value the best accuracy was achieved when using
The first set of tests were used to determine the most 37% of the maximum error rate as the threshold as seen in
appropriate choice for the number of images to be used table II. By increasing this threshold more images that were
when training the image recognition classifier. Two key mea- not trained on would be seen as recognized and decreasing this
surements were taken: the fail match rate (FMR) and the threshold will cause some correct images to not be recognized.
fail non match rate (FNMR). The FMR is calculated using As can be seen from the results the accuracy of the system
the total number of images that get recognized. The FNMR begins and reaches a peak at 37% and then begins the process
Number of Images FMR (%) FNMR (%) Accuracy (%)
25 4 50 46
store the encryption key for encrypting the user pin. The key
30 14 41 45 used for the encryption should not be hard coded into the
35 20 40 40 application. This leads to the common concern area where
40 8 34 58 the algorithm and methods used to secure the system are
45 6 14 80
50 6 16 78 secure but the implementation is not. In future iterations of
TABLE I the Android operating system the requirement of encrypting
TABLE INDICATING THE ACCURACY FOR SPECIFIED NUMBER OF the locally stored key will not be of as high an importance as
TRAINING IMAGES
Google has decided to encrypt by default all storage include on
Threshold Value (%) FMR (%) FNMR (%) Accuracy (%) the mobile device, although this will not prevent an attack by
44 34 0 66 a malicious application. In the meantime a more appropriate
43 32 4 66 manner may be to use a secure certificate exchange with the
42 32 10 58
41 30 10 60 web server, already included in the system, in order to retrieve
40 22 12 66 the required key which is stored server side. By storing the
39 20 12 68 key securely on the server it is inherently more secure, due
38 20 12 68
to the fact that Android provides no fine grained permission
37 6 14 80
36 6 20 74 management for external files [26]. Facial recognition was
35 6 22 72 chosen as the biometric access due to the relative ease of which
34 6 26 68 other biometric systems (fingerprints) have been broken into
33 2 34 64
TABLE II and shown to be ineffective.
TABLE INDICATING THE ACCURACY FOR SPECIFIED THRESHOLD VALUE A number of challenges are inherent in the current design
of the system. Insufficient lighting or incorrect use of the
orientation overlay of the system could negatively affect the
of decreasing again. This is the best fit that we could achieve operation of the system. The system is limited by the available
in order to have a high degree of accuracy for our system. It resources of the device on which it is executed. If the battery
should be noted that the table has been shortened in order to power source of either the wireless transmitter or receiver
limit space wastage. The testing involved continued until 1% is discharged. Both modules would need to be restarted for
was reached and the highest values are shown in the table. connection between them to be re-established. These problems
3) Performance Test: The test that was run to compare the need to be examined in future iterations of the system.
execution time, was done to find a good balance between The two way communication between the Android applica-
accuracy and the time it takes to complete the recognition tion and the web server could be used to inform the user of an
process. The execution time for larger numbers of training alarm triggering. The web server can also implement image
images however is quite large and would be infeasible in the processing techniques and automatically record and forward a
system. The results for this test can be seen in figure 2. These video to the user, which will come up as a simple notification,
results show that the more training images used have a direct when movement is detected. A similar technique of facial
effect on the execution performance of the system. As the recognition but applied on the web server could automatically
main aim of the system is useability a careful decision must analyze friend or ’foe’ scenarios when applied within the
be made as to what number of images to use. The user does home. The level of security application that can be applied
not want to wait for an extended length of time in order to with the simple automation of a home using an Android mobile
access the application. and a connected home is nearly boundless. Switching lights
D. General Comments on and off, opening and closing curtains and turning music on
and off at random times during the day, or night, can simulate
By storing the pin in an encrypted manner the security of the presence of the home owner and thereby deter criminals
the system is improved. One concern is the method used to from assuming a easy target.
With the increasing number of wireless sensor networks
Graph showing Execution Time against Training Images Number being deployed around the world the problem of user authen-
25
tication and security is at the forefront of many implementers
20 minds. Many approaches have been proposed to the problem
Execution Time (s)
Another advantage that our system can provide is the [5] M. Jung, G. Kienesberger, W. Granzer, M. Unger, and W. Kastner,
separation of the two sources of the authentication data. A “Privacy enabled web service access control using saml and xacml
for home automation gateways,” in Internet Technology and Secured
separate camera from the one provided on a mobile phone Transactions (ICITST), Conference for, 2011, pp. 584–591.
can be used. This photo can then be processed and an alert [6] M. Cheminod, L. Durante, and A. Valenzano, “Review of security issues
on a mobile device can be used to provide the pin number in industrial networks,” IEEE Transactions onIndustrial Informatics,
vol. 9, no. 1, pp. 277–293, Feb 2013.
on the phone for the second authentication. As an example [7] G. Hancke, K. Markantonakis, and K. Mayes, “Security challenges for
this method will be perfect for any reconfiguration options user-oriented rfid applications within the” internet of things”,” Journal
performed on devices with cameras. The device takes my of Internet Technology, vol. 11, no. 3, pp. 307–313, 2010.
[8] C. Opperman and G. Hancke, “A generic nfc enabled measurement
picture and submits it for authentication, while it requests the system for remote monitoring and control of clientside equipment,” in
second authentication method. Once both authentications are Workshop on Near Field Communication (NFC), Feb 2011, pp. 44–49.
successful the device accepts the reconfiguration settings. [9] W.-C. Hu, Y. Zuo, L. Chen, and H.-J. Yang, “Contemporary issues
in handheld computing research,” International Journal of Handheld
Computing Research (IJHCR), vol. 1, no. 1, pp. 1–23, 2010.
VI. C ONCLUSION [10] H. Falaki, R. Mahajan, S. Kandula, D. Lymberopoulos, R. Govindan,
and D. Estrin, “Diversity in smartphone usage,” in Proceedings of the 8th
international conference on Mobile systems, applications, and services.
The operation of the system is succesful and due to the ACM, 2010, pp. 179–194.
design is both robust and interoperable with a range of [11] Y. Park and J. V. Chen, “Acceptance and adoption of the innovative
applications within an automated home. The main aim of im- use of smartphone,” Industrial Management & Data Systems, vol. 107,
no. 9, pp. 1349–1365, 2007.
plementing reliable facial recognition to be used as a method [12] R. Fisher, L. Ledwaba, G. Hancke, and C. Kruger, “Open hardware: A
of providing biometric security to the control application of role to play in wireless sensor networks?” Sensors, vol. 15, no. 3, pp.
a smart home has successfully been achieved. This makes 6818–6844, 2015.
[13] C. Kruger and G. Hancke, “Benchmarking internet of things devices,”
the use of the smart application more enjoyable for the end in Industrial Informatics (INDIN), 2014 12th IEEE International Con-
user. Additionally, the implementation of a pin entry system ference on. IEEE, 2014, pp. 611–616.
allows for the secure access in conjunction with the biometric [14] J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma, “Robust face
recognition via sparse representation,” Pattern Analysis and Machine
system and ensures secure operation of the application. From Intelligence, IEEE Transactions, vol. 31, no. 2, pp. 210–227, 2009.
this implementation we have shown that it is possible to [15] M. Sharkas and M. Elenien, “Eigenfaces vs. fisherfaces vs. ica for face
make use of biometric features in order to protect wireless recognition; a comparative study,” in Signal Processing, 2008. ICSP
2008. 9th International Conference on, Oct 2008, pp. 914–919.
sensor network system. Our system is also interoperable and [16] X. Huang, G. Zhao, M. Pietikainen, and W. Zheng, “Robust facial
could be deployed across a number of different application expression recognition using revised canonical correlation,” in Pattern
examples. The industrial application could allow employees Recognition (ICPR), Conference on, Aug 2014, pp. 1734–1739.
[17] W. Zheng, X. Zhou, C. Zou, and L. Zhao, “Facial expression recognition
to clock in at a given camera equipped device and providing using kernel canonical correlation analysis (kcca),” Neural Networks,
their pin to assist with authentication. Another simple example IEEE Transactions on, vol. 17, no. 1, pp. 233–238, Jan 2006.
is to enable employees to control environmental controls [18] J. Choi, S. Lee, C. Lee, and J. Yi, “A real-time face recognition system
using multiple mean faces and dual mode fisherfaces,” in Industrial Elec-
within their working environment from their mobile devices. tronics, 2001. Proceedings. ISIE 2001. IEEE International Symposium
Many different applications can be created from the simple on, vol. 3, 2001, pp. 1686–1689 vol.3.
application that we have shown. [19] M. Li and B. Yuan, “A novel statistical linear discriminant analysis
for image matrix: two-dimensional fisherfaces,” in Signal Processing,
Some further work needs to be completed into the security Conference on, Aug 2004, pp. 1419–1422.
of storing encrypted information on an Android smartphone. [20] T. Qi, H. Deng, and W. Hu, “Face recognition using eigenfaces-fisher
Additionally some research into new facial recognition tech- linear discriminant and dynamic fuzzy neural network,” in Computer
Science and Information Technology (ICCSIT), 2010 3rd IEEE Interna-
niques need to be completed. Although 80% is a relatively high tional Conference on, vol. 8, July 2010, pp. 166–170.
pass rate an improvement could greatly improve the experience [21] V. Kshirsagar, M. Baviskar, and M. Gaikwad, “Face recognition using
for the user and reduce unwanted wasted time when interacting eigenfaces,” in Computer Research and Development (ICCRD), 2011
3rd International Conference on, vol. 2, March 2011, pp. 302–306.
with the application. Any improvement in this department will [22] S. Alakkari and J. J. Collins, “Eigenfaces for face detection: A novel
greatly affect the end user experience. study,” in Machine Learning and Applications (ICMLA), 2013 12th
International Conference on, vol. 1. IEEE, 2013, pp. 309–314.
[23] A. O. Sudana, I. D. Putra, and A. Arismandika, “Face recognition system
R EFERENCES on android using eigenface method.” Journal of Theoretical & Applied
Information Technology, vol. 61, no. 1, 2014.
[1] H. Hassan, J. Martinez-Rubio, A. Perles, J. Capella, C. Dominguez, [24] B. Bozorgtabar, H. Azami, and F. Noorian, “Illumination invariant face
and J. Albaladejo, “Smartphone-based industrial informatics projects and recognition using fuzzy lda and ffnn,” Journal of Signal and Information
laboratories,” IEEE Transactions on Industrial Informatics, vol. 9, no. 1, Processing, vol. 3, p. 45, 2012.
pp. 557–566, Feb 2013. [25] S. Aina, M. Zhou, J. A. Chambers, and R. C.-W. Phan, “A new sponta-
[2] C. Opperman and G. Hancke, “Using nfc-enabled phones for remote neous expression database and a study of classification-based expression
data acquisition and digital control,” in AFRICON, Sept 2011, pp. 1–6. analysis methods,” in Signal Processing Conference (EUSIPCO), 2013
[3] C. Kruger and G. Hancke, “Implementing the internet of things vision in Proceedings of the 22nd European. IEEE, 2014, pp. 2505–2509.
industrial wireless sensor networks,” in Industrial Informatics (INDIN), [26] A. P. Felt, E. Chin, S. Hanna, D. Song, and D. Wagner, “Android
2014 12th IEEE International Conference on, 2014, pp. 627–632. permissions demystified,” in Proceedings of the 18th ACM conference
[4] A. Ray, J. Akerberg, M. Gidlund, and M. Bjorkman, “A solution for in- on Computer and communications security. ACM, 2011, pp. 627–638.
dustrial device commissioning along with the initial trust establishment,” [27] H.-L. Yeh, T.-H. Chen, P.-C. Liu, T.-H. Kim, and H.-W. Wei, “A secured
in Industrial Electronics Society, IECON 2013 - 39th Annual Conference authentication protocol for wireless sensor networks using elliptic curves
of the IEEE, Nov 2013, pp. 5570–5575. cryptography,” Sensors, vol. 11, no. 5, pp. 4767–4779, 2011.