See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/328650918

A Study of Mobile Forensic Tools Evaluation on Android-Based LINE Messenger

Article  in  International Journal of Advanced Computer Science and Applications · November 2018

DOI: 10.14569/IJACSA.2018.091024

CITATIONS READS
7 1,324

3 authors:

Imam Riadi Abdul Fadlil

Ahmad Dahlan University Ahmad Dahlan University
202 PUBLICATIONS   940 CITATIONS    63 PUBLICATIONS   118 CITATIONS   

SEE PROFILE SEE PROFILE

Ammar Fauzan
Universitas Nahdatul Ulama Sumatera Utara
3 PUBLICATIONS   20 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

DDoS Attacks Classification Using Numeric Attribute-Based Gaussian Naive Bayes View project

THESIS View project

All content following this page was uploaded by Ammar Fauzan on 02 November 2018.

The user has requested enhancement of the downloaded file.

 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 10, 2018

A Study of Mobile Forensic Tools Evaluation on

Android-Based LINE Messenger
Imam Riadi1 Abdul Fadlil2 Ammar Fauzan3
Department of Information System Department of Electrical Engineering Department of Informatics
Universitas Ahmad Dahlan Universitas Ahmad Dahlan Engineering
Yogyakarta, Indonesia Yogyakarta, Indonesia Universitas Ahmad Dahlan
Yogyakarta, Indonesia

Abstract—The limitation of forensic tool and the mobile National Institute of Standard Technology (NIST)
device’s operating system are two problems for researchers in considers that forensic tools might have a degree of error and
mobile forensics field. Nevertheless, some kinds of forensic tools need to be evaluated by the test against different mobile
testing in several devices might be helpful in an investigation. devices [4]. Experiments conducted with mobile device
Therefore, the evaluation of forensic tool is one gate to reach the forensic tools can indicate the capability of the tools. The
goal of a digital forensics study. Mobile forensics as one of the forensic tools should produce valid results based on the fact in
digital forensics branch that focusing on data recovery process terms of data objects that are acceptable in the court.
on mobile devices has some problems in the analytical ability
because of the different features of forensic tools. In this II. LITERATURE REVIEW
research, the researchers present studies and techniques on tools
ability and evaluated them based on digital evidence of LINE A. Related Work
analysis. The experiment was combined VV methods and NIST In [5] the researchers conducted a comparative evaluation
standard forensic methods to produce a model of forensic tool of forensic tools for WhatsApp analysis on Android-based
evaluation steps. As the result of the experiment, Oxygen smartphones. The author choose WhatsApp because of its
Forensic has 61.90% of index number and MOBILedit Forensic
easiness for expanding the user base. When installing it, one
has the highest index number at 76.19% in messenger application
can virtually reach all contacts in his/her address book on the
analysis. This research has successfully assessed the performance
of forensic tools.
phone who have installed the same apps [6]. The researcher
evaluating performance and ability of some forensic tools, i.e.
Keywords—Forensic; investigation; mobile; evaluation; WhatsApp DB/Key Extractor, Belkasoft Evidence, and
performance Oxygen Forensic. The evaluation using the NIST forensic tool
parameter and additional parameters from the researcher. The
I. INTRODUCTION author did at least four steps to conduct this evaluation, i.e
Cybercrime is escalating and the race against simulation, forensic analysis, analysis result, and conclusion.
cybercriminals is never ending since the internet established. In [7] the authors want to emphasize on the forensic
The huge number of mobile phone users nowadays add the investigation process and to compare mobile forensic tools
new problem of this issue. As a result of this many users, in used in this research by using a framework developed by
addition to the traditional usage of mobile phones including National Institute of Standard and Technology (NIST). The
making phone calls and texting in SMS, now mobile phones authors used four forensic tools to examine one Android
are also used for making video calls and chatting in the instant device. The performance of forensic tools was rated
messenger. quantitatively. There is no strong reason in this work and the
The development of Android smartphone technology has previous reference why the forensic process has to use NIST
an impact on the fast-growing number of applications method or the specific tools.
developed for Android. Even though, cybercrime can happen According to [8] the researchers suggest the decision
in Android smartphone. The investigator has to be able to method theories trough performance and relevance parameter
solve the crime case with a mobile forensic method to find a while doing a hypothesis testing on forensic method and tools
digital evidence. Digital evidence is fragile, volatile and selection. This paper is inspired by the freedom of choice
vulnerable if it is not handled properly [1], especially in the necessitates theory. The freedom choice theory is a sense of
mobile device. Mobile forensic is a science field that studies responsibility that asks for separation between true and false.
the process of digital evidence recovery using the appropriate Sometimes the selecting process for choosing the right
way from a mobile device [2] which usually doing in a digital forensics tool is complex with major consequences. The
forensic investigation by the police. Digital forensic author suggest the project to evaluate the performance of more
investigation is the phenomenon that solves the digitally tools against a broader set of mobile devices will help in the
committed crime and explores the culprit legally [3]. It is selection of the most appropriate forensics tool. In the
important for examiners and investigators to have the previous work, the National Institute for Standards and
knowledge about mobile forensic methods and the tools.

201 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 10, 2018

Technology (NIST) conducted an evaluation of the forensics Experiment in the past concentrated on the trustworthiness of
tools as an independent third party. digital evidence that is the product of the process and not the
validity of the tools. Recently, there is an attempt to formalize
In reference [9] the researcher doing "Validation the theory of digital forensics and dissertation about definitive
approach" since the tools were of propitiatory nature and there research that focuses on the model the process has already
was no access to their documentation and source code. This started to appear. There is also research on validation of the
paper presents the findings with respect to the reliability of the investigation results forensic (that is the reliability of the
tools only. The authors evaluate XRY and UFED forensic tool evidence), only a few on the reliability of tools that produces
in the light of NIST Smartphone Tool Specification which the evidence. The researcher have to consider that when the
consists of a number of specifications with their associated examiner/investigator want to conduct an analysis, they need
Test Assertions and Conformance Indicators. to use a method along with forensically tested tools [15]. Each
Performance can be measured from historical data or from tools can be validated and verified on its merits and the
the results of carefully designed experiments. Historical data examiner can focus on the results required rather than the
included performance evaluation results by both the vendors domain of all possible functions and all possible
and a trusted third party. The problems, however, were that: specifications.
(i) vendor evaluation lacked trust and (ii) trusted third party's
C. Mobile Device Forensic Tools Evaluation
evaluation used different mobile devices to evaluate the
forensic tools. The tools were not evaluated on equal grounds Mobile device forensic tools evaluation is consist of the
and thus the results cannot be generalized for comparing their validation and verification process. Validation is the
performance. confirmation by examination and the provision of objective
evidence that a tool, technique or procedure functions
Every digital forensic method has different stages in each correctly and as intended, while verification is the
handling of the digital evidence found, so in the handling of confirmation of a validation with laboratories tools,
various evidence, it requires different digital forensic models techniques, and procedures [14]. It is important for a forensic
[10]. In many references, digital forensics process at least can examiner to know how reliable and accurate a tool is before
be divided into four steps as in Fig. 1, collection, preservation, being used. The researcher have used the evaluation to gauge
analysis, and presentation [11]. The naming four stage of and verify the reliability and accuracy of two most prominent
digital forensic model is very flexible to be changed as needed mobile forensic tools such as MOBILedit Forensic and
for investigation. Sometimes at the end of the process called Oxygen Forensics based on the Smart Phone Tools
"reporting" instead of presentation and at the beginning begins Specifications by NIST [16]. The parameters for tool
with the identification process before collection/preservation. evaluation are depend on the needs of researchers, but they are
not far from the issue background.
III. RESEARCH METHODOLOGY
A. Evaluation Method
This article is inspired by many previous works of forensic
tool evaluation, one of them is validation verification (VV)
methodology that was proposed by Guo, Slay, and Beckett
Fig. 1. Digital Forensic Process. [17]. The first step in evaluation is listing the forensic tools
function. From the documentation of both tools; Oxygen
Having knowledge of the digital forensic process is Forensic and MOBILedit Forensic, their function as seen as in
important, same as forensic tools that have a vital role in the Table 1.
whole forensic process. Examiners must understand the
capabilities of a forensic tool with insights from good TABLE I. FORENSIC TOOLS FUNCTIONS
references of tool testing. But, most of the mobile forensic tool
testing and evaluations are done by the vendors. Mobile Oxygen Forensic MOBILedit Forensic
forensic tools developed in the forensic world are rarely
validated independently and scientifically. Moreover, forensic Device Identification Device identification
tools are used almost in all the stage of mobile forensics
Application data extraction for
process. Data Extraction
Android and iOS
B. Digital Forensics Problem
Messenger Application Analysis Application Analysis
There are many proprietary forensic tools have been
developed. As a result, a wide variety of tools exist to extract Data Report Data Report
evidence from mobile devices, no one tool or method can
acquire all the evidence from all devices [12]. The software Case Management -
applications for mobile forensics available today are not 100%
forensically sound [13]. The complexity formally representing - Deleted data retrieval
all the science need to start with a literature and discussions
with industry leaders from diverse backgrounds [14].

202 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 10, 2018

 Device Identification : The ability of a forensic tool in C. Tool Validation

device recognition Forensic tools validation can be done accurately by
 Data Extraction : The ability in data extraction from the judging the performance index number as shown in equation
device (1). Performance is measured in terms of probability of
successful (Ps) extraction of a particular type of digital
 Messenger Application Analysis : The Ability to show evidence by a specific forensics tool using the equations
the content of messenger application below:
 Data Report: The ability of tools evidence (1)
documentation in form of report file (.xml, .pdf, .xsl,
etc.) The number of objects extracted by two forensic tools,
Oxygen Forensic and MOBILedit Forensic. Objects that
 Case Management : Management of cases during the populated in this experiment is from LINE messenger by
analysis process manual acquisition. Equation (1) used to calculate the index
 Deleted data retrieval : The capability of a forensic tool number of the messenger application analysis and the data
to retrieve any deleted data from the device report from each forensic tool. This equation also can be used
in validating the data report for each forensic tools. The
Six aspects above need for validation and verification for equipment that used in this research can be seen in Table 2 as
evaluating the tools. Validation technique used quantitative follows:
calculation so that assessment more objective, but to verify,
the researcher simply apply quantitative assessment. Among The whole research processes can be drawn as in Fig. 3.
the aspects that can be considered qualitatively are device This research model is an adoption of NIST method with
identification, data extraction, case management and deleted alteration with VV methodology as needed for the research
data retrieval. While to messenger application analysis and purposes.
data report can be assessed quantitatively in the term of the
performance in producing the evidence. TABLE II. EVALUTION RESULT FROM OXYGEN FORENSIC AND
MOBILEDIT FORENSIC

No. Equipment Description

1 SONY Xperia Z Android Smartphone
2 ASUS A455L Workstation, OS Win.10
3 Oxygen Forensics Suite 2014
4 MOBILedit Forensic Express Ver. 4.0
5 USB Cable Ver. 2.0

Fig. 2. A Brief Process of Forensic Tools Evaluation.

This experiment was conducted using simulations on a

smartphone and two forensic tools. The brief process of this
experiment is described in Fig. 2. The explanation of the tool
verification and validation will be described in the next
section.
B. Tool Verification
In the verification process, the researcher compare the
function of the forensic tools with the experiment they did.
Some functions that need to be verified are device
identification, data extraction, case management in Oxygen
Forensics, and deleted data retrieval in MOBILedit forensic.
Verification is done manually by comparing one by one
function then assessed by its performance. Fig. 3. Tools Evaluation Methodology.

203 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 10, 2018

The evaluation methodology can be modified according to Both devices are quite good in the device identification
the needs and the expected results. The method above is one function. Although Oxygen forensics has its lack, at least it
method that can be applied in evaluation tool research. can recognize the device's manufacturer name. These results
may be different on the other devices.
IV. RESULT AND ANALYSIS
B. Data Extraction
The evaluation is ended with the documentation process.
This documentation can be either a report or a presentation file Data extraction on both devices is desirable, as this is
to show to the examiner and investigator. The results of the much needed in a long-period investigation. Data extraction
evaluation process that is conducted by applying VV methods on Oxygen Forensics is quite successful because it is able to
are as follows: create backup files from data acquisition devices, as shown in
Fig. 6.
A. Device Identification
While in MOBILedit, as seen in Fig. 7, data extraction to
The device identification is the first step that must be done generate backup data is not as good as expected, because the
by any forensic tool. The collection of information about the data extraction data that we get was corupted and error.
device is very useful in the report on the final process. Oxygen
forensic is able to identify the device that the researcher use, Both forensic tools have different ways of extracting data.
Sony C6602 or known as Sony Xperia Z, as can be seen in MOBILedit is not success in performing its functions.
Fig. 4. But, Oxygen forensics is not able to recognize the However, Oxygen can be used in investigations over a long
IMEI number or the serial number of this device. period of time, so the examiner can analyze the digital
evidence more deeply. The difference of the forensic tools
While in MOBILedit forensic the device identification result can be aspect that can be considered by the examiner.
result is as expected. Metadata from the device like serial
number, IMEI, IMSI, ICCID, Root status. All of the important
metadata can be revealed and documented as in Fig. 5.
MOBILedit forensic is quite successful in identification
mobiledit mobile device.

Fig. 6. Data Extraction by Oxygen Forensic.

Fig. 4. Device Identification by Oxygen Forensic.

Fig. 7. Data Extraction by MOBILedit Forensic.

C. Messenger Application Analysis

Analysis of messenger apps in this experiment will see the
ability of forensic tools on LINE messenger analysis. LINE
messenger that was tested is the latest version, with simulated
conversations that have been done in it. In Oxygen forensic,
messenger analysis result is presented in table form by
displaying ID, the direction of the message, remote party, text,
and the timestamp. In the Fig. 8 there are no any images nor
Fig. 5. Device Identification by MOBILedit. videos that can be displayed.

204 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 10, 2018

TABLE III. EVALUATION RESULT FROM OXYGEN FORENSIC AND

MOBILEDIT FORENSIC
MOBILedit Forensic
Function Oxygen Forensic
Express
Device Identification As expected As expected
Data Extraction As expected Not As expected
Case Management As expected N/A
Deleted Data Retrieval N/A As expected
Messenger
61,90% 76,19%
Application Analysis
Data Report 90% 100%

Table 3 shows a summary of the results from tools

Fig. 8. LINE Messenger Analysis in Oxygen Forensic. evaluation that we have been done. It can be seen that
MOBILedit looks better than Oxygen. However, for some
While in MOBILedit forensic, analysis of messenger is functions, such as data extraction and case management,
presented in the report file with a colored block display like a MOBILedit needs to consider installing it on the tool.
message application look, as shown in Fig. 9.
V. CONCLUSION
Analytical ability of MOBILedit Forensic has the highest
index number as much as 76.19% while Oxygen Forensic has
61.90% of index number. In this case LINE messenger
analysis. Oxygen Forensic can be better in data report than
MOBILedit forensic. MOBILedit has a limit in extracting
video in LINE messenger. However, MOBILedit Forensic is
don’t have case management function to as in Oxygen
Forensic, but MOBILedit is very efficient in term of data
report and data extraction.
VI. FUTURE WORK
Considering the growing number of smartphones and
forensic methods emerging, research on forensic evaluation
has to be done. In Future work, the researchers suggest the
evaluation of forensic methods and forensic tools more
Fig. 9. LINE Messenger Analysis in MOBILedit Forensic. detailed, so that the reference to this issue more complete.
Some suggestions about the evaluation parameter can be
Both of forensic tools have the ability to analyze the data, discuss in the further research as well as additional variations
but in different way. In this experiment, MOBILedit is of forensic tools that can be evaluated.
perform better than Oxygen Forensics.
REFERENCES
D. Data Report [1] I. Riadi, R. Umar, and A. Firdonsyah, “Identification Of Digital
Evidence On Android’s Blackberry Messenger Using NIST Mobile
Oxygen forensic has the ability to create reports in the Forensic Method,” Int. J. Comput. Sci. Inf. Secur., vol. 3, no. 5, pp. 29–
form of pdf, rtf, xls, xml, csv, tsv, and html. While 36, 2017.
MOBILedit has the ability to create reports in html, pdf, and [2] I. Riadi, A. Fadlil, and A. Fauzan, “Evidence Gathering and
excel formats. In oxygen forensic, only pdf files that unable Identification of LINE Messenger on Android Device,” Int. J. Comput.
work properly while others are pretty good. In MOBILedit Sci. Inf. Secur. (IJCSIS), vol. 16, no. June, pp. 201–205, 2018.
report is very complete and works entirely. [3] U. Kumar Singh, C. Joshi, U. Neha Gaud, and U. Chanchala Joshi, “A
Framework for Digital Forensic Investigation using Authentication
E. Case Management Technique to maintain Evidence Integrity,” Int. J. Comput. Appl., vol.
154, no. 6, pp. 975–8887, 2016.
Case management in Oxygen Forensics is reliable for
[4] R. Ayers, S. Brothers, and W. Jansen, “Guidelines on mobile device
deeper analysis. While on MOBILedit, there is no case forensics,” NIST Spec. Publ., vol. 1, no. 1, p. 85, 2014.
management like in Oxygen forensic. For this feature, [5] R. Umar, I. Riadi, and G. Maulana, “A Comparative Study of Forensic
MOBILedit has to consider for completed their tools. Tools for WhatsApp Analysis using NIST Measurements,” Int. J. Adv.
Comput. Sci. Appl., 2017.
F. Deleted Data Retrieval
[6] T. Sutikno, L. Handayani, D. Stiawan, M. A. Riyadi, and I. M. I.
The function for deleted data recovery was found on the Subroto, "WhatsApp, viber, and telegram: Which is the best for instant
MOBILedit forensic express. While on Oxygen this function messaging?," Int. J. Electr. Comput. Eng., 2016.
the researcher did not find it. This function is helpful in [7] I. Riadi and A. Firdonsyah, “Forensic Investigation Technique on
criminal cases where the perpetrator removes some data from Android ’ s Blackberry Messenger using NIST Framework,” Int. J.
Cyber - Secur. Digit. Forensics ( IJCSDF ) Soc. Digit. Inf. Wirel.
digital devices. Commun., vol. 6, no. 4, pp. 198–205.

205 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 10, 2018

[8] S. Saleem, O. Popov, and I. Baggili, “A method and a case study for the [13] K. Curran, A. Robinson, S. Peacocke, and S. Cassidy, “Mobile Phone
selection of the best available tool for mobile device forensics using Forensic Analysis,” Int. J. Digit. Crime Forensics, vol. 2, no. 2, pp.
decision analysis,” Digit. Investig., 2016. 1941–6210, 2010.
[9] A. K. Kubi, S. Saleem, and O. Popov, “Evaluation of some tools for [14] J. Beckett and J. Slay, “Digital forensics: Validation and verification in a
extracting e-evidence from mobile devices,” 2011 5th Int. Conf. Appl. dynamic work environment,” Proc. Annu. Hawaii Int. Conf. Syst. Sci.,
Inf. Commun. Technol. AICT 2011, no. 10, 2011. no. February 2014, 2007.
[10] R. Ruuhwan, I. Riadi, and Y. Prayudi, “Evaluation of integrated digital [15] R. Umar, I. Riadi, and G. M. Zamroni, “Mobile Forensic Tools
forensics investigation framework for the investigation of smartphones Evaluation for Digital Crime Investigation,” Int. J. Adv. Sci. Eng. Inf.
using soft system methodology,” Int. J. Electr. Comput. Eng., 2017. Technol., vol. 8, no. June, pp. 949–955, 2018.
[11] N. Widiyasono, I. Riadi, and A. Luthfi, “Investigation on the services of [16] National Institute of Standards and Technology, “Mobile Device Tool
private cloud computing by using ADAM Method,” Int. J. Electr. Specification Version 2.0,” 2016.
Comput. Eng., vol. 6, no. 5, pp. 2387–2395, 2016. [17] Y. Guo, J. Slay, and J. Beckett, “Validation and verification of computer
[12] E. Benkhelifa, B. E. Thomas, L. Tawalbeh, and Y. Jararweh, forensic software tools-Searching Function,” Digit. Investig., vol. 6, no.
“Framework for Mobile Devices Analysis,” Procedia Comput. Sci., vol. SUPPL., 2009.
83, pp. 1188–1193, 2016.

206 | P a g e
www.ijacsa.thesai.org
View publication stats