A Guide to

Functional
Testing in
Mobile
Introduction
Mobile devices have become little pocket computers that we, as users, can do anything and everything on.
We shop, game, and keep track of things by use of mobile applications associated with each - anything we
can do on the computer, we can also do on our mobile devices and tablets via mobile applications and
browsers. With the innovative expansion of technology to allow the world at our fingertips at lightning
speed, we have grown accustomed to a certain quality standard with mobile applications.

Industry research shows that the tolerance for poor mobile experiences is extremely low. So low, in fact,
that research shows that one in four mobile applications will be abandoned after initial use. Why is that?
Given the market influx of mobile applications, if a consumer has a poor experience on one app, there
are countless other apps that provide the same use and may serve a better experience. Not to mention
market ratings and reviews that can also have an impact on app abandonment. So, we are seeing
consumer competition with mobile applications, and the key to success is to deliver a top-notch
standard of quality where consumers will have a positive experience when using the mobile application.
It is a given that more positive consumer use yields more revenue flow, and thus, a successful application
that then, in turn, increases consumer retention and minimizes app abandonment.

In the world of software testing, one of the foundational testing methods that catches the most defects and
ensures a mobile application is running correctly to bring the best experience possible is functional testing.

What is Functional Testing?

What is functional testing?

Functional testing is a subset of software testing and is the most

important testing to be done. Functional testing is executed in
order to validate that parts and features of an application meet
specific, functional requirements prior to releasing to the public. In
simplest terms, it is the type of testing that ensures your
application is working as expected, meets business (functional)
requirements, and is overall functional for consumer use.

But, what are these business requirements to meet? Business

requirements are typically high-level requirements spelled out
and agreed upon between business analysts/stakeholders of a
company and the DevOps team in the planning stage of the
software development life cycle (SDLC), and an application
cannot be delivered unless it functionally does what it was
originally created to do. For example, a banking company’s
stakeholders will meet and plan with the DevOps team to create
business requirements that the banking application will view and
transfer funds in an account.
2
The DevOps team cannot deliver said application until it is verified that its function involves viewing and
transferring funds within an account. When we look at the testing cycle, functional testing can start
shifting the farthest left - the moment a feature is coding complete. Functional testing also extends
further into testing the usability and accessibility of an application. Let’s take a look at the different types
of functional testing.

Types of Functional Testing & Who Performs It

Unit & Integration Testing

Unit testing is a functional testing method that tests the

functionality of individual components/modules of an
application's source code. Once all modules of the
source code are merged together, integration testing
can be done. Integration testing is executed to test the
functionality of grouped modules to ensure one is
functional with the other and as a whole. By verifying
individual components are working as expected and
additionally with one another, one can ensure the
foundational code of an application is functional, else
one can catch defects at the earliest stages of the
SDLC. Developers mainly perform unit and integration
testing, though new trends show software developers in
test may perform unit and integration testing.

System Testing

System testing is a functional testing method that tests

the functionality of the complete and fully integrated
application. Within system testing, testers test and
validate the layers of the application: database testing,
API testing, and UI testing by verifying each subset of
the system is functionally working as expected and
meets business requirements. Quality assurance
engineers, QA analysts, and software testers perform
system testing.

3
User Acceptance Testing (End-to-end testing)

The main idea behind user acceptance testing is that

user flows are tested, either by the end consumer or by
testers in the end-user mindset, to ensure that the
application, as a whole, is functionally working as
expected. By this process, the UI, API, and database
are tested together with any service or microservice in
between. Additionally, user acceptance testing also
acts as UX testing in that the user flow is easy to follow
for the consumer. For example, in a login user flow,
the login button is subsequent from the password input
field. The user inputs username, password, and the
login button is clear and apparent to proceed with
logging in. The login button is not in the bottom left-
hand corner or in some obscure location on the page
that may otherwise confuse the user. Quality
assurance engineers, QA analysts, and software
testers perform user acceptance testing.

Maintenance Testing (Regression Testing)

From release to release, new code is merged and

pushed in with existing code. Regression testing is
performed to ensure new code does not impact existing
code, either within the same module of an application
or in different modules. Usually, at the end of the
release cycle, regression testing is done that
incorporates entire system testing. It’s as if it is the last
checkpoint in functional testing efforts before releasing
to production. This ensures that the application as a
whole, from system testing to user acceptance testing
to accessibility testing, functions as expected.
Regression testing tends to be the gate-keeper to
releasing: if no critical defects are found in regression
testing, testers give the OK to release.

4
Why is Functional Testing important?
As mentioned before, functional testing is the most important part of software testing as it is the type of
testing that catches the most defects. Without functional testing, it would be like releasing to the public
blindfolded. What will happen? Will the application work? Will ALL of the application work? When
defects make their way to production, they can have a huge impact on customer satisfaction, which in
turn, largely impacts the company. Furthermore, functional defects can impact the security of an
application by potentially leaking sensitive data. A dysfunctional mobile application is not ideal in any
sense and results in one-star ratings and bad reviews, along with app abandonment.

The ROI of Shift Left Testing

Most functional testing of a mobile application takes place in shift-left testing, testing early in the release
cycle and testing often. Not only does shift-left testing yield better quality and efficiency within the cycle,
but it also proves to be a major cost savior. The later defects are found in the release cycle, particularly
functional defects, the more expensive they are to fix.

SHIFT-LEFT The ROI of Shift Left Testing

Introducing testing early into the Life Cycle

Requirements Design Development Testing Release Maintenance

Testing

{
Better Quality
Shift-Left Better Efficiency
Lower Cost

5
THE ROI OF THE “SHIFT-LEFT” PHILOSOPHY

Cost of defect discovery by stage

x2 x4 x6 x8 x10

$ 250 $ 500 $ 750 $ 1000 $ 1250

Req. Coding. Integration Beta Post Release

Data Sourced from The Economic Impact Of

Inadequate Infrastructure for Software Testing.
Image by Kobiton. Assumes cost per defect $125.

It's plain to see that functional testing should start as early as possible from the moment requirements
are spelled out. And although no application is perfect and defects will slip into production, it's
important to perform all functional testing prior to releasing, especially regression testing. By
implementing this best practice, more defects are found earlier and can be corrected earlier,
shortening the release cycle and avoiding potential show-stopping defects. Time, money, and resources
are saved!

6
Manual vs Automated Testing
What is Manual Functional Testing?

Manual functional testing is pretty self-explanatory. A tester will perform functional testing by directly
interacting with the application. They test the functionality of an application by executing tests
themselves and compare expected results to actual results - either verifying the test case meets
requirements or logging a defect. The majority of functional testing is manual with goals to fully
automate.

Let’s take a look at the pros and cons of this approach.

Pros/Cons of Manual Functional Testing

PROS CONS

• Most intuitive way to test • Extremely time-consuming

• Minimal false positives or false • Extremely tedious with the amount of
negatives in testing functional testing that needs to get
• Fast ability to gather troubleshooting done
notes when logging defects (ie, • Repetitive testing (you might manually
provide errors and log information) test one feature and need to test again in
• When a test fails, you likely know regression)
exactly where, when, and why

Although manual testing can be ideal given the intuitive nature of humans, the amount of time it takes to
perform manual functional tests is a huge drawback in the fast-paced world that is technology. Teams
want to release as soon as possible, and manual functional testing can really draw out the release cycle.
This can have a huge, crippling impact on teams.

7
What is Automated Functional Testing?

Automated functional testing is when a manual functional test case is set up to run without the need of a
tester to execute each test step manually. This can be achieved either programmatically, in script-based
automation, or from a top-layer/ UI perspective in scriptless automation.

Let’s take a look at the pros and cons of this approach.

Pros/Cons of Automated Visual Testing

PROS CONS

• Far faster than manual testing • Can be hard to implement

• Resolves the issue of repetitive • Tedious and hard to maintain
testing by being able to kick off an • False negatives/ false positives
automated test resulting in additional, manual testing
• Run tests in parallel to cover device (counterproductive)
fragmentation
• Ability to achieve continuous
integration/ continuous deployment
(CI/CD)

It is every tester's dream to achieve 100% automation, especially for functional testing. It can cut testing
time significantly so that teams can release sooner to deliver the best of their application to consumers.
But mobile application automation is difficult given the unique ‘rules of the road’ developed by Android
and Apple. It is not as open and straight-forward compared to web application automation. That is why
the majority of teams are only able to achieve roughly 20% of automated functional test cases.

8
What are the different ways to approach
automated Functional Testing?
As mentioned above, mobile application automation for functional testing is the most ideal way of
testing, but difficult to achieve. Not to mention device fragmentation can prove to be a huge blocker for
mobile application automation. That is why we are seeing new, AI-driven tools to help teams achieve
automation by offering a scriptless approach. By use of scriptless or scripted, or a combination of both,
testers are given a chance to achieve their automation dreams.

Scripted Functional Testing

Scripted functional testing is a programmatic approach to mobile application automation. One can code
out the test steps of a functional test case and add assertions to ensure business requirements are met.
This is the traditional approach to automation testing.

Let’s take a look at the pros and cons of scripted functional testing.

Pros/Cons of scripted Functional Testing

PROS CONS

• Far more granular control of scripts from • Programmatic approach is extremely

a programmatic approach tedious and time-consuming
• Ability to automate all types of functional • Can be slower than scriptless automation
testing (Unit, Integration, API testing, etc) • Can be flaky/ inconsistent
• Parallel testing is possible, but difficult to • Hard to maintain
set up • Hard to convert manual test cases
• Achieve CI/CD into reliable automation
• Free, open-source tools available • Huge learning curve

9
Scriptless Functional Testing

Scriptless functional testing for mobile applications is possible by AI & machine learning algorithms. One
can run a manual session once and be able to convert that into an automated test case without a single
line of code. The innovative approach that is scriptless automation has really helped bridge the gap of
playing catch-up from mobile application automation compared to the older, web-based automation.

Let’s take a look at the pros and cons of scripted functional testing.

Pros/Cons of scriptless Functional Testing

PROS CONS

• Far faster to set up compared to scripted • Loss of granular control

automation. Testers can hit the ground • Top-layer automation is difficult to
running with scriptless automation automate additional functional testing
• Easy setup and manual-to-automated ie, API automated testing
conversion • Comes at a subscription price
• Run once, test anywhere in parallel with
easy setup
• Usually comes with baked-in assertions,
so no time needed to code out assertions
• Easy to maintain
• Usually has ability to also convert
scriptless into scripted automation for
further use
• Also able to achieve CI/CD
• No learning curve - less technical
individuals can still achieve automation

It is clear that scriptless automation has fewer cons, though coming at a price can deter teams from
utilizing scriptless automation.

10
Emulators/Simulators VS. Real Devices
Whether performing manual or automated mobile application functional testing, you’re going to need a
device or device-like system to test the functionality of a mobile application. There are two solutions
offered: emulators/simulators and good ol’ real devices. But what are emulators/simulators? How might I
procure real devices? Which one should I use for my functional testing? Let’s take a moment to explore
what emulators/simulators actually are, the pros and cons of each solution, and which one to use when.

What is an Emulator?

An emulator is a software program that imitates the hardware and/or software of a target device on your
computer. Emulator programs behave in the same manner as a real device, as if the device was plugged
right into the computer, but set in an all-virtual environment. By creating a virtual environment that
mimics a target device’s hardware and/or software, developers and testers are able to perform basic
functional testing of an application running on an emulator.

What is a Simulator?

A simulator is a program that creates a virtual environment that mimics the internal behavior and
configurations of an application on a target device’s operating system. A key difference between
emulators and simulators is that simulators do not mimic the hardware of a target device. Instead, a
simulator spins up a virtual environment of a target device’s operating system to run applications within
the virtual operating system to build and test against. Like emulators, this virtual space can be used by
developers and testers to perform basic functional testing of an application, though simulators lack the
hardware functionality testing that emulators offer.

Let’s take a look at the pros and cons of emulators/simulators.

Pros/Cons of Emulation/Simulation

PROS CONS

• Free, easily available, and open-source • Cannot produce real-world conditions to

• Wide range of devices & OS offered test functionality against
(though basic versions) • Lack of functional testing overage
• Rich debugging (unable to execute end-to-end &
• Quick and convenient regression testing)
• No additional dependencies needed • Tend to have false negatives/ false
positives in functional testing
• Less reliable
• Slow

11
There are many pros to using emulators/simulators. They are best used in the early stages of the release
cycle to build and test the functionality of mobile applications as they are being developed. However,
emulators/simulators come with quite a handful of drawbacks and limitations, including lack of testing
coverage and inability to emulate/simulate real-world conditions.

Real Devices

Real devices are important to functional testing of a mobile application, especially towards the end of the
release cycle in end-to-end and regression testing. By testing on a real device, one can have the end-
user mindset as if the real device was in the consumer's hand. This provides the most accurate testing of
a mobile application's functionality. However, given the issue of device fragmentation, real devices are
hard to come by; it’s hard to cover the functionality of an application on every device.

Let’s take a look at the pros and cons of real devices.

Pros/Cons of Real Devices

PROS CONS

• The most reliable in functional testing • Costly

• Full functional testing coverage • Limited availability
• Most accurate understanding of UX • Hard to manage physical resources
(for user acceptance and accessibility (upkeep is tedious and time-consuming)
testing)
• Wide range of devices & OS offered in
Device Lab Management cloud services

At the end of the testing day, nothing beats the real deal that is performing functional tests on real
devices.

Which One to Use and When

The two aren’t mutually exclusive regarding functional testing. In the early stages of the release cycle,
one can spin up a free emulator/simulator to begin functional testing. As we move farther down the
testing cycle prior to releasing, it is imperative to perform functional testing on real devices. The cost to
procure real devices will surely have its benefit when more defects are caught on real devices before
releasing to production.

12
Functional Testing Tools for Mobile
Test Automation
If you are interested in investing in a functional testing automation tool, here are a few vendors that
currently offer functional testing automation:

Scripted

XCUITest

XCUITest is a native and hybrid framework for iOS mobile devices. Since XCUITest is baked right into
Xcode's IDE, it provides the fastest possible testing for iOS devices. XCUITest allows for automated UI
functional tests that can be applied to end-to-end and regression automation testing.

Espresso

For teams exclusively developing applications for Android devices, Espresso is a great framework for
automation testing at the UI level. It is free and fast to use for native Android applications and can also
be applied to end-to-end and regression automation.

Appium

Appium is one of the leading automation frameworks for mobile application. It helped solve the issue of
device fragmentation in that it can be used for both Android and iOS devices. You can develop one script
to run against all devices, though the caveat is that Appium is slower than the native frameworks
mentioned above.

Scriptless

Waldo.io

Waldo is a no-code platform for automation mobile application testing that builds end-to-end regression
suites that can be automatically run against new versions of your mobile app. Waldo.io's codeless
recorder records "flows" to create one test flow, or chain together multiple test flows to create
dependencies when building your tests. Create "flows," or test cases, and every time you upload a new
version of your app, Waldo.io will automatically kick off your tests.

13
21Labs

21Labs is a scriptless automation tool aimed at autonomous testing and maintenance to provide instant
coverage in less time. 21Labs promises AI-based learning to learn the structure of your application
through functionality of each screen and navigation. In doing so, 21Labs claims to execute a wide range
of self-teaching mechanisms to keep track of elements and their relative hierarchy, allowing you to
create stable, consistent, self-healing, and autonomous tests. However, be advised that 21Labs is not
“click through and automate” as advertised, and may get complex.

Kobiton

Kobiton is the industry leader in pure-play mobile test automation and mobile Continuous Testing. With
support for both AI-driven scriptless and AI-assisted script-based approaches, Kobiton offers solutions
around test authoring, execution, and remediation that will fit the needs of any mobile development or
testing team.

How do these vendors compare?

Waldo.io 21Labs Kobiton

Automated Functional Testing

“In-house ” functional harness

Built for mobile

Access to real mobile devices

Data Driven Testing

Rich logs + session exploration

Completely scriptless

Export to scripted?

14
About Kobiton
Kobiton is the mobile and IoT experience platform trusted by leading organizations globally. Our best-
in-class software platform helps drive improved revenue on the mobile and IoT channels by lowering app
abandonment, improving quality and reducing time-to-market.
Used by over 60,000 developers and testers worldwide, Kobiton is transforming the way companies
deliver mobile apps and IoT devices through innovative applications of Artificial Intelligence, Real-Device
Testing and the industry’s first and only mobile scriptless automation solution.

15
Drive quality across the entire SDLC. Prevent bugs before pushing test to code with advanced ADB
debugging and access to real devices within your IDE. Perfect your application site with the Kobiton
Intelligent Quality Suite, and build, execute, and report against scriptless and/or scripted automated
Functional, Visual, and Performance Tests. When issues are found in a test, you can easily resolve them
with AI-assisted remediation and ADB debugging. Finally, all of this is seamlessly integratable with your
CI/CD pipeline/tooling so that you can kick off tests with confidence and ultimately release faster than
your competition.

Run your tests on the industry’s most flexible and high-performance real device cloud or on-premise
solution for 30FPS video streaming, in-depth session exploration, and analytics solutions that offer
visibility and traceability throughout your entire testing process. With Kobiton, build, test, deploy, and
release better mobile apps, websites, and IoT devices. There's a reason the world's mobile elite choose
Kobiton to deliver perfect mobile and IoT experiences for their users.

16
www.kobiton.com · Phone: (678) 235-4095 · [email protected]