Real device testing:

Web and Mobile Device

Testing Best Practices
Real Device Testing

Device Testing
Options
Introduction In-house
Software development companies have two
major options for device testing — real and
virtual. Each solution has its advantages and mobile
There are over 14 billion mobile devices in
drawbacks depending on the application’s
lifecycle stage.
device
operation worldwide. Companies seeking to take Virtual device testing solutions like simulators or
testing lab
advantage of the global online consumer base emulators are great for early-stage testing, while
must constantly optimize their offering for web Some companies opt to build and maintain an
real device testing is crucial to ensure an app’s
and mobile devices. in-house device library for employees to quickly
success before it gets into the hands of end-
test on a variety of devices.
Network testing on real devices ensures that users.
mobile apps, mobile websites, web and desktop While this strategy might suffice for a quick
Often, utilizing a combination of testing options
apps remain functional and dynamic across unit test of a new or a recently updated feature,
produces the best end product.
operating systems, network connections, maintenance of an in-house device lab is
For example, a large e-commerce company
browsers, devices, and a global user base. difficult. Acquiring devices and updating their
might have its employees test a feature on its operating systems isn’t always the most cost-
Explore how to leverage real, in-the-wild web and in-house device lab, hire a networked testing efficient or scalable solution. An internal QA
mobile device testing for consistently flawless company for skilled manual testing, and utilize a team is normally focused on testing strategy,
user experiences. digital device farm during development sprints. management, and automation and too lean
to employ enough manual testers to cover all
devices. Using internal device labs is becoming
increasingly outdated due to their inability to
handle localization tests, device coverage, and
the drastic rise in remote work.

3 4
Real Device Testing

Emulators
Emulators have limited
potential applications,
and work best for lower
priority devices that still
need coverage, or during
the initial stages of testing. This technology
allows QA engineers to test unique devices on
their PCs by working on various device profiles. Device Device Testing Options
As part of a strategy, emulators can be an
effective tool, but there are disadvantages to farms
relying on device emulators as a primary testing In-house Device Lab Emulator / Simulator Device Farm In-the-wild devices
With device farms,
option including:
QA teams can test
• Extensive setup app performance on a
• Slow test time Pay-by-use, Included with third
variety of devices hosted in the cloud. Cost Upfront & ongoing Generally free
subscription party testing service
• Inaccurate screen resolutions and image
rendering Although these digital device labs provide easy
access, the simplified cleansing methods they
• External conditions (like the effect of
use to present the devices and the centralized Thousands of
loud noise on sensors or slow network devices/OS available
connections on task completion) can’t be location of the tests don’t match real-world Hands-on testing, Quick to switch Many devices
tested Advantage
environments. The device screen size and real-time between devices available Powered by real
• Internal hardware differences (CPUs and responsiveness are accounted for but the OS people doing real
testing
GPUs) aren’t tested and other physical elements such as battery life
• Touchscreen issues cannot be emulated or device storage are not.

Higher price point

Requires ongoing Location, UI via

Simulators In-the-wild maintenance mouse, solid network
Location, UI via
mouse, solid network
Individual devices
Disadvantage not on a connected
Device simulators devices Limited access for Doesn’t simulate
Only the most
network
remote employees hardware
are often used for popular devices
Optimal testing includes supported
regression testing and
deploying and testing
maintaining quality in
on real hardware
existing functionality.
with hundreds of device/OS combinations Function, regression,
Simulators can test the base quality and Parallelized test UX, location-based,
in the hands of human testers. Validation is Quick feedback for CI
functionality of core features, but apps often Developer testing automation localization, real-
builds
more accurate when it includes the device’s Best Use Case during sprints, real- device testing
run differently in a simulated environment.
responsiveness, how a real user interacts with time Specific devices for
Simulators can’t cover the full extent of the Developer checks
debugging Remote, burstable
that device, battery life, network connectivity
mobile environment because they have more teams
issues, and location considerations.
processing power than mobile devices, and
an inability to test external conditions like the
device battery.

5 6
Real Device Testing

Two real-life examples

Why Test on
Large enterprise, millions of monthly users,
lots of automation
Emerging growth company, small team,
no automation
Real, In-the-Wild
In-house Device Lab
• 2 devices per person, split between iOS &
Android & data plans vs. wifi only
In-house Device Lab
• QA person with 5-6 devices
• Dev & QA devices spread across models/OS
Devices?
• Drawer with older phones and wearables for
testing
While emulators, simulators, and device farms Some applications require extensive real device
have use cases that prove their effectiveness, testing to ensure that the product is interfacing
Emulators/Simulators Emulators/Simulators they can’t be the singular testing strategy. In- with sensors properly and the graphics fit
• Build verification tests, concentrating on • Developer quick checks the-wild device testing is the only strategy that each user’s screen. Responsiveness can’t be
functionality
• Quick feedback accounts for browsing, network connection, sacrificed during testing, and the multitude
and unlimited device and operating system of consumer screen sizes require real device
combinations experienced by real users. testing for proper functionality.
Device Farm Device Farm
• Extended automated tests concentrating on • Available for specific tests (low-level subscription/
functionality pay by use)
• Functional coverage: 2 iOS versions, 5 Android Example of device-specific bug
versions

In-the-wild devices In-the-wild devices What it should look like on iPads What it looks like on iPads
• 47 Device/OS permutations • 12 Device/OS permutations

Portrait Landscape

7 8
Real Device Testing

For software development teams building

consumer-facing products, the ability to test
apps on the same real devices used by their
target users is critical for several reasons:

1. Network realities 4. Fit and finish 5 Reasons to Test on Real Devices

As apps depend increasingly on mobile internet Human touch plays a significant role in
plans and wifi connection to access the network measuring how easy and user-friendly a
for data or third-party services, network glitches software application is and whether it meets Network Realities Location Device Realities
impact functionality. design expectations. Structured exploratory
testing helps uncover issues missed from
2. Location-based testing automated scripts while usability (UI/UX)
Wifi, data, 5G, 4G, etc. Location-based capabilities Bluetooth connectivity
Bandwidth with connection Movement Camera
Apps that offer services to users based on testing revolves around the entire app-driven (AR, scanning, photography)
Spoofing can be difficult
their geographic location, using GPS data from experience – something that requires real people Other apps running
cell towers or Wi-Fi, require testing in different testing on real devices and in real locations.
Resources like battery,
locations for enhanced accuracy and stability. memory and CPU
5. To find more bugs pre-release
3. Device realities The more devices you use for testing, the more
In most consumer devices, multiple applications likely you are to find a device-specific issue. Fit & Finish Find more bugs
are running on the same device and are pre-release
consuming hardware resources, such as CPU,
Smooth & expected rotation “human in-the-wild” devices
memory, network bandwidth, etc. In-the-wild reveal more device-specific
Usability, Accessibility
device testing can also account for Bluetooth Gestures bugs
connectivity and camera features (AR, scanning, Pixel peeping
photography).

“Testing an application across multiple,

in-the-wild, hardware and network
configurations is crucial because in the
real world, users report issues on real
devices, not on simulators or emulators.”

Kristi Kaljurand,
Testlio Engagement Manager

9 10
Real Device Testing

Which Real
Devices/OS
Combos to Use?
Testing and QA teams must be strategic when 3 ways to determine For instance, corporate finance/expense
approval apps have a user base that skews
choosing which device and device/OS version
combinations to cover when catering to a target
Avarage number of
device/OS combos tested
your real device needs towards latest and greatest devices, while users
of a family-oriented app are likelier to use older
user base, keep an eye on potential growth, and based on aggregate client
balance risk tolerance with deadlines. data
User analytics devices and OS.

device/OS When strategizing device coverage for an Analyzing users’ geographical distribution can
Based on a comprehensive analysis of client combinations
existing application, nothing is more important give insight into product growth and the type of
release data and device-OS configuration tests,
than usage analytics, including: user who might be adopting it in the near future.
Testlio found that leading mobile app companies
in most industry segments follow a best • Operating systems: do users typically
Avarage number of Technology innovation
practice of testing 24 or more unique device/OS devices tested per week upgrade to the newest OS version, or do you
for a top-tier shopping app need to support past versions? If so, how far When testing on brand new devices/ OS
combinations before each update.
back? versions, focus on getting a good breadth
device/OS
combinations • Devices and brands: which devices are of coverage. iPads and iPhones come in
commonly used? Are users on the most various sizes and models, but they have a sole
current versions? If there are multiple distributor. With Android, each manufacturer
versions of one device, how similar or
tweaks the OS slightly to fit with different
A Testlio client, a top-tier shopping app that different are those versions, and can they be
combined? offerings that differentiate their brands, such as
shares the highest App Store rating (4.9) with
auditory or motion sensors or a curved screen.
only three other Top 30 shopping apps, uses • Phone vs. tablet: what is the breakdown
41 device/OS combinations—almost twice the of use? How much of the device coverage Because Apple is pretty assertive with upgrading
median number because they have a large user should be allocated to phones vs. tablets? its OS, the latest two versions are generally
base and greater than average risk. sufficient to provide test coverage. For the latest
versions of Android, the Google Pixel series is
User profile generally a good bet as other Android phones
→ Check out the full report: State of App
Testing – Commerce and Retail. User profiles can provide focus on target tend to stick with the operating system in use at
consumers and what types of device they use. the time they were built.

11 12
Real Device Testing

Testlio’s research shows that iOS apps have higher ratings than Android apps on average in the
Commerce & Retail space.

Learn more about

4.6 4.3 -7% +4% Web and Mobile Device
The average iOS rating for the The average Android rating for On average, the rating for the top- Testlio clients see a 4% higher
testing
top-30 Commerce & Retail apps the top-30 Commerce & Retail 30 Android apps is -7% lower than rating for iOS than the Top-30
(Industry avg) apps (Industry avg) iOS apps (Industry avg) iOS apps

Learn More
While testing methods like emulators and in-house device labs have niche use cases, in-the-wild
device testing provides a holistic approach for companies wanting to build a better app, update, or
site. Look for these important features in any real device testing solution:

• Vetted testers
• Available in the global markets you need
• Ability to shrink and expand testing resources on-demand
• Can cover popular devices from the past four years
• Test cycle management
• Collaboration between freelancers to reduce duplicate issues
• Ability to utilize in-house QA teams alongside experienced testers in one platform

13 14
About Testlio
Testlio is the originator of networked testing. clients include athenahealth, CBS, Equinox,
Our award-winning combination of platform, Microsoft, the NBA, and Netflix. Collectively, they
freelancers, and services enables you to deliver have awarded us an industry-leading 4.7 G2
great customer experiences—fast. In any rating and an NPS of 75.
location. On any device. In any language. Testlio