Unified Approach Towards Automation of any

Desktop Web, Mobile Web, Android, iOS, REST

and SOAP API Use Cases
Sarath Kumar Purushothaman Nishant Kashyap Divya HA
Technology Technology Technology
Myntra Designs Private Ltd. Myntra Designs Private Ltd. Myntra Designs Private Ltd.
Bangalore, India Bangalore, India Bangalore, India
[email protected] [email protected] [email protected]

Sneha Agarwal Sandal Iqbal Vaishali Behere

Technology Technology Technology
Myntra Designs Private Ltd. Myntra Designs Private Ltd. Myntra Designs Private Ltd.
Bangalore, India Bangalore, India Bangalore, India
[email protected] [email protected] [email protected]

Abstract—In today’s world, testing cannot be efficient suite against different channels (i.e., desktop web, mobile
without automation. We need automation for avoiding web and native mobile application) with minimal changes.
repetitive work, making sure that the time from development
to deployment is reduced with good quality. We just don’t need II. LITERATURE SURVEY
automation tests, rather tests which are reliable, robust, easy to
code, debug, scale and can run in parallel on distributed A. Karate
environment. Karate[1] is a java library for testing web services -
SOAP and REST. Karate library is written on top of
The paper explores the challenges that can be faced while Cucumber- JVM. It is an open-source project that provides
automating web, mobile web, android, ios, api tests via a libraries for web api testing, mock and performance testing.
common framework such as, a common way of interaction Karate also provides a standalone executable for teams that
with web and mobile apps, automatically identifying connected are using Non-Java programming languages, to write
mobile devices and run the parallel test instances, writing programming language neutral testing code to deal with
hybrid tests; a combination of web and api use cases, not XML, HTTP or JSON. Some of the important features of
tightly coupled with a specific application rather can be used
Karate.
for automating any web, mobile based application.
• DSL support with support for XML and JSON
This paper further discusses the approach to bring
efficient, generic and re-usable solution for these challenges • Tests are very readable
while ensuring that the users write reliable, robust and
effective tests. The approach followed here spans across • Embedded UI for debugging and re-play steps while
following areas:- Webdriver management and parallel run. editing
DSL layer on top of the existing api libraries for easy to use • Built-in support for switching between multiple
interface. Reporting and logging mechanism. Utilities and
environments
integration with third parties (like Slack, Jira, Bitbucket,
Jenkins, Device Farms, etc ) • Pluggable HTTP client abstraction to support
Apache and Jersey
The benefit of this framework is, it is application agnostic
capability, which helps to use the same framework against any • Built-in test reports powered by Cucumber JVM with
web applications (web,mobile) for test automation. the option of choosing any third-party reporting tools

Keywords—api, rest, soap, webservice, automation, mobile, • Mock server for API calls that maintains CRUD
web, ios, android. across multiple calls

I. INTRODUCTION • Allows re-using test suites for Gatling performance

tests that asserts server for OK responses under load
How to develop an automation framework which can be
used to automate api, ui, across channels, seamlessly and
generic enough to work with any web/cloud based Things that could have been better
application, can be distributed as a jar and work with any
test automation project, appium session and webdriver • Support for JDBC entity mapping
management is automatically taken care, right driver and
matching browsers are downloaded and used based on • Handling the conversion between API response to
platform. In order to overcome such road blocks, the POJO and custom mapping
proposed system was devised to work with any application • Common Utilities are written in Javascript
across channels and platforms. This approach was
developed with a vision to have a unified automation code, • No much of documentation on asynchrony
which could be used to run and automate functional test

978-1-5386-0576-9/18/$31.00 ©2018 IEEE

Authorized licensed use limited to: JADAVPUR UNIVERSITY. Downloaded on August 14,2023 at 10:13:59 UTC from IEEE Xplore. Restrictions apply.
B. Rest Assured • Test management
The Rest Assured[2] is a Java library for testing REST - Test data are placed in flexible folder structures to
web services. Rest Assured allows us to create highly maximize reusability.
customizable HTTP requests and send to a RESTful web
service. Rest Assured library provides utilities to handle - Support dynamic data driven using CSV, Excel or
JSON / XML and handle assertion using Hamcrest database
expressions. Some of the important features of REST
• Usability and Functionality
Assured.
- Provides Object Spy for capturing locators of
• DSL support with support for XML and JSON elements in the application.
• Ability to serialize-deserialize request-response as • Analytics
Plain Old Java Objects
- A dashboard application to get Intelligent report on
• Reusing response validation using flaky tests, coverage of the test execution to improve
ResponseSpecBuilder strategy
• Provides support for testing Spring controllers as it - Screenshots, Videos, Logs.
comes with Spring Mock Mvc module
- Customizable post execution workflow to notify or
• Supports testing testing of asynchronous RESTful web report any malfunction.
services.
• Record and Replay
- Record the test execution flow and save it as code to
Things that could have been better be replayed any time.
• Issues when executing tests in parallel[3] • Integration with JIRA, Slack, CI/CD tools, etc.,
• Validation of complex JSON[4] • Parallel execution of tests supported via integration
C. Selenide with SauceLabs, BrowserStack, etc.,
Selenide[5] is a wrapper on top of Selenium library for • Provides an eclipse-like IDE for writing code and
writing UI tests. Selenide was create to resolve the main recording steps.
problems in UI testing - instability caused due to loading of
dynamic content, javascript and Ajax calls, timeouts, etc,[6]. • Supports API testing - REST and SOAP for functional
Some of the features of Selenide. and end-to-end testing.

• Smart waiting based on context.

Things that could have been better
• Transparent WebDriver - no need to interact with
Selenium web driver directly. • Everything happens within the Katalon studio IDE
• Convenience methods - lot of utility methods to • Project need to be hosted with the Katalon studio to
handle actions on web page like upload, download, run the project
click, set value, etc,.
• Ajax support - waits on condition.
III. PROPOSED SYSTEM
• Automated screenshots - ability to allow take
screenshots on success, failed test cases. Our proposed framework is an unified solution to
perform API testing and UI testing - desktop web, mobile
• Has built-in mechanism to do profiling of every web testing, android and iOS app testing. UI testing feature
action made on the web page. is enabled through flag, so as to launch the WebDriver for
different channels. Our proposed framework uses standard
• Parallel execution of tests at class and method level. open-source libraries to provide a complete suite of features
for testing API and UI. REST API and SOAP over HTTP
are supported by our proposed framework. Our proposed
Things that could have been better. solution is platform agnostic and both API tests and UI tests
• Externalizing configuration that could be loaded[7]. written using the framework, and can be run on any
platform - Windows, Macintosh, Linux seamless with
• A good reporting of the test run - depends on TestNg absolutely no change in the test intend objects. It is an
or Junit libraries as of now. hybrid framework supporting both API and UI testing, can
both can be used simultaneously. In API tests, UI utilities
• Support mobile app testing is missing. can be added and vice-versa.
• No built-in logging support. Our proposed framework is exposed as a standard
D. Katalon maven[9] dependency, which can be downloaded using any
build tools like maven, gradle[10], sbt[11], etc.,
Katalon[8] is built on the top of Selenium and Appium.
Katalon studio provides lot of functionalities. Katalon Our proposed framework uses TestNG[12] as the testing
provides with Katalon Studio, Katalon Recorder, Katalon library, and thus allowing us to leverage all the capabilities
Analytics. Katalon Studio bundles with everything. Some of of TestNG. The test project can be written in any JVM
the features of Katalon language.

Authorized licensed use limited to: JADAVPUR UNIVERSITY. Downloaded on August 14,2023 at 10:13:59 UTC from IEEE Xplore. Restrictions apply.
 Configuration management in the proposed framework methods, the commonly faced reasons of tests being flaky
hap- pens using properties file. JVM argument takes the are handled and thereby the consumer of the framework can
highest precedence, followed by the project specific be rest assured that the test failure is not because of test
config.properties, which is expected to be present in the being flaky but because of an issue in the application.
classpath of the project. If the given property is neither
present in JVM arguments not in the config.properties, it Mostly common reasons being,
falls back to default.properties bundled in the framework. • Stale element references.
Switching between environments is supported.
• Dynamic loading of content using AJAX calls.
Our proposed framework broadly contains 2 layers: (1)
Test Runner layer (2) Utilities layer • Trying to operate on element before the element is
enabled or available to be operated.
A. Test Runner layer
All the above scenarios are handled as part of the interface
Test Runner layer packs the interfaces for the test intents
provided by the framework with which the test intends
to interact with the framework and API calls or interacting
interact with. When test intends interact with the interface
with WebDriver incase of UI tests.
all the reasons of test being flaky and providing false
1) API Testing: Test Runner exposes RestClient positive test results are handled and this provisioning
interface for API testing, which has pluggable enables the consumers of the framework to just adhere to
implementation. Our current implementation is using the task of writing the test intent and rest all are taken care
UniRest[13] for making REST API calls and SOAP web by the framework out of the box. Configuration
service calls over HTTP. UniRest, even though it is a management of UI tests is simple and almost any behaviour
lightweight HTTP client, yet powerful and supports can be overridden at individual project level, like the wait
seamless parallel execution and asynchrony out of the box. timeouts, capability of webdriver - saving downloaded file
All test intents interact with the interface directly and to the location, position and dimension of the browser
underlying implementation will never be known to the window, etc.,
consumers of the framework. Even in future, if there is a
Alongside the above capabilities, listeners are integrated
need to switch to any other HTTP library, the test intents
with the framework to enable taking screenshots of
need to make absolutely no change and can be ported easily
webdriver instance and highlighting the point of failure.
to the new library as long as it adheres to the interface
Incase of tests run real devices, capturing the device logs for
having abstract methods to make different HTTP calls like
the application, performance stats, etc., and for web
GET, POST, PUT, DELETE, etc., Our proposed framework
browsers, capturing browser and network logs are also
provides a clean DSL builder pattern to build the request
provisioned. Thus allowing the consumers of the framework
with headers, request parameters, path parameters, request
to directly analyze the reason of failure of the testcase
body. Request body can be a simple JSON String, XML
directly from the allure reports, instead of digging deep in
String or it could be a Plain Old Java Object (POJO) also.
the logs.
This conversion of Plain Old Java Object to appropriate
String - JSON or XML happens based on Content-Type The proposed framework also provisioned with capability to
header that is passed along with the request. Our proposed maintain locators of web elements in a structured way for
framework supports with different utility methods for easy usage. The framework encourages Page Object
conversion of response to Plain Old Java Object and vice- Model[16] pattern for UI tests for having well-structured
versa, saving response to file, parsing and extracting data and reusable components of the application. Thus
from complex XML or JSON response, etc., makes handling externalizing the invariable and variable locators to be
with response very easy. Writing any utility methods will be modified without making change in the Page Objects. This
super simple as it can be used in JVM language. feature is exposed using AbstractBasePage object to allow
consumers to directly make use of the locators capability
2) UI Testing: UI testing uses standard libraries like
with the wrapped webdriver utilities.
Selenium[14] for desktop web and appium[15] for mobile
testing. Based on which channel - namely desktop web, Parallel execution of API tests is controlled using TestNG
mobile web, android app, ios app, the corresponding configuration of providing thread count, which can also be
WebDriver is created. With the feature of emulating mobile overridden by the framework to have a better control. Incase
browser on desktop chrome browser, the framework also of UI tests, again the parallelism is controlled using TestNG
allows to run mobile web tests on desktop chrome browser configuration when it comes to running tests for desktop
as mobile browser. This is driven through configuration web and emulated mobile chrome. When it comes to
whether to use real device or emulated chrome browser, and running tests on real devices from local machine,
also which device need to be emulated - Google Pixel 2, parallelism is controlled based on number of devices
iPhone 8, etc., The proposed framework also allows to use connected to local machine. When using any external device
custom phone dimensions and pixel ratio. For testing mobile farms, depends on the subscription. Parallelism at both test
apps and mobile web on real device, the proposed class and method level supported by the framework.
framework starts the appium session through code and the
device to be run on is identified using Android Debug B. Utilities layer
Bridge (ADB) commands in case of Android and idevice id All the test intends and helper classes interact with the
or xcrun command in case of iOS real devices and simulator Utility layer to perform various utility operations.
connected to the local machine respectively. This works
only on local machines. Connecting to external device farms
like BrowserStack, SauceLabs, Amazon Device Farm, etc.,
is also provisioned the framework configuration.
Framework also exposes utility class containing methods to
interact with the WebDriver. In these utility classes and

Authorized licensed use limited to: JADAVPUR UNIVERSITY. Downloaded on August 14,2023 at 10:13:59 UTC from IEEE Xplore. Restrictions apply.
 and Excel utilities. All these utilities are implemented using
open-source libraries.

IV. INTERPRETATION OF RESULTS

From the Table 1, we can infer that the most evident

advantage of the proposed system over the existing systems
is that it supports Both API and UI testing. This is very
important when we need to automate a Hybrid scenario.
From the Table 1 and Table 2 the proposed system
overcomes most of the disadvantage of the existing systems.

Fig. 1: Components of the framework

I. COMPARISON OF FEATURES FOR API AUTOMATION

1) Logging: Unified logging mechanism is bundled in the
framework, to allow usage of any implementation of Logger RestAssu Proposed
S.No. Features Karate
depending on choice. Logging has been exposed as an red System
interface so that it can be used in any class. Default
1 Support for SOAP API Yes Yes* Yes*
configuration for logging using log4j[17], log4j2[18],
logback[19] are bundled in the framework. Depending on 2 Support for REST API Yes Yes Yes
which implementation is chosen, corresponding
configuration will be chosen for logging. Project specific Easy understanding of tests
3 Yes No No
configuration can also be added to enabled controlling the (plain English)
different parameters of logging. 4 DSL Support Yes Yes Yes
2) Database: Database support is enabled using Built-In support for
configuration flags. Our proposed framework uses standard 5 Yes No Yes
logging
Spring JDBC[20] with Hikari[21] connection pooling for
maintaining database connections and operate on them. The Built-In support for
6 Yes No Yes
reporting
advantage of using Spring JDBC with Hikari connection
pool is that, it provides a comprehensive and optimal way to Built-In support for
7 Yes Yes No
maintain connection pooling and lot of utility functions to performance testing
manipulate the result set in a more easy way. This addresses
Support for Database
the JDBC entity mapping between the result set and Java 8
testing
No No Yes
object.
Easy handling of API
3) Command Line Executor: CommandLineExecutor 9
response to POJO
No Yes Yes
interface to execute native system or terminal commands.
Most commonly used methods like handling find process, 10 Support for mocking Yes Yes No
kill process, etc., are added into the framework itself
11 Seamless parallel execution Yes No* Yes
handled for all operating systems.
*Rest Assured and our proposed framework supports SOAP over HTTP only
4) Reporting: Reporting is provisioned using Allure[22]
Reports as part of the framework. However, consumers are
free to add and extend their own reporting strategy and plug II. COMPARISON OF FEATURES FOR UI AUTOMATION
it in using TestNG listeners. For API tests, out of the box we
are getting all the requests and responses attached to the test S.No. Features Selenide Katalon
Proposed
System
case and easy debugging. For UI tests, browser logs, device
logs, appium server logs, performance stat logs, screenshots Support For desktop web
taken on test failure, etc., are attached to every testcase. This 1 Yes Yes Yes
and mobile web testing
allows consumer of the framework to analyze and
understand about the test run and failures directly from the Support for android and ios
2 No Yes Yes
app testing
reports than reading through the logs.
3 Smart waiting Yes No Yes
5) File Utilities: Utility methods to read, write files,
handling, archiving and unarchiving of files, directory 4 Transparent web driver Yes Yes Yes
creation, etc., and few more commonly used utilities are part
of the framework. 5 Utility methods Yes No Yes

6) Listeners: As part of framework, few implementation of Solution for handling

6 Yes No Yes
AJAX and timing issues
TestNG listeners are added. Test logging on start and end of
test case, retrying on test failure, custom html report, etc., 7 Automated screenshots Yes Yes No
are added. Consumers of the framework are free to leverage
the capabilities of TestNG and have their own listeners. Parallel execution of tests at
8 Yes No Yes
any level
Few more utility classes that are added - RedisClient,
RabbitMQClient, EmailParser, YAML reader, CSV reader Easy configuration
9 Yes No Yes
management

Authorized licensed use limited to: JADAVPUR UNIVERSITY. Downloaded on August 14,2023 at 10:13:59 UTC from IEEE Xplore. Restrictions apply.
 18. Log4j 2.x. [Online]. Available: https://logging.apache.org/log4j/2.x/
Proposed
S.No. Features Selenide Katalon 19. Logback. [Online]. Available: https://logback.qos.ch/
System
20. Spring jdbc. [Online]. Available:
Build-In support for https://docs.spring.io/spring/docs/4.0.x/spring-framework-
10 No Yes Yes
reporting reference/html/jdbc.html
21. Hikari cp. [Online]. Available:
11 Built-In support for logging No Yes Yes
https://brettwooldridge.github.io/HikariCP
22. Allure. [Online]. Available: https://docs.qameta.io/allure/
12 No Yes No
Automated video recording

Support for analysis of test

13 No Yes Yes
failures

Support for record and

14 No Yes No
replay

Easy identification of object

15 No Yes No
locators

Project can be hosted and

16 Yes No** Yes
run anywhere
**katalon studio should be installed in order to run the tests

V. CONCLUSION
Each of the existing system provides a good support
either for API, Web or Mobile Application. But the
proposed system is the best for a combination of all. The
proposed system has a flexible configuration management
allowing the users to take advantage of features provided by
the system at their convenience. It is backed with an inbuilt
support for reporting and logging making debugging and
analysis simple. It allows users to write tests with absolutely
no change for different channels. The tests automated using
the proposed system can be run anywhere. The proposed
system uses only open-source libraries. These library
features are wrapped around the interface, thus giving the
framework the flexibility to switch to any other library that
comes up in the market with better features without
affecting the tests that are automated using the proposed
system.
REFERENCES

1. Karate. [Online]. Available: https://github.com/intuit/karate

2. Rest assured. [Online]. Available: https://github.com/rest-
assured/rest-assured/wiki
3. Request and response in thread local. [Online]. Available:
https://github.com/rest-assured/rest-assured/pull/851
4. Validating complex response. [Online]. Available:
https://groups.google.com/forum/#!topic/rest-
assured/vEXHeVQY1LQ
5. Selenide. [Online]. Available: http://selenide.org/
6. Selenium vs. selenide. [Online]. Available:
http://selenide.org/documentation/selenide-vs-selenium.html
7. Externalizing selenide configuration. [Online]. Available:
https://github.com/codeborne/selenide/issues/763
8. Katalon studio. [Online]. Available:
https://www.katalon.com/katalon-studio/
9. Apache maven. [Online]. Available: https://maven.apache.org/
10. Gradle. [Online]. Available: https://gradle.org/
11. Sbt. [Online]. Available: https://www.scala-sbt.org/
12. Testng. [Online]. Available: https://testng.org/
13. Unirest. [Online]. Available: http://unirest.io
14. Selenium. [Online]. Available: https://www.seleniumhq.org
15. Appium. [Online]. Available: http://appium.io
16. Page object pattern. [Online]. Available:
https://www.seleniumhq.org/docs/06_test_design_considerations.jsp#
page-object-design-pattern
17. Log4j. [Online]. Available: https://logging.apache.org/log4j/

Authorized licensed use limited to: JADAVPUR UNIVERSITY. Downloaded on August 14,2023 at 10:13:59 UTC from IEEE Xplore. Restrictions apply.