Automating Functional Tests Using Selenium

Najeeb Ullah

Abstract recent years, tools such as Fit and FitNesse [1] have
helped make it easier to automate functional testing of
Ever in search of a silver bullet for automated software applications. However, web applications
functional testing for Web Applications, many folks have long remained difficult to test because of multi-
have turned to Selenium. Selenium is an open-source tiered architecture, multiple browsers, and web
project for in-browser testing, originally developed by technologies such as JavaScript. Some teams have
ThoughtWorks and now boasting an active community chosen expensive solutions such as Mercury
of developers and users. One of Selenium’s stated Interactive’s WinRunner or Rational Robot (with
goals is to become the de facto open-source mixed results). Recently the open-source tool
replacement for proprietary tools such as WinRunner. Selenium, originally developed by ThoughtWorks, has
Of particular interest to the agile community is that it gained attention as a possible silver bullet for the
offers the possibility of test-first design of web problems of automated testing for Web Applications.
applications, red-green signals for customer We have been using Selenium on five different web
acceptance tests, and an automated regression test bed application projects for about a year. Our teams
for the web tier. consisted of 4 to 12 developers and 1 to 2 business
This experience report describes the standard analysts, with the latter role responsible for writing our
environment for testing with Selenium, as well as Selenium tests (along with other responsibilities). One
modifications we performed to incorporate our script of the projects used a .NET architecture, while the four
pages into a wiki. It includes lessons we learned about others were J2EE applications.
continuous integration, script writing, and using the
Selenium Recorder (renamed IDE). We also discuss 1.1. Background
how long it took to write and maintain the scripts in
the iterative development environment, how close we Since our teams practice an agile methodology, we
came to covering all of the functional requirements are committed to writing acceptance tests before
with tests, how often the tests should be (and were) beginning development on stories. Passing these tests
run, and whether additional automated functional signals completion of a story, so we write functional
testing below the GUI layer was still necessary and/or tests that capture these acceptance criteria. For a web
appropriate. application, a functional test could be simply that a
While no silver bullet, Selenium has become a user manually navigates through the application to
valuable addition to our agile testing toolkit, and is verify the application behaves as expected.
used on the majority of our web application projects. But since automating a test is the best way to make
It promises to become even more valuable as it gains sure it is run often, we try to automate our functional
widespread adoption and continues to be actively acceptance tests whenever we can. And our holy grail
developed. is to be able to write the tests before development, so
that the development team can have a runnable
1. Introduction verification that the story is complete.
Our team came to Selenium after using several
Automating functional tests is one of the traditional other tools to automate web testing. We tried the open-
problems facing software development projects. source tools Canoo WebTest and HttpUnit, but found
Whereas automated unit testing has achieved deep them to be insufficient, as they could not handle most
traction, in agile and non-agile projects alike, instances of in-page JavaScript. The JavaScript
functional testing frequently is still done manually. In problem is solved by the proprietary tool QuickTest
Professional (from Mercury Interactive), which offers

1
record-and-play of test scripts and also runs tests their applications in Firefox and record their
directly in a browser. However, we found that most of actions, forming tests; and,
our recorded scripts in QTPro would break after small 2. a “Remote Control” server which allows users
page changes, so a significant amount of script to write tests directly within the programming
maintenance was necessary. In addition, it was not language of their choice -- thus enabling
obvious that we would be able to write QTPro scripts conditional logic within tests, try/catch
before development. As a result, we turned to blocks, and other powerful functionality
Selenium – it showed the promise of easy-to-write available only in programming languages.
scripts that were relatively easy to maintain and which
could be written before the code. 1.3. Getting started

1.2. Selenium Setting up Selenium is easy, although there is a

catch. The basic installation of Selenium must be
Selenium is a web testing tool which uses simple hosted by the same web server as the Application
scripts to run tests directly within a browser. It uses Under Test (AUT). This restriction is due to the fact
JavaScript and iframes to embed the test automation that JavaScript has built-in security against cross-site
engine into the browser. [2] This allows the same test scripting. [3] After installation, the user simply needs
scripts to be used to test multiple browsers on multiple to begin writing and running tests.
platforms.
2. Writing good tests
Selenium tests are not difficult to write. Because
Selenium allows the identification of elements using
the browser’s DOM object, the test can be written
using specific identifiers of the necessary element,
such as name, id, or xpath:

Table 1: Input data for a Selenium Test

type name=theField Text to submit

clickAndWait id=SubmitButton
assertText xpath=//h1/span Success!

Figure 1: Selenium inside Internet Explorer

This test might be written for an HTML page as simple
Selenium gives the user a standard set of commands as:
such as open (a URL), click (on an element), or type <html>
(into an input box); it also provides a set of verification <input name="theField">
commands to allow the user to specify expected values <input id="SubmitButton" type="submit">
or behavior. The tests are written as HTML tables and <h1><span>Success!</span></h1>
run directly in the browser, with passing tests turning </html>
green and failing tests turning red as the user watches
the tests run. When the test is run, each command is highlighted
Because Selenium is JavaScript-based and runs as it executed, and the assert steps turn red or green to
directly in the browser (the user can see the test indicate success or failure. After the whole test is
running), it overcomes some of the problems complete, the test is marked as red or green in the
encountered by users of HttpUnit or Canoo WebTest, Suite.
particularly problems related to testing JavaScript
functionality. 2.1. Keeping tests self-contained
Two additional useful tools are also available for
use with Selenium: Selenium naturally supports a Suite of tests, and
1. the Selenium IDE (originally called the tests for a certain story or iteration can be grouped
Recorder), which allows users to navigate together in a Suite and run sequentially. But for

2
flexibility and maintainability, we strove to keep our requirements, stories, and tests. A wiki page can be
tests as independent and self-contained as possible. changed by any user and is open to the whole team,
This allowed us to move tests around and delete them, including the stakeholders.
as well as refactor tests mercilessly. We frequently By including our Selenium tests in our wiki, we
included one test inside many others to reduce code gave them the highest possible profile (everyone can
duplication, and we used SetUp and TearDown tests see them, track them, and run them) and also were able
throughout our suites in order to get the application to to integrate them with our other automated functional
the desired state for testing one piece of functionality. tests – a byproduct of using FitNesse as our wiki.
In fact, we found that our test-writing style became:
1. Write the tests before development; 3.1. FitNesse
2. After development, get them to green; then,
3. Refactor as mercilessly as possible. FitNesse is an automated test tool, wiki, and web
By refactoring our tests, we reduce duplication of test server all rolled into one application. Its stated goal is
code and increase the maintainability of the test suite. to enable customers, testers, and programmers to
collaboratively learn what their software should do,
2.2. Writing our own extensions and to automatically compare that to what it actually
does do. [5] Because any user can modify a page and
Some would argue that the top benefit of an open FitNesse keeps past versions of pages, all of the
source project is that it can be licensed for free; cost is members of the team, including the customers, use it as
undoubtedly a top reason, but it is the availability of a communication tool and everyone is an owner of the
source code that makes open source what it is. [4] As content.
an open source project, Selenium allows end users to
write and share extensions or other code modifications,
even ones that are project-specific.
We took advantage of this freedom by writing some
custom functions to make some of our tests easier to
write. We created a special Selenium command that
selected a value from a JavaScript-based tree after
storing the parent element into a reusable variable.
Selenium is built to handle such user extensions, as it
merely requires adding the JavaScript code to one
Selenium file.
Using such custom functions also allowed us to
have more readable tests.

3. Putting Selenium tests into the wiki

After writing enough tests to have a successful
implementation for our projects, we found that there Figure 2: FitNesse Wiki
were a few recurring pain points:
1. Keeping the tests organized was a nuisance; 3.2. Integrating Selenium and FitNesse
2. Writing the tests in HTML was unnatural;
and, Selenium’s built-in functionality is to read the tests
3. Using variables across multiple tests was from a designated URL, so we pointed it to our wiki
tricky. We realized early on that using suite. There were a couple of technical challenges,
variables would prevent us from having to such as authenticating with the wiki, but in the end we
change every test when a field ID would were able to run our wiki suite and tests from
change, but it was tricky enough that we Selenium’s TestRunner.
weren’t doing it. Because the wiki was centrally located, a developer
We decided that we could address all of these issues or tester was also able to run their own instance of
by leveraging the power of our project wiki. A wiki is Selenium and the application, using the central test
a collaborative web environment where any user can bed, and thus run the Selenium tests against their own
change the pages. It is our company’s standard project code at any time.
center – where we capture all of our meeting notes,

3
4. Continuous integration Cobertura is a code coverage tool for Java projects.
It calculates the percentage of code accessed by tests.
After every developer checks in code, we run all of [7]
our unit tests and integration tests to make sure that the
check-in did not break the build or the tests. Because
it’s important not to break the passing acceptance tests
as well, we wanted to add our Selenium tests to the
continuous integration build.
For our project’s integration builds, we used
CruiseControl. By setting up a deployment of our
application and Selenium on our integration machines,
we were able to include Selenium tests into integration
and nightly builds.

4.1. CruiseControl

CruiseControl is a framework a for continuous build

process. It incorporates Ant and source control tools to
allow immediate feedback on the build and test status
of a project’s checked-in code. [6]
Figure 4: A sample Cobertura report

By including Selenium tests in the coverage

analysis for our projects, we had a very good sense of
which code in our projects was not being exercised by
the application’s various tests. For some of our
projects, developers were very interested in writing a
Selenium test in order to exercise code that they didn’t
feel was appropriate to test with JUnit tests. We ran
the Selenium tests against the nightly build, and
included the results in the coverage reports, tracking
the coverage analysis every morning.
By incorporating Selenium into our nightly builds,
with coverage included, they were a natural topic for
our stand-up meetings every day, and everyone on the
team saw value in them. On the other hand, because
our Selenium tests were only running nightly against
Figure 3: A CruiseControl Build Result the newest builds, they didn’t provide the immediate
feedback that they otherwise might have. Below, we
Because our Selenium suites consisted of many discuss some ways that we’d like to leverage their
tests and involved many application-level transactions, value even more.
the entire suite tended to take a very long time to run
after several iterations (for one project, after 12 5. Lessons learned
iterations – 24 weeks – the test suite took four hours to
run). As a result, it was impractical to include the 5.1. Selenium is open-source
Selenium tests as part of our continuous integration
builds. However, we do run the tests as part of a We began using Selenium at version 0.5. Even
nightly build process, and we even run the tests against today, it is only at a 0.6 release, and is under active
instrumented code in order to report on test coverage development. Thus, certain stability and completeness
using Cobertura. issues were to be expected. One of our team members
became an active contributor to the user forums, and
4.2. Cobertura steady monitoring of the user and developer forums for
patches and usable extensions paid off. Our
recommendation is to remember that Selenium isn’t (or

4
at least doesn’t have to be) considered a black box. Suites of Suites, but it was not clear how to get
By understanding, modifying, and extending its Selenium to do the same.
functionality, we were able to make our tests more Another feature we found desirable is the ability to
readable and to handle multiple-step actions with “tag” a test with multiple tags (e.g. “Iteration 3”, “User
one command. Story A”, “Functionality Set 7”) and then be able to
easily see a Suite made up of tests with a given tag. In
5.2. Value to the team and the end users general, members of our team did not run single tests
in an ad hoc manner, but instead waited for the report
The greatest value to our teams was that Selenium from the nightly build. However, if we were able to
tests build up a regression test suite for the web layer more easily identify the appropriate tests, a developer
of the application through the iterations of the agile could run just a small subset of them and get feedback
process. This allowed developers to refactor the front- more quickly than waiting for the nightly build.
end code without fear of breaking the previously
passing acceptance criteria for the application. 5.4. Test first?
While our automated acceptance tests carried great
value, there were instances where a failing Selenium Because Selenium tests are easy to write, a tester or
test gave a false alarm. One of the most fundamental analyst can write the shell of a Selenium test very
problems of GUI testing is that the team can make a quickly without knowing what the implementation will
change that keeps the application completely correct be. Although we hoped that we could code to these
(even “the same” to the end user), but break an tests, the test would seldom turn green after
automated test. A simple example of such a situation development. The reasons for this were usually minor:
is renaming a button. a field wasn’t naturally identified by the name the
Because of this inherent fragility to front-end tester chose, or the test command used needed to be
testing, some team members tended to dismiss a clickAndWait instead of just click, etc. As a result,
breaking Selenium test as “the test’s fault.” This we did not usually require that the developer code to
assessment was correct enough times to make these the test, and our process of writing the test before
tests less valuable as immediate feedback than, say, a development (for its specification value), but getting
unit test. However, any given test did tend to stabilize the test green immediately after development,
after an iteration, i.e. two weeks. The result was that emerged.
everyone did value the tests and the feedback, but we
would try to make sure that a failed test really was “the 5.5. IDs, xpath and performance
code’s fault” before our 10:00 AM standup meeting.
In addition to having value throughout the The first project we used Selenium for was an
development lifecycle, Selenium tests were extremely ASP.NET project, which automatically assigns IDs for
valuable artifacts to hand off to the client at the end of every element on the page. However, when we
iterations. Because our wiki directly linked stories switched to JSP-based applications, the IDs were only
with Automated Acceptance tests, and because it was present when the code specified it.
very easy to demonstrate the Selenium tests really were Selenium supports several different techniques for
testing the application (by running and watching!), the identifying page elements, including names, IDs, xpath
client was able to track the Selenium tests throughout and more. For the most part it was not difficult to find
the project and see that they were all green at the end a unique identification for an element, but occasionally
of the iteration. And the fact that the tests carried so parsing the HTML was tedious. Some tools we found
much regression value gave the whole team comfort valuable for this purpose were the DOM inspector,
that the rapidly-changing application was still stable. XPath checker, and the Selenium Recorder. The
Selenium IDE is even better at this task. However,
5.3. Putting tests into suites frequently these tools do not find the “easiest”
representation of an element (for example, it might find
As mentioned above, our whole Suite of Selenium an xpath expression for an element instead of an id).
Tests took, in some cases, many hours to run. We This would frequently contribute to longer-running
found that keeping all of the tests in one Suite was the tests, which was one of the most significant problems
easiest way to manage the tests, but that frequently we we encountered using Selenium. In general, using
wanted to group the tests in other ways (by story, by xpath expressions tends to make tests take much longer
iteration, by functionality set). FitNesse supports to run, especially in Internet Explorer. So we used IDs

5
or names where they existed, and went out of our way
to add them into our JSP code when possible. 6. Summary
Due to the processing and memory needs of
browsers running Selenium, as our suites grew larger While Selenium is no silver bullet for the problems
and contained more tests, we needed to have a more facing the web application tester, it handles many of
robust environment. For example, one suite running the problems very well and doesn’t add significant new
on a P2 – 600 MHz machine with 512MB of RAM ones. The active and growing community of users and
took over 11 hours to run. Upgrading our test developers indicates that it is filling a need for a
environment to a P4 - 3 GHz machine with 1GB RAM variety of different user types, and widespread
took the time required down under 3 hours. adoption seems imminent. Selenium is certainly worth
evaluating for anyone looking to add a powerful web
5.6. Ability to test all the requirements testing tool to their toolkit.

We have used a number of testing tools in the past

and Selenium is among the best, if not the best, at
being able to perform every browser action that a user
can perform, including such events as onMouseOver
and onKeyPress. In addition, because Selenium allows
users to write their own extensions, it is easy to create
custom actions that do sophisticated manipulations.
There are some JavaScript-restricted actions, such as
downloading or uploading files, that are not supported,
but even these actions have some workarounds for
testing in some browsers.
Selenium doesn’t perform database tests or other
back-end tests, so we do need to do such tests using
FitNesse or other techniques. One recent addition to
the Selenium world, Selenium Remote Control,
addresses some of these issues by allowing the user to
write Selenium tests in other programming languages,
thus leveraging the power of Selenium within more
traditional automated test beds.