How to create self-documenting reports in a Selenium

WebDriver automation project

A few months ago, if you asked me what I believe about adding reporting to an automation
project, my answer was "Why do it?"

I thought that reports that show the test execution status are useless because Jenkins has
plugins that generate them. Why put any effort in creating my own if they are available out-of-
the-box?

But recently I realized that there are other types of reports that could be very useful.

All Selenium tests are created for a QA team so its testers can run them attended or
unattended for new builds with bug fixes or new features. Most of the QA people do not know
enough coding so they can just read it to find out what the code does. How can they know the
steps that a Selenium test goes through without looking in the code?

How can they see what the test does, step by step?

The answer is by using self-documenting reports for the Selenium test. This is a report that
shows all steps that a Selenium test goes through.

There is nothing out-of-the-box in Jenkins that does this so I started looking into it.

The first one creates a simple console reporter that writes to the console all steps of a
Selenium test and screenshots taken after each step.

The second does the same thing but with the log4j framework so all information is logged in
an HTML file.

selenium-training-vancouver.com Page 1
The layout of the HTML file is not great which leads to the third solution which is to extend the
TestNG HTML reports.

This third attempt is probably good enough but it requires adding code to each page class
method to be included in the final report. This is code duplication which we should try to stay
away from.

Is there a solution that can generate the self-documenting reports with no code changes?

There is one called the Allure framework.

This framework does it all for you without adding code to the page methods. You just add
annotations to all methods to be added to the final report. The screenshots are added to the
same report through a listener.

All 4 solutions are built with Java, Selenium WebDriver, TestNG and Maven.
All code is structured using page object model.
The page classes use a base page class for common page information.
The test class uses a base test class for the driver variable and test fixtures.

You can see the first solution below. It will give you an idea of the process followed for the
other 3.

The last solution is different from the first 3 because it relies not on my own code but on
another library. The last solution, the one based on the Allure framework, is the best.

The first solution is free for you. The other 3 are not.
If you want all of them (and the full code for each), please make a PAYPAL transfer of USD 12
to [email protected] including your email address. An Amazon gift card works as well.

selenium-training-vancouver.com Page 2
All solutions implement a simple test case for the Vancouver Public Library site:

1. open the site (http://www.vpl.ca)

2. on the home page, type a keyword in the search box
3. execute the search by clicking the search button
4. on the results page, click the next page to go to page 2
5. on the results page, click the next page to go to page 3
6. on the results page, click the next page to go to page 4
7. on the results page, click the previous page to go to page 3
8. assert that the current page number is 3

The goal for each solution is to generate a report such as

- open the site

- type a keyword in the search box
- execute the search
- click the next page
- click the next page
- click the next page
- click the previous page
- assert that the current page number is 3

selenium-training-vancouver.com Page 3
 1. Create a console reporter

selenium-training-vancouver.com Page 4
The Selenium test is very straightforward:

package tests;

import pages.HomePage;
import pages.ResultsPage;
import static org.testng.Assert.assertEquals;
import org.testng.annotations.Test;

public class SeleniumTest extends BaseTest {

@Test
public void pagingTest() {

HomePage home = new HomePage(driver);

home.open();

home.typeKeyword("java");
ResultsPage results = homePage.executeSearch();

results = results.goToNextPage();
results = results.goToNextPage();
results = results.goToNextPage();
results = results.goToPreviousPage();

assertEquals(results.pageNumber(), 3, "is page number equal to 3!");

The pagingTest() is part of the SeleniumTest class which extends the BaseTest class.

BaseTest class has the driver object and the setUp()/tearDown() methods to be executed
before and after each Selenium test:

selenium-training-vancouver.com Page 5
package tests;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;

public class BaseTest {

public WebDriver driver;

@BeforeMethod
public void setUp() {
System.setProperty("webdriver.chrome.driver",
"C:/Selenium/BrowserDrivers/chromedriver.exe");
driver = new ChromeDriver();
}

@AfterMethod
public void tearDown() {
driver.quit();
}

The Selenium test relies on 2 page classes: HomePage and ResultsPage.

HomePage.java has the following public methods:

- open(): opens the site and checks that the home url is correct

- typeKeyword(): types a keyword in the search box

- executeSearch(): executes the search by clicking the search button; returns an object of
ResultsPage class

selenium-training-vancouver.com Page 6
package pages;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;

public final class HomePage extends BasePage {

private static final String URL = "http://www.vpl.ca/";

private static final By SEARCH_BOX_ID = By.id("edit-search");

private static final By SEARCH_BUTTON_ID = By.id("edit-submit");

public HomePage(WebDriver driver) {

super(driver);
}

public void open() {

driver().get(URL);

if (urlContains(URL) == false)
throw new IllegalStateException("home page is not displayed!");
}

public void typeKeyword(String keyword) {

WebElement searchBox = driver().findElement(SEARCH_BOX_ID);
searchBox.sendKeys(keyword);
}

public ResultsPage executeSearch() {

WebElement searchButton = driver().findElement(SEARCH_BUTTON_ID);
searchButton.click();

return new ResultsPage(driver());

}

selenium-training-vancouver.com Page 7
ResultsPage.java has the following public methods:

- goToNextPage(): goes to next page by clicking the NextPage link and waiting until the
results count changes for the next page; returns a new ResultsPage object

- goToPreviousPage(): goes to previous page by clicking the PreviousPage link and waiting
until the results count changes for the previous page; returns a new ResultsPage object

- pageNumber(): returns the number of the current page

package pages;

import static org.openqa.selenium.support.ui.ExpectedConditions.*;

import org.apache.commons.lang3.StringUtils;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;

public final class ResultsPage extends BasePage {

private static final String URL = "vpl.bibliocommons.com";

private static final By NEXT_XPATH =

By.xpath("(//a[@data-key = 'next-btn'])[1]");

private static final By PREVIOUS_XPATH =

By.xpath("(//a[@data-key = 'previous-btn'])[1]");

private static final By RESULTS_COUNT_XPATH =

By.xpath("(//span[@data-key = 'pagination-text'])[1]");

public ResultsPage(WebDriver driver) {

super(driver);

if (urlContains(URL) == false)
throw new IllegalStateException("results page not displayed!");
}

selenium-training-vancouver.com Page 8
 public int pageNumber() {
int result = 0;

if (url().indexOf("page=") == -1)
result = 1; //no page parameter in url for page 1
else {
String resultText = StringUtils.substringBetween(url(),
"page=", "&query=");

result = Integer.parseInt(resultText);
}

return result;
}

public ResultsPage goToNextPage() {

int n = pageNumber();

clickNext();

String fromToText = String.format("%d to %d", from(n + 1), to(n + 1));

waitUntilResultsCountIncludes(fromToText);

return new ResultsPage(driver());

}

public ResultsPage goToPreviousPage() {

int n = pageNumber();

clickPrevious();

String fromToText = String.format("%d to %d", from(n - 1), to(n - 1));

waitUntilResultsCountIncludes(fromToText);

return new ResultsPage(driver());

}

selenium-training-vancouver.com Page 9
 private void clickNext() {
WebElement nextPageLink = explicitWait().until(
elementToBeClickable(NEXT_XPATH));

nextPageLink.click();
}

private void clickPrevious() {

WebElement previousPageLink = explicitWait().until(
elementToBeClickable(PREVIOUS_XPATH));

previousPageLink.click();
}

private void waitUntilResultsCountIncludes(String keyword) {

explicitWait().until(
textToBePresentInElementLocated(
RESULTS_COUNT_XPATH, keyword));
}

private int from(int n) {

return (n - 1) * 10 + 1;
}

private int to(int n) {

return n * 10;
}

selenium-training-vancouver.com Page 10
Both HomePage and ResultsPage classes inherit from BasePage.java:

package pages;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

import reporter.ConsoleReporter;
import reporter.CustomReporter;

public abstract class BasePage {

private final WebDriver driver;

private final WebDriverWait wait;

public BasePage(WebDriver driver) {

this.driver = driver;
this.wait = new WebDriverWait(driver, 10);
}

protected WebDriver driver() {

return driver;
}

protected WebDriverWait explicitWait() {

return wait;
}

protected boolean urlContains(String keyword) {

return wait.until(ExpectedConditions.urlContains(keyword));
}

protected String url() {

return driver.getCurrentUrl();
}

selenium-training-vancouver.com Page 11
BasePage provides access to the driver and an explicit wait objects and also to page url and
title.

It is created as abstract so objects cannot be created for it. Its methods are protected so they
can only be used from classes that inherit from BasePage.

We know everything needed about the core components of the project.

What about the reporter?

Before creating the reporter class, let's have an interface for it:

package reporter;

public interface CustomReporter {

public void log(String... text);

public void logWithScreenshot(String text);

The CustomReporter interface defines 2 methods for logging a text and logging a text and a
screenshot. We create the interface first so any of the next reporter classes implements it. This
way, we can use any of the reporter implementations interchangeably.

selenium-training-vancouver.com Page 12
The first reporter class is very simple:

package reporter;

import org.openqa.selenium.WebDriver;

public class ConsoleReporter implements CustomReporter {

private WebDriver driver;

public ConsoleReporter(WebDriver driver) {

this.driver = driver;
}

public ConsoleReporter() {
}

@Override
public void log(String... texts) {
for (String text: texts)
System.out.println(text);
}

@Override
public void logWithScreenshot(String text) {

Screenshot screenshot = new Screenshot(driver);

screenshot.save();

log("", "method: " + text, "screenshot: " + screenshot.path());

}

selenium-training-vancouver.com Page 13
The class implements the 2 methods from its interface:

- log(): displays in the console all String values that it gets as parameter

- logWithScreenshot(): takes a screenshot of the current page and displays in the console
the String parameter and the screenshot path

logWithScreenshot() relies on the Screenshot.java class for taking screenshots and saving
them. Each screenshot uses the current local time as file name:

package reporter;

import java.io.File;
import java.io.IOException;
import java.time.LocalTime;

import org.apache.commons.io.FileUtils;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;

public class Screenshot {

private String fileName;

private WebDriver driver;

public Screenshot(WebDriver driver) {

this.driver = driver;

this.fileName = LocalTime.now().toString()
.replace("-", "_").replace(":", "_").replace(".", "_")+ ".png";
}

selenium-training-vancouver.com Page 14
 public void save() {
try {
File file = ((TakesScreenshot)driver).
getScreenshotAs(OutputType.FILE);

FileUtils.copyFile(file, new File(path()));

} catch (IOException e) {
throw new IllegalStateException(
"exception while taking screenshot - " + e.getMessage());
}
}

public String path() {

return System.getProperty("user.dir") +
"/target/screenshots/" + fileName;
}
}

The ConsoleReporter class creates the self-documenting report.

This is done in a few steps.

1. Add method to BasePage class for giving access to a reporter object to the page classes:

package pages;

//same as before

import reporter.ConsoleReporter;
import reporter.CustomReporter;

public abstract class BasePage {

// same as before

protected CustomReporter reporter() {

return new ConsoleReporter(driver);
}
}

selenium-training-vancouver.com Page 15
Notice that the reporter() method returns a ConsoleReporter object as a CustomReporter
variable. This is possible becase ConsoleReporter implements the CustomReporter interface.

2. Add reporting to the page method classes.

First to HomePage.java

package pages;

//same as before

public final class HomePage extends BasePage {

//same as before

public void open() {

driver().get(URL);
if (urlContains(URL) == false)
throw new IllegalStateException("home page is not displayed!");

reporter().logWithScreenshot("open home page");

}

public void typeKeyword(String keyword) {

WebElement searchBox = driver().findElement(SEARCH_BOX_ID);
searchBox.sendKeys(keyword);

reporter().logWithScreenshot("type search keyword");

}

public ResultsPage executeSearch() {

WebElement searchButton = driver().findElement(SEARCH_BUTTON_ID);
searchButton.click();

reporter().logWithScreenshot("execute search");

return new ResultsPage(driver());

}
}

selenium-training-vancouver.com Page 16
and then to ResultsPage.java

package pages;

//same as before

public final class ResultsPage extends BasePage {

//same as before

public ResultsPage goToNextPage() {

int n = pageNumber();

clickNext();

String fromToText = String.format("%d to %d", from(n + 1), to(n + 1));

waitUntilResultsCountIncludes(fromToText);

reporter().logWithScreenshot("go to next page");

return new ResultsPage(driver());

}

public ResultsPage goToPreviousPage() {

int n = pageNumber();

clickPrevious();

String fromToText = String.format("%d to %d", from(n - 1), to(n - 1));

waitUntilResultsCountIncludes(fromToText);

reporter().logWithScreenshot("go to previous page");

return new ResultsPage(driver());

}

//same as before

selenium-training-vancouver.com Page 17
These changes add information to the report about the page class methods.

3. Create an assertion class that logs information to report and executes assertion:

package assertions;

import reporter.ConsoleReporter;

public class Assertions {

public static void assertEquals(int a, int b, String assertionMessage) {
new ConsoleReporter().log("ASSERTION - " + assertionMessage);
org.testng.Assert.assertEquals(a, b, assertionMessage);
}
}

4. In the test class, use assertEquals() from the custom assertion class:

package tests;

//same as before

import static assertions.Assertions.*;

public class SeleniumTest extends BaseTest {

@Test
public void pagingTest() {
HomePage home = new HomePage(driver);
home.open();

home.typeKeyword("java");
ResultsPage results = home.executeSearch();

results = results.goToNextPage();
results = results.goToNextPage();
results = results.goToNextPage();
results = results.goToPreviousPage();

assertEquals(results.pageNumber(), 3, "is page number equal to 3!");

}
}

selenium-training-vancouver.com Page 18
Running the test again shows in the console the self-documenting report:

-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running TestSuite
Configuring TestNG with: TestNG652Configurator
Starting ChromeDriver 2.40.565498 on port 30825

method: open home page

screenshot: C:\Alex\workspace\Basic-Report/target/screenshots/12_33_03_140.png

method: type search keyword

screenshot: C:\Alex\workspace\Basic-Report/target/screenshots/12_33_03_665.png

method: execute search

screenshot: C:\Alex\workspace\Basic-Report/target/screenshots/12_33_09_809.png

method: go to next page

screenshot: C:\Alex\workspace\Basic-Report/target/screenshots/12_33_11_372.png

method: go to next page

screenshot: C:\Alex\workspace\Basic-Report/target/screenshots/12_33_12_920.png

method: go to next page

screenshot: C:\Alex\workspace\Basic-Report/target/screenshots/12_33_14_497.png

method: go to previous page

screenshot: C:\Alex\workspace\Basic-Report/target/screenshots/12_33_15_546.png

ASSERTION - is page number equal to 3!

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.839 sec - in

TestSuite

This first solution works for simple cases but what if you want to log the information to a file?

Solution 2 shows how to do this with the log4j framework.

selenium-training-vancouver.com Page 19