Catching up with Allure. Part 3: report customization

A good test report is a report that gives us a clear understanding of a current state of a product quality. For example, we can see how many tests were failed, the weakest components in a project, test coverage. That’s fine!

A perfect report is a report that gives us much more – information about errors and failures; easy navigation, categorizing and filtering; additional logs and media for debugging; statistical information about an environment, a test run, tests behavior…
This list can be extended but… lots of you have already recognized Allure, right? If you haven’t, then you have to read this post because I’m going to show you how to extend the default naked Allure report and add a lot of useful info. This post is a continuation of the “Catching up with Allure” series (post #1, post #2), and I recommend you to familiarize with the previous posts if you haven’t done that yet.

 

Challenge for this post:

To extend the default Allure report with the following cool features: test environment information, detailed test attributes (steps, human readable descriptions, severity), links to issues and test cases in a bug-tracking system and TMS, screenshot attachments for failed tests.

Environment

Except for JSON files, Allure can handle one more file which contains information about our test environment – environment.xml. A typical environment.xml looks like this:

The logic is pretty simple: we can specify as many parameters as we want, just by putting each of them inside the <parameter> </parameter> tags and providing name, key and value child nodes.

Ideally, this file should be present in the allure-results folder before we run Allure Command Line tool. However, as you remember, the allure-results folder is inside the target folder which is cleaned before each test run. This fact makes our life more difficult but it can be overcome by the maven-resources-plugin. The code below added to the <build> section in the pom.xml copies environment.xml from the src/test/resources folder (where I store it) and put to the allure-results folder while building the project:

After the report generating, those 3 parameters will appear in the Environment chapter (we’ll see it later).

Steps

When we run a test case manually, we should have a list of steps, i.e. a sequence of actions. When we work with test code, each step may assume one line or several lines of code. From the Allure point of view, a step is a method (not always a test method) which is annotated by the @Step annotation. If you remember, the example project we are working with, it has different pages (AmazonMainPage, ItemPage, CartPage, etc.) with their methods inside. Let these public methods be steps! As an example, let’s have a look at the TodaysDeals page class (the other page classes have the same steps logic):

Test attributes & information

After we figured out with steps, we still have a lot of options for customizing the report. As you may notice from the previous article, the generated Allure report had the Behaviours tab. It’s designed for grouping our tests by different features and user stories and it can be pretty useful in case of BDD tests or user-acceptance scenarios.

In order to utilize this tab in the report, we should annotate a test method by the @Feature and @Story annotations respectively. For example, @Feature(“Searching”) and @Story(“Search by titles”). It is possible to have several annotations of each kind for the same test.

Another important parameter of each test is severity. Allure framework provides 5 levels of severity: BLOCKER, CRITICAL, NORMAL, MINOR, TRIVIAL. All we need is to add one more annotation for a test method like this: @Severity(SeverityLevel.CRITICAL). The specified severity will be reflected in the Graphs tab and on the test case page.

Summarizing the above-mentioned, let’s see how it looks like with a real test:

Please notice that I added the description option to the @Test annotation. Allure is smart enough to pick it up and display in the report also.

Links to a TMS and a bug-tracker

One of the most useful features of Allure framework is to add direct links to a test management system (where we store associated test cases) and a bug-tracking system (if a test has associated bugs). If we have a look at the Allure documentation, we will see that the format of adding those links is the following:

@Issue(“777”)
@TmsLink(“12345”)

However, what do these numbers mean? Apparently, they are the IDs of the associated bug and test case in the appropriate systems. But they are not links, aren’t they? In order to convert them to URLs, we have to specify Allure properties which are responsible for TMS and BTS patterns. We can do that through the Maven’s pom.xml:

The IDs will be placed instead of {} symbols, and we will have full-fledged URLs in the report.

Attachments

There are 2 different kinds of attachments in Allure framework – screenshots and text (log files). We can add attachments in 2 ways:

  1. by adding and calling a method annotated with the @Attachment. Such method should return either String (in case of a text attachment) or byte[] (in case of screenshot);
  2. by calling one of the overloaded static Allure.addAttachment() methods according to a type of attachment.

In order to understand the mechanism of attachments, I am going to provide a screenshot attachment for each failed test. For obtaining a screenshot, I’m going to utilize the Screenshots.takeScreenShotAsFile() method which comes together with the Selenide framework. And finally, the magic of decision making for failed tests only will be implemented through the TestNG listeners.

Let’s have a look at the BaseListener class that contains the above-mentioned logic:

As you can see, after a test is run, I check if it was successful or not. If it wasn’t, I call the attachScreenshot() method which converts a taken screenshot to a byte array. That’s what Allure needs! If you want to attach logs or any other string information, you can use the same logic described above and pass text information to the Allure.addAttachment() method.

Finally, don’t forget to add this BaseListener class to the listeners of your test class:

When everything is done, we can run mvn clean test and then allure serve commands (see the previous post for details). The generated report looks more informative and useful:

Like this post?

Subscribe to updates from my blog, if you don't want to miss more interesting future posts and materials

Please check your email and confirm subscription

Pin It on Pinterest

Share This