Generating focused reports with Serenity BDD

Many Serenity users that I work with and talk to have projects with a very large number of tests, spread over many functional areas and many different teams. It is useful to have an overall view of all of the test results, but it is also useful to limit the test results to show only the tests that are relevant to a particular team, or at a particular point in time. For example, an individual Scrum team may want to have a report that includes just the tests related to the features they are working on, or even just those for the current sprint.

Serenity lets us do this in a number of ways:

• During test execution, so that you only run the tests you are interested in,
• Or during the reporting phase, so you take a set of larger test results, for example from a nightly build job, and generate a more specific report.

Filtering tests during test execution is easy: you simply use the tags option. Suppose you mark each test suite with a tag to indicate what scrum team is working on them.

@RunWith(SerenityRunner.class)
@WithTag("scrum:cheetahs")
public class AddNewTodos {
    ...

So to run only the tests for the Cheetahs scrum, you could run the following:

$mvn clean verify -Dtags="scrum:cheetahs"

You will also need to configure the serenity-maven-plugin to use the tags you provide at the command line:

<plugin>
    <groupId>net.serenity-bdd.maven.plugins</groupId>
    <artifactId>serenity-maven-plugin</artifactId>
    <version>${serenity.maven.version}</version>
    <configuration>
        <tags>${tags}</tags>
    </configuration>
    ...

When you run the tests with this configuration, you will get a test report with only the tests related to the Cheetahs scrum team. However, it will still produce a requirements report that contains all of the requirements for the whole project.

We can narrow this down further by using the serenity.exclude.unrelated.requirements.of.type property. You can pass this property in on the command line or place in the serenity.conf file in src/main/resources, or in the serenity.properties file in the project home directory.

The serenity.exclude.unrelated.requirements.of.type property tells Serenity to exclude requirements that are not related to the tests that were executed, or the tests that match the tags that you provide using the tags filter if you provided one.

For example, in the report illustrated above, six capabilities are listed. Not all of these (for example, 'Focus' and 'Money') have any tests associated with them. Others (for example, 'Accessing the applicaton') have tests, but they are not associated with the Cheetah scrum team.

We could make this report more focused by telling Serenity to exclude any capabilities that do not contain tests that are associated with the Cheetahs scrum, by setting the serenity.exclude.unrelated.requirements.of.type property to capabilities:

serenity.exclude.unrelated.requirements.of.type = capabilities

We could then generate a report that documents only the tests and requirements associated with the Cheetahs scrum team like this:

mvn verify -pl acceptance -am -Dtags="scrum:cheetahs"

If the full test suite has already been run, we could use the generated test results and produce a focused report by running the mvn verify phase while skipping the tests:

mvn verify -DskipTests -pl acceptance -am -Dtags="scrum:cheetahs"

This would produce a report that contains only the tests and requirements related to the Cheetahs scrum team:

In this example, the "Completing todos" capability is made up of two features: 'Complete a todo' and 'Toggle all todos'. Now suppose that not all of the functionalities in the 'Toggle all todos' feature were to be implemented by the Cheetahs scrum. In this case, the Capability report below is misleading, as it contains tasks that is not part of the Cheetah team's backlog:

We can fine-tune the requirements reporting further, by telling Serenity to filter not only unrelated capabilities, but also unrelated features:

serenity.exclude.unrelated.requirements.of.type = capabilities, features

Features in the generated report would now only refer to acceptance criteria assigned to the Cheetah's scrum:

The examples here use the default capabilites and features requirements hierarchy. If you have customised the requirements names, you would simply use the customised names instead of the ones shown here.