What’s new in Serenity 2

Serenity BDD is a library that helps you write better automated acceptance criteria, and produce better living documentation and executable specifications. In this article, we will look at the key new features in Serenity BDD 2.0.

Introduction

Serenity BDD is a test automation and reporting library that is used in conjunction with other testing libraries such as Cucumber, JUnit and JBehave. It is designed to help test automation engineers write higher quality, more maintainance automated acceptance tests, and also to produce high quality living documentation. Many organisations still rely on heavy manual approval processes for each new release. Serenity 2.0 includes a number of improvements and new features intended to help organisations streamline this process by providing clear, business-friendly evidence and documentation about the features that will go into a release.

The sample project

To illustrate the sort of living documentation that we can produce with Serenity, we will look at a simple sample application. This Serenity/Cucumber project tests a compliance application for an imaginary bank. Banks are required to have controls and checks in place to prevent their services being used for illegal activities such as money laundering or terrorism. They also have rules to determine how much risk a new customer presents for the bank. The application we are testing is designed to enforce these rules for our imaginary bank.

Serenity 2.0 works best with Cucumber: while JBehave is still supported, many of the richer living documentation features are only possible with Cucumber.

The basic project structure looks like this:

.
├── build.gradle
├── pom.xml
├── serenity.properties
├── src
│   └── test
│       ├── java                                                            
│       │   └── acceptancetests
│       │       ├── AcceptanceTestSuite.java
│       │       .
│       │       .
│       │       .
│       └── resources
│           ├── assets                                                      
│           │   └── customer-due-diligence.png
│           ├── data
│           │   └── transactions.csv
│           ├── features                                                    
│           │   ├── approval_testing
│           │   │   └── transaction_monitoring_approval_tests.feature
│           │   ├── customer_acceptance_policy
│           │   │   ├── business_activities.feature
│           │   │   ├── country_risk_ratings.feature
│           │   │   ├── customer_risk_profiles.feature
│           │   │   ├── enhanced_due_diligence.feature
│           │   │   ├── non-face-to-face-customers.feature
│           │   │   ├── pep.feature
│           │   │   ├── readme.md
│           │   │   └── standard_due_diligence.feature
│           │   ├── customer_identification
│           │   │   ├── identifying_a_new_customer.feature
│           │   │   └── readme.md
│           │   ├── ongoing_monitoring
│           │   │   ├── monitoring_new_customers.feature
│           │   │   ├── monitoring_suspicious_cash_transactions.feature
│           │   │   └── readme.md
│           │   └── readme.md

	Test runners and step definitions
	Images available to be used in the living documentation reports
	Cucumber feature files, organised in a requirements hierarchy

We will be coming back to some of these elements later on. You can see the actual sample project here. But for now, lets see what happens when we run these tests and generate the Serenity living documentation.

A new dashboard

The biggest change in Serenity 2.0 involves the reports themselves, which have been overhauled and upgraded. The home page (shown below) presents a summary of the test results, with a table where the result counts are broken down by outcome:

The Test Results tabs gives an overview of all of the tests executed:

As you can see, this includes both automated tests. and any that you have flagged as @Manual as well.

The Functional Coverage Overview

Further down on the summary page, you will see the Functional Coverage Overview section, which gives readers a succinct summary of the state of the application.

This section includes a breakdown by feature, including features that have yet to be implemented. For example, the Enhanced Due Diligence feature has been scoped and planned, but work hasn’t yet started. We just place a feature file with the title (and ideally a short description), and it will appear in the feature coverage list:

@Know_Your_Customer_Rules:Enhanced_Due_Diligence
Feature: Enhanced Due Diligence

Customers that pose higher money laundering or terrorist financing risks present increased exposure to banks; due diligence policies, procedures, and processes should be enhanced as a result. Higher-risk customers and their transactions should be reviewed more closely at account opening and more frequently throughout the term of their relationship with the bank.

By default this section lists the application features, and this works fine for many projects. But you can also configure it to present information that is of particular interest to stakeholders. For example, in this project stakeholders will always want to know which compliance rules have been implemented so far. The same rule might be implemented by more than one feature, and some features describe scenarios that are relevant to more than one rule.

To give stakeholders the information they want, we use tags. We can tell Serenity to group the functional coverage results displayed on the home page by tag type, rather than by Cucumber feature. Let’s see how this works.

In Serenity, tags have two parts; a type and a name, separated by a colon. So the following feature has a "Business_Activites" tag of type "Know_Your_Customer_Rules":

@Know_Your_Customer_Rules:Business_Activities
Feature: Business Activities
  ...

Country Risks, another Know Your Customer Rules requirement, has a similar tag.

@Know_Your_Customer_Rules:Country_Risk
Feature: Country risk ratings
...

If we tell Serenity to group coverage results by Know_Your_Customer_Rules, it will show a line for each tag of type Know_Your_Customer_Rules in the functional coverage overview, as we saw above. To make this happen, we added the following configuration in the serenity.properties file:

report.tagtypes=Know_Your_Customer_Rules

The Requirements Tab

The report home page shows the test results from a QA perspective - what passed, what failed, and what functional areas have been tested. The Requirements tab, on the other hand, is about Executable Specifications. A lot of the power of the Serenity reporting comes from here. You can think of this tab as a blend of business-readable specifications with real-time test results and feedback:

The text section at the top comes from the readme.md file you may have seen in the src/test/features directory. This gives an introduction to the specifications, an overview of the application as a whole. It is similar to what you might see in the introduction of a traditional requirements documment. You can see an extract here:

## Customer Due Diligence

Supervisors around the world are increasingly recognising the importance of ensuring that their banks have adequate controls and procedures in place so that they know the customers with whom they are dealing. Adequate due diligence on new and existing customers is a key part of these controls. Without this due diligence, banks can become subject to reputational, operational, legal and concentration risks, which can result in significant financial cost.

Any readme.md file placed in a features subfolder will be used as an introduction to the requirements at this level. For example, if you select the Customer Acceptance Policies entry on the requirements tree above, you will see the following summary at the top of the page.

Displaying features

The lowest-level nodes in the requirements tree represent features. If you click on a feature, you will get a summary of the feature and the scenarios that illustrate its behaviour. For example, the feature file for the Identifying a new customer feature looks likke this:

Feature: Identifying a new customer
  The bank needs to identify any person or entity that maintains an account with the bank or those on whose behalf an account is maintained

  Scenario: Identifying a new customer living in the UK
    Given Joe is a new customer
    And Joe lives in the UK
    When Joe provides one piece of identity and one proof of address
    Then his identify should be established

  Scenario: Identifying a new customer not living in the UK
    Given Joe is a new customer
    And Joe does not live in the UK
    When Joe provides one piece of identity and proof of addresses over the past 3 years
    Then his identify should be established

In the reports, it appears like this:

This page is designed to be the executable specifications view, giving you an overview of what the feature does, illustrated with concrete examples in the form of scenarios. Each scenario corresponds to an acceptance criteria: you can click on any one of the scenarios to see the detailed test execution results:

Multi-layered reporting

The feature file format is an excellent way to describe and document a specific requirement. But a short description and a sequence of scenarios is not always rich enough for stakeholders to fully engage with. In Behaviour Driven Development, we engage with stakeholders, even high level ones, to describe and specify their needs in a common language shared between the business people and the technical team.

For example, our bank evaluates the potential risk associated with new business customers in part by considering their business activites and associations. A casino is considered more risky than a vet clinic. We could explain these rules simply enough in a few paragraphs of text and a few example tables, like the ones shown below:

The scenarios, which are represented in our feature file, go into more detail. Our high level business stakeholders might not be interested in the details, but product owners and QA certainly would:

In Serenity 2.0, we can combine both these views, to create expresive, multi-layered executable specifications. By including special tags such as {Examples} and {Scenario} in the description at the top of the feature file, you can embed the example tables, or the Given/When/Then narrative, of selected scenarios in the overall description. You can use either the name of a scenario, or the name of an actual examples table, to identify the table you want. For example, the following line would embed an examples table with the label "High Risk".

{Examples} High Risk

A more complete extract of the feature file used to generate this report is shown below:

@Know_Your_Customer_Rules:Business_Activities
Feature: Business Activities

  Some business activities are considered more risk-prone than others, and certain customers and entities may pose specific risks.

  The business activity risk factor for an industry is determined by the potential risk they present to the bank. For example:

  {Examples!WithTitle} High Risk

  Business activity risk for a customer with an estabilished business (in activity for 2 years or more) depends on the business activity risk factor:

  {Examples} The business activity risk factor for an established business

  The Business Activity Risk factor for a customer is determined by the industry risk factor, how long the business has been in activity. A business with less than 2 years activity is considered high risk:

  {Examples} Businesses established less than 2 years ago are considered higher risk

  Scenario Outline: The business activity risk factor for an industry

  The industry risk factor is a value from 0 to 10.

    When a customer works in <Business Category>
    Then their base business activity risk factor should be <Risk Factor>

    Examples: High Risk
      | Business Category        | Risk Factor |
      | Casino                   | 10          |
      | Precious Metals Exchange | 9           |
      | Currency Exchange        | 9           |
      | Jewellery Store          | 8           |
    ...

Test Evidence

One of the purposes of the Serenity reports is to streamline the release process in large organisations. Many organisations require structured evidence of testing, including screenshots, database extracts, sample report contents and so on. A simple test report is not enough, and many organisations waste a lot of time preparing such documents before a long phase of UAT testing can repeat the whole exercise.

The Serenity reports aim to help provide the evidence stakeholders need (either to trust that a release has been well tested, or for audit purposes), so that any manual UAT testing phases can be eliminated or reduced to a bare minimum.

To add evidence to a report, you simple need to call the Serenity.recordReportData() method inside your test code, like in the example below:

Serenity.recordReportData()
        .withTitle("Reported Transactions")
        .asEvidence()
        .fromFile(Paths.get("src/test/resources/data/transactions.csv"));

This will result in an Evidence section in the reports, where readers can view or download the recorded evidence:

Conclusion

This article was a short introduction to the Serenity BDD 2.0 reporting capabilities. To learn more, take a look at the Serenity Book. You can also see a live example of the reports discussed in this article in the sample report here.