Running parallel tests in Cucumber with Serenity BDD

Running parallel tests in Cucumber with Serenity BDD has always been tricky. Until now.

People often ask me how to run their Cucumber/Serenity BDD test suites faster. Fast feedback is key to a smooth build pipeline and to an effective test suite. A quick-running test suite means we can know when something goes wrong, and get it fixed, sooner. It gives us more confidence in a new release, sooner. And it helps us get features in front of our users, sooner.

Unfortunately, in many large projects, our acceptance test suites are anything but quick. Acceptance tests, particularly if they are web based, or have complicated setup requirements, can be painfully slow to run.

A common work-around to this issue is to run your acceptance tests in parallel. For web tests, this can be scaled further by using Selenium Grid or SAAS testing platforms such as Saucelabs and BrowserStack. When done well, running your tests in parallel can result in substantial time savings.

Parallel tests with Cucumber in Java

But if you are using Cucumber, things are not so simple. Cucumber is a great tool for teams defining and writing executable specifications, and Serenity BDD adds killer reporting and well structured glue code to the mix. But Cucumber tests in Java have always been tricky to run in parallel. The traditional approach is to write a large number of separate JUnit test runners (for example one per feature file, or per group of feature files), and run these in parallel. But this results in a lot of classes to maintain, especially for larger code bases.

One solution to this problem is the Cucumber JVM Parallel Plugin. This plugin generates test runners for you dynamically when it runs your Cucumber tests, saving you the trouble of creating them yourself. And while it is designed to be used with vanilla Cucumber, it is quite easy to set up to work with Serenity BDD. Let's see how.

The Cucumber JVM Parallel Plugin with Serenity BDD

The Cucumber JVM Parallel Plugin works nicely with the Cucumber conventions that Serenity BDD uses. The plugin will look for feature files underneath the src/test/resources directory and create runners for each feature. But you will need to tweak your pom.xml file a little.

Setting up the maven-failsafe-plugin

The Cucumber JVM Parallel Plugin works with the maven-failsafe-plugin, so your Cucumber scenarios will be executed when you run mvn verify. By default, it generates test runners with theIT suffix, so make sure your maven-failsafe-plugin configuration will look for files matching this format.

The maven-failsafe-plugin is also where you configure your parallel execution. I found that the parallel parameter was the simplest to use. In the code below, we run 4 parallel threads per processor core:

            <plugin>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>2.20</version>
                <configuration>
                    <systemPropertyVariables>
                      <webdriver.base.url>${webdriver.base.url}</webdriver.base.url>
                    </systemPropertyVariables>
                    <parallel>classes</parallel>
                    <threadCount>4</threadCount>
                    <perCoreThreadCount>true</perCoreThreadCount>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

This will run your feature runner classes in parallel. Next, you need to configure the cucumber-jvm-parallel-plugin to generate these runners. The configuration will look something like this:

           <plugin>
                <groupId>com.github.temyers</groupId>
                <artifactId>cucumber-jvm-parallel-plugin</artifactId>
                <version>4.2.0</version>
                <executions>
                    <execution>
                        <id>generateRunners</id>
                        <phase>generate-test-sources</phase>
                        <goals>
                            <goal>generateRunners</goal>
                        </goals>
                        <configuration>
                            <!-- Mandatory -->
                            <!-- List of package names to scan for glue code. -->
                            <glue>
                                <package>planner.cucumber</package>
                            </glue>
                            <parallelScheme>FEATURE</parallelScheme>
                            <customVmTemplate>src/test/resources/cucumber-serenity-runner.vm</customVmTemplate>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

There are a bunch of configuration options on the web site, but the most important are the parallelScheme and the customVmTemplate.

• The parallelScheme must be FEATURE. The SCENARIO option will generate a test runner for each scenario. This would result in a better distribution of test execution. Unfortunately, each scenario in scenario outlines are also executed in parallel, which messes with Serenity's reporting.

• The glue package should be the root package where your step definitions are located.
• The default configuration will use the default JUnit Cucumber runner. For Serenity, we need to use the CucumberWithSerenity class instead. To do this, simply define a custom template to use to generate the test runners. Create a file called cucumber-serenity-runner.vm and place it in your src/test/resources` directory. The file should look like this:

#parse("/array.java.vm")
#if ($packageName)
package $packageName;

#end##
import org.junit.runner.RunWith;

import cucumber.api.CucumberOptions;
import net.serenitybdd.cucumber.CucumberWithSerenity;

@RunWith(CucumberWithSerenity.class)
@CucumberOptions(
strict = $strict,
features = {"$featureFile"},
plugin = #stringArray($plugins),
monochrome = $monochrome,
#if(!$featureFile.contains(".feature:") && $tags)
tags = #stringArray($tags),
#end
glue = #stringArray($glue))
public class $className {
}

And that's it! Now when you run mvn verify, your Cucumber features will be executed in parallel.

Related courses and workshops

If you want to learn more about Serenity BDD, take a look at the Serenity Dojo online training programme. The Serenity Dojo Online Training Programme is an innovative and exciting way of learning good BDD and Test Automation practices, including Cucumber, Selenium WebDriver, and Serenity BDD, and also the advanced Java development skills you need to be a great test automation specialist.