Parallel Test Execution With Cucumber and Serenity BDD

Parallel execution is a great way to speed up almost any test suite, but for a long time, running tests in parallel with Cucumber has been challenging.

But there is great news on this front: Cucumber 6 supports parallel execution of features in JUnit natively.

And if you are using Serenity BDD with Maven, this means it is now very easy to run features in parallel. First of all, you need a single test runner class to run all of your feature files, like this:

@RunWith(CucumberWithSerenity.class)
@CucumberOptions(
    plugin = {"pretty"},
    features = "src/test/resources/features"
)
public class AcceptanceTests {}

Next, you need to configure the maven-failsafe-plugin with the 'parallel' configuration property set to methods.

You then define the number of threads by either setting useUnlimitedThreads to true, or by defining the threadCount property. Check out the maven-failsafe-plugin documentation for more details on this.

A full maven-failsafe-plugin configuration for parallel execution looks like this:

<plugin>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>3.0.0-M5</version>
    <configuration>
        <includes>
            <include>**/*AcceptanceTests.java</include>
        </includes>
        <systemPropertyVariables>
            <webdriver.base.url>${webdriver.base.url}</webdriver.base.url>
        </systemPropertyVariables>
        <parallel>methods</parallel>
        <useUnlimitedThreads>true</useUnlimitedThreads>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>integration-test</goal>
                <goal>verify</goal>
            </goals>
        </execution>
    </executions>
</plugin>

The downside is that Cucumber with JUnit only supports executing features in parallel, not scenarios.

Nevertheless, you can still get some good speed increases from this approach. Here are some benchmarks I got from running a simple test suite of 40 scenarios (including one scenario outline with 6 rows) across 15 feature files:

• Running the test suite sequencially: 1 minute 56 seconds
• Running the test suite in parallel on a single machine with unlimited threads in headless mode: 40 seconds
• Runing the test suite sequentially on Browserstack: 6 minutes 37 seconds
• Running the test suite in parallel on BrowserStack with 5 threads: 2 minute 5 seconds

I have a fast machine, so the local results may not be indicative of all machines. But running the tests in parallel resulted in approxiamtely a three-fold increase in speed.

On Browserstack, with 5 parallel executors we also get a three-fold increase in speed. But the speed increase on Browserstack and similar platforms tends to be more linear than on a local machine; with unlimited local threads I have maxed out performance on my local machine, but with 10 parallel executors on BrowserStack, we would get a 5-6 times speed increase (until we are limited by the number of feature files that can be executed in parallel).

If you want to take your Serenity BDD learning further, make sure you keep an eye on the Serenity Dojo Training Library, where you can already get a comprehensive overview of Serenity BDD fundamentals in the Effective UI Test Automation With Serenity BDD and Selenium course, and there will be a new course focusing on getting the best out of Cucumber and Serenity BDD very soon.

Enjoy!