Living Documentation: it’s not just about test reports
What is Living documentation
Living Documentation comes from the world of Behaviour Driven Development, and is closely related to the idea of Executable Specifications. But what do we mean by "living documentation"?
Living documentation is, as the name suggests, both documentation, and living.
-
It is documentation. It describes how an application works, and what business rules affect it. And it does so in a way that normal users can understand. New team members can use living documentation to understand what a product does, and how it works. Development teams can pass it on to a maintenance team when the application goes live. Auditors can review the living documentation, to see that the relevant regulations have been respected.
-
It is living, because it is generated by the automated test suite, each time the application is built. So it is always up-to-date.
Different teams use different tools to build living documentation. Although Living Documentation is a key aspect of Behaviour-Driven Development, traditional BDD tools such as Cucumber and JBehave only provide very basic reporting capabilities. Other ATDD tools, such as Concordion, put more emphasis on producing good living documentation reports. Serenity BDD is an example of a library that provides very strong living documentation support to existing BDD tools such as Cucumber and JBehave, or even acceptance tests written in plain JUnit.
You can see a simple living documentation report, produced by Serenity BDD, in the illustration below:
A living documentation report
Living Documentation is not Test Reporting
Living Documentation should not be confused with conventional test reporting. There are a number of important differences between the two. And understanding these differences makes it easier to write good quality living documentation.
The key differences fall into four different categories:
- Timing,
- Authors,
- Audience and Language, and
- Purpose and scope
Timing
Conventional test reports happen late in the development process. Testers start working on test cases as soon as the user stories are finalised. But work on the test reports themselves comes much later, once the features are delivered.
For a team practicing BDD, the work on Living Documentation starts much earlier. Teams practicing BDD work with product owners early to define acceptance criteria. These acceptance criteria will lead to automated acceptance tests and for living documentation.
Authors
Test Reports are the responsibility of QA. It is the testers who design the report structure and generate the test reports.
Living Documentation is written collaboratively by many different actors. BAs, developers, testers and product owners all play a role in the BDD process. They all help to express the business needs in a way that can turned into automated acceptance tests.
Audience and Language
Testers are also the primary consumers of test reports. In many projects, other team members will only ever see test report summaries or overviews.
Living Documentation is for the whole team. Pending scenarios describes what a feature should do before it is implemented. And Passing scenarios show that a delivered feature behaves as expected. In addition, they document what the feature does, and how it works. This is why we write them in business terms, in a way that is easy to understand for non-testers.
Purpose and scope
Produced by testers and for testers, a test report naturally has a very strong testing focus. The emphasis is primarily on knowing whether a test passes or fails, and, at a higher level, knowing what proportion of tests pass or fail.
Living Documentation is more like a very detailed illustrated user manual. The focus is about describing what the application does, in business terms.
Suppose you are implementing a user registration feature. A test report might check whether the right countries appear in a country drop-down list. A living documentation scenario would be more interested in describing the user registration process as a whole. It might show how this process works for different types of users. It might also illustrate the business constraints that limit the choice of countries.
These differences can be summed up in the following table:
| Test Reports | Living Documentation |
|---|---|
| Produced after development is done | Written before development starts |
| Written by testers | Written collaboratively by the whole team |
| Written mostly for testers | For the whole team and beyond |
| Reports on passing and failing tests | Describes worked examples of functionality |
Living Documentation and other types of tests
The aim of testing is to provide confidence in the quality of an application. Living Documentation works together with unit and integration testing to do this. In Living Documentation, we focus on key examples of positive and negative scenarios. The more exhaustive testing is left to the unit and integration tests.
Of course there are some exceptions to this rule. For example in regulated environments, living documentation will often be more exhaustive and detailed. And if you are working with a legacy application with little unit testing, you may need more comprehensive high level tests.
Conclusion
As development practices evolve, we need to evolve our thinking about test reporting. Traditional test reports, and traditional test reporting tools, are a legacy of Waterfall. They force us to think in terms of tests that happen only post-implementation. Unlike living documentation, they do not encourage collaboration and communication within the team. Instead they enourage silos They do not inject quality into the beginning of the development process. And they do little to streamline throughput.
