Today I want to take a look at this paper (PDF) entitled „Contract Driven Development = Test Driven Development – Writing Test Cases„. Like the paper on SDD I found this essay on my research for a synergetic development approach that takes both DbC and TDD into account.
The paper was written by A. Leitner et al. at the ETH Zurich in 2007, when important steps towards automatic testing had been already made. One of the co-authors of the paper is Bertrand Meyer, the inventor of the Eiffel programming language which can be seen as cradle of DbC. Hence it’s not surprising that the paper and the described tool are based on Eiffel.
Content of the paper
The paper describes Contract Driven Development (CDD) as new approach to lower the effort for writing tests with intensive use of contracts. This approach is based on a mechanism to extract test cases completely automatically from failure-producing runs of a component, where contracts act as test oracle.
In the introduction the paper takes a short look at current developments towards automatic testing and shows the drawbacks that come with many tools. Automatic testing tools don’t know the semantics of a program and the insights a programmer has and thus those tools cannot distinguish between meaningful and meaningless inputs. CDD wants to overcome this drawback. It relies on the observation that developers are often writing implicit test cases when manually running a program to determine the correct behavior of their code. But many developers don’t extract those test cases as automatic unit tests and thus those tests cannot be run in a reproducible way.
The paper presents CDD as method that captures those implicit test cases automatically and extracts them into explicit tests. One important limitation is that it only captures test cases which produce a failure e.g. through a broken contract or other exceptions. With this approach the resulting test suite contains tests for mistakes made by the developers in the past. The developer builds up this test suite by running the application with corresponding input values. The authors state that the test suite would have similar properties to a suite as result from TDD when you write contracts before the implementation of a feature. You can find my personal thoughts on this below.
The paper describes a mechanism by which CDD observes program executions and detects the last uninfected state when a failure occurs. In this case CDD takes a snapshot of this state and a new test case for the failing component is created automatically. The snapshot serves as starting context for the test case by which the failure can be reproduced.
With this process the developer has to provide the inputs triggering a failure only once. Then CDD jumps in, saves the values that lead to the failure and extracts a test case to reproduce the failure. The developer can go forward with his implementation and can choose to fix the bug later on while the failure is saved as reproducible test case. Contracts can help a lot in this process since they can act as oracle for expected behavior and can express assertions that go beyond some technical exceptions. With this the benefit of CDD is mainly driven by a extensive utilization of contracts.
The authors finish with a conclusion of their work:
This article explains the fundamentals of the Contract Driven Development approach. A tool autonomously observes the developer while he is working on a program and extracts test cases from failures either provoked by the developer (in the spirit of test driven development) or by mistake (leading to a regression test). The approach is novel in that complete test cases are extracted not only from the information provided by the system under test, but also from non-permanent clues given by the programmer during development.
My thoughts
Personally I feel a little ambivalent about the CDD approach. First I like the idea of making implicit test cases which are run by the developer explicit by extracting them automatically. This includes the developer’s knowledge about the context of an application into the generated test cases and thus goes beyond the technical aspect of automatic testing. It takes some work on writing test cases from the developer’s shoulder, but it doesn’t have as many advantages as one may think…
Expected functional behavior can be defined by contracts (postconditions, invariants) which act as test oracle for CDD. But it’s a limitation as well. Important characteristics of code are difficult to define with contracts while they can be easily defined with explicit unit tests (aspect of expressiveness). Thus CDD is limited in its expressiveness as well.
Furthermore CDD extracts tests for failing runs only. But what about the other side of the medal? For successful runs no test case will be generated automatically which leads to the problem that if anything in the code changes there is no possibility to check the successful tests without running them manually again. Thus CDD is not suitable for complete regression testing.
Last but not least the authors state that a test suite created with CDD is very similar to TDD (beside other parallels to TDD). Moreover the title of the paper gets it to the simple formula „CDD = TDD – Writing Test Cases“. Here I strongly disagree! CDD and TDD have very different properties. TDD takes things like code design and specification into account, is well-suited for documentation purposes and creates a real regression testing suite. None of those aspects is in scope of CDD, so TDD goes far beyond CDD.
At the end I think CDD is an interesting approach, but the benefits are bought dearly and eaten up by the drawbacks. One of the most serious drawbacks for me (personally) is the design aspect. If you rely on CDD you are not driving your API design to be clear and lucid. There will be no unit tests for which you want to have loosely coupled components and furthermore CDD will everytime run hard against your concrete infrastructure.
To make a final advice: The only scenario where I would use CDD is for automatic test generation when running the software in production. If something fails in production with CDD you as developer could get immediate feedback about what went wrong and you could immediately reproduce the failure by running the extracted test case. This would be a nice support for bugfixing and it would be really valuable. But for testing purposes in the development process I don’t see a chance for CDD.
