Cem Kaner compiled this list of “properties” that good tests have. Kaner’s list was originally compiled with respect to testing organizations, and he didn’t necessarily have agile testing in mind when he compiled this list. But I think that these properties are relevant to agile testing.
This part bears repeating: these aren’t necessarily properties of JUnit tests, or even automated tests. These are properties of tests, even if they’re carried out by people.
- Power: A test should reveal problems when they exist.
- Valid: When the test states that there’s a problem, it is a genuine problem.
- Value: Tests should reveal problems that the client wants to know about.
- Credible: The test should be trusted to actually test the things that people expect it to test.
- Representativeof events that are likely to be encountered by the user. (Not necessarily “commonly”, but likely). Hacking attempts, for example, might be likely, but not common.
- Non-redundant: The test reproduce the work of another test
- Motivating: Your client will want to fix the problem exposed by this test.
- Performable: Designing a test that can’t actually be performed is a bit useless.
- Maintainable: If the end product changes, it’s important to be able to update the tests
- Repeatable: It should be easy and inexpensive to execute the test again and/or reuse it as appropriate
- Insightful: A good test should reveal things about our assumptions. Kaner uses the word “pop” to describe this property. It’s a tricky property to describe in one word.
- Coverage: The test exercises the code in a way that isn’t already handled by another test.
- Easy to evaluate: it should be relatively easy to determine whether the test has passed or failed.
- Supports troubleshooting: If the test fails, it should help the programmer identify the problem.
- Appropriately complex
- Accountable: Can you explain why the test exists, and can you demonstrate that it was executed?
- Opportunity Cost: The test obviates the need to perform some other work.