Automated Testing: Software testing is the most generally used technique for verifying and validating software. While testing may not provide the same rigor as model checking, it is far more scalable and usually does not require a significant tool infrastructure; thus its initial cost and learning curve is low. However, for complex systems these benefits are more than offset by the fact that it is hard to develop the “right” test cases that provide sufficient coverage. This difficulty often reduces coverage despite use of many test cases. The consequence is that testing is a very labor intensive process that typically accounts for about half the total cost of software development and maintenance while yielding poor reliability due to inadequate coverage (Bezier 1990)].
Automating testing not only decreases the cost of verifying and validating software but also increases its reliability. A recent report by the National Institute of Standards and Technology estimates that software failures currently cost the US economy about $60 billion every year, and that improvements in software testing infrastructure might save one-third of this cost (NITT 2002). Testing, and in particular test case generation, lends itself to automation and has been the focus of much research attention; it has also been adopted in industry (Parasoft; T-VEC; Drusinsky 2000; Grieskamp et al. 2002). We distinguish between two main approaches to automatic test case generation: black boxand white box. In the black-box approach, test cases are generated based on a specification of the inputs, while in the white-box approach, test cases are generated based on the analysis of the software under test or of a model of this software.
The quality of a test suite is measured in terms of test coverage, such as structural code coverage, MC/DC, path coverage, and requirements coverage. In addition to program inputs, test cases also need to specify the expected output. This can be done by using testing oracles that encode the functional specifications of the software under test as assertions in the code, post-conditions, etc.