Verification versus validation; critical systems

Verification checks that the software conforms to its specification ("Are we building the product right?"), while validation checks that the software meets the customer's real needs ("Are we building the right product?"), as defined by Boehm.

Verification versus validation

Similarities: both aim to find defects and build confidence that the system is fit for purpose. Both together form V&V, which runs through the whole life cycle and can use a large part of the development budget. The level of confidence needed depends on the software's function (how critical it is), user expectations and the marketing environment (time-to-market pressure). V&V uses two complementary approaches: static techniques (inspections, reviews, static analysis) that need no execution, and dynamic techniques (testing) that run the program.

Critical system

A critical system is a system whose failure can cause serious economic loss, physical damage, or threat to human life or the environment. Its most important property is dependability (availability, reliability, safety, security). Types:

• Safety-critical: failure can injure or kill people, e.g. a chemical plant control system or an insulin pump.
• Mission-critical: failure stops an important goal, e.g. a spacecraft navigation system.
• Business-critical: failure causes high business cost, e.g. a bank's customer accounting system.

Critical systems need much more V&V effort than ordinary systems.

How partitioning augments V&V (V-model)

In the V-model, the system is partitioned step by step into sub-systems, modules and units. Each level of partitioning has its own test plan, prepared during development and used during testing. Partitioning helps V&V because:

1. Each partition can be verified and tested separately and in parallel by different teams.
2. A fault is localised to a small partition, so it is easier to find and fix.
3. Test plans are made early, so testing is planned along with design.
4. Integration is incremental: units, then sub-systems, then the whole system.

Requirements spec ......... Acceptance test
   \                          /
  System spec ..... System integration test
     \                      /
   System design .. Sub-system integration test
        \                 /
      Detailed design    /
           \            /
     Module and unit code and test

  .... = test plan made on the left, used on the right

Example (ATM system): the system is partitioned into card-reader, account-database, cash-dispenser and user-interface sub-systems. Each module is unit tested; the "withdraw cash" sub-system (UI + account + dispenser) is tested against the sub-system integration test plan; the whole ATM is tested against the system integration test plan; finally the bank runs acceptance tests. At the input level, the withdrawal amount is also partitioned into equivalence classes (below minimum, valid multiple of 500, above daily limit) so that a few test cases cover the whole input range.