Top-down integration problems; regression testing

Problems with top-down integration

In top-down integration, modules are integrated by moving downward through the control hierarchy, starting with the main control module. Lower-level modules that are not ready are replaced by stubs (dummy modules that simulate them). Integration can be depth-first or breadth-first.

              M1   (main, tested first)
            /  |  \
          M2   M3   M4     <- stubs at first
         /  \        |
       M5    M6      M7    <- integrated last

Testing M1 first needs stubs S2, S3 and S4. Each stub is then replaced by the real module, one at a time, and tests are rerun. The following problems may be encountered:

1. Stubs are costly and hard to write. A stub must return realistic values for the calling module to be tested properly. For complex modules (e.g. a database or a calculation engine) a realistic stub is almost a full module, which adds large overhead.
2. Upper levels cannot be fully tested without lower-level processing. Often the real data that flows upward is produced only by low-level modules. If stubs return simple values, upper-level logic is not tested adequately. The tester then has three poor choices:
   • delay many tests until stubs are replaced by real modules, which loses the link between a test and a specific module and makes error location hard;
   • write more capable stubs, which costs much effort;
   • change to bottom-up integration for that part.
3. Low-level modules are tested late. Critical utility modules (input/output, device drivers, maths routines) are at the bottom and are integrated last. Errors in them are found late, when they are expensive to fix.
4. Test output is hard to observe. Upper-level modules often do not produce visible results directly; outputs may be produced only by lower modules. Extra code may be needed to observe results.
5. Hard to create test conditions. Some situations are produced only by low-level modules (such as a hardware error), so they are hard to trigger from the top.
6. Temptation to defer testing. Because of the effort above, testers may postpone tests, and some combinations are never tested.
7. Parallel work is limited. Lower modules may be ready before upper modules, but they cannot be integrated until the upper layers exist.

The advantage that balances these problems is that major control and decision points are checked early, and a working skeleton of the system is available early for demonstration. In practice a sandwich (mixed) approach is often used.

Regression testing

Regression testing is re-executing a subset of previously run tests after a change to the software, to make sure the change has not introduced new errors or brought back old ones in parts that used to work.

Every time a new module is added during integration, or a bug is fixed or a feature added during maintenance, the program changes. New data flow paths, new I/O and new control logic may break existing functions. Regression testing catches these side effects.

The regression test suite usually contains three classes of tests (Pressman):

1. A representative sample of tests that exercise all software functions.
2. Additional tests that focus on functions likely to be affected by the change.
3. Tests that focus on the components that were changed.

Approaches: retest all (safe but costly), selective regression (only affected tests), and prioritised regression (most important tests first). Because tests are run again and again, they are usually automated with tools such as JUnit or Selenium in a continuous integration pipeline.

Example: in an online shop, the discount calculation is changed. Regression testing re-runs the cart total, tax, payment and invoice tests to make sure they still give correct results.