Software maintenance and fixing common bugs

How bugs arise in a (distributed) software system

Maintenance is the work done after release to correct faults, adapt to new environments, and improve the product. In distributed software, bugs come from:

1. Requirement and design errors that testing did not find.
2. Coding errors: logic, boundary, memory and data-type mistakes.
3. Concurrency and timing problems: race conditions, deadlocks and synchronisation errors between nodes.
4. Communication faults: message loss, delay, ordering, network failures, and protocol or data-format mismatch.
5. Integration and version mismatch between modules or nodes, and configuration or environment differences (OS, hardware, library).
6. Changes made during maintenance that introduce new bugs (regression), and poor documentation.
7. Unexpected loads and inputs in the field, and resource exhaustion.

Developing a good corrective action

1. Report and record the problem in a bug-tracking system, with the steps to reproduce it, logs, version, severity and priority.
2. Reproduce and isolate the problem using logs, debuggers and tracing across nodes; identify the module and the root cause (not just the symptom).
3. Analyse the impact on other modules and nodes before changing anything.
4. Design and implement the fix with the smallest possible change, following the coding standards, and then review the code.
5. Test: test the fix, run the regression tests, and test in a staging environment that duplicates the distributed set-up.
6. Controlled release: use version control and configuration management; deploy as a patch (rolling update) with a roll-back plan.
7. Update the documentation and inform the users.
8. Preventive actions: analyse the cause to find similar defects, improve tests and processes, add monitoring and logging, and carry out periodic maintenance.