October 11, 2016

Space Age Testing

The NASA approach that could save millions on testing.

There is nowhere that can truly say they test more than at the National Aeronautics and Space Administration or NASA. The possibility of failure at any point in a mission, from launchpad to landing could mean the loss of millions of dollars, critical scientific data, and worst of all – human life. As such it is unsurprising that NASA has one of the strictest and rigorous testing styles in the world.

NASA is no stranger to disaster, and its history is pock-marked with the very public failures of its otherwise illustrious career. Worst of all, all of these failures were preventable. A classic example was in January 1967 during the preparation for the first mission of the Apollo program – AS-2014 or Apollo 1. During a training exercise in the Command Module, the cabin caught fire. The crew attempted to open the door, but could not open the hatch due to the internal pressure in the cabin being so high, and the door having to open into the highly pressurised cabin. The cabin was a high-pressure, pure oxygen cabin atmosphere, and the fire quickly consumed the cabin, killing all three crew members.

At the Apollo 204 Accident Review Board it was found that five different major factors had combined to cause the fire. Each of these factors would have been easily preventable if they had been considered before the test. It was a major blow to the American space program, and NASA suffered heavy media and political backlash as a result.

The preventability of these failures can be seen in NASA’s approach to testing. Thirty per cent of all development effort is devoted to system and acceptance testing. They realise just how important it is to get the system not only working, but as close to perfect as possible.

The verification and testing cycle in NASA consists of six areas: code reading, unit testing, integration testing, build/release testing, system testing and acceptance testing. Whilst most of these aspects may be familiar to the reader, one may stand out – code reading. Every line of code in a NASA system must be peer reviewed.

The NASA Software Engineer Laboratory (SEL) Manager’s Handbook for Software Development describes their Code Reading procedure as the following:

Code reading is a systematic procedure for reading and understanding the operation of a program. Studies have shown that code reading detects more errors at a lower cost than either functional testing or structural testing alone. In the experience of the SEL, the most powerful verification combination is human inspection followed by functional testing.


This is what makes the NASA test system really stand out – they check each line of code before they even begin functional testing. The purpose of this is to verify that the code of a program is correct with respect to its intended function.

As this is performed at the lowest level possible (the code itself), more errors can be spotted in the code before they have the chance to reach the functional test phase. This is absolutely vital in a high cost, high risk industry such as aeronautics and spaceflight. By reducing the number of defects during the verification phase, they reduce the amount of time that will be required for fixing defects during the testing phase – and potentially preventing the interruption of testing due to showstoppers and blockers.

During the Code Reading verification phase, code is read by an experienced developer, who is not the original programmer. Two code readers are assigned to each unit to be tested, as it was found that only one quarter of total errors found in code reading are found by both readers independently.

All code is read - the only exception is when a team is utilising the formal, more mathematical approach of the Cleanroom software engineering methodology – with each reader reviewing the code independently. This allows for better peer review, as there is no conferring until the code has been read. As such, it allows for code readers to get better at their job, as having an error pointed out – that they missed – can help them spot a similar error later.

The reader ensures the code is consistent with the specifications and conforms to standards and conventions. This is a form of Static Testing, thus allowing the system to be checked to ensure that it is conforming to the required specifications along each step of the development process.

The question is, why perform such a gruelling task? Well, for one, it improves the quality of the code that developers write. A system of personal pride evolves, whereby many who have worked for NASA claim that they endeavoured to get their coding right first time – no defects. In many ways, they couldn’t make mistakes – people’s lives quite literally depended on their code being correct.

I believe that code reading could be a part of the modern software development cycle. It improves developer familiarity with the system, it encourages better coding practices, reduces defect occurrence and thus the amount of time spent in testing waiting for defects to be fixed. Whilst this would increase the initial development time, it has the benefit of reducing the amount of time required for functional testing, thus making more time for other processes.

Using a code reading based development system can also help with the fixing of defects. It reduces the risk of incorrect syntax or parameters occurring during testing, and as such can point the developer towards other aspect of the code.

However, we will have to wait and see whether or not a developer wants to sit down and read another developer’s code for hours at a time.


By Colin Campbell, Test Analyst, Edge Testing