Maxim

SQA Standard that signed in Blood

Before I'll continue with post about QA standard and its specifications from IEE standards, I want to present some sad story:

Space Shuttle Challenger Disaster

The Space Shuttle Challenger disaster occurred in the United States, over the Atlantic Ocean, off the coast of central Florida, at 11:39 am. EST (16:39 UTC) on January 28, 1986. The Space Shuttle Challenger disintegrated 73 seconds into its flight after an O-ring seal in its right solid rocket booster (SRB) failed at liftoff. The seal failure caused a breach in the SRB joint it filled, allowing a flare to reach the outside and impinge upon the adjacent attachment hardware and external fuel tank. The SRB breach flare led to the separation of the right-hand SRB and the structural failure of the external tank. Aerodynamic forces promptly broke up the orbiter. The shuttle was destroyed and all seven crew members were killed. The crew compartment and many other vehicle fragments were eventually recovered from the ocean floor after a lengthy search and recovery operation.

The disaster resulted in a 32-month hiatus in the shuttle program and the formation of the Rogers Commission, a special commission appointed by United States President Ronald Reagan to investigate the accident. The Rogers Commission found that NASA's organizational culture and decision-making processes had been a key contributing factor to the accident. NASA managers had known that contractor Morton Thiokol's design of the SRB's contained a potentially catastrophic flaw in the O-rings since 1977, but they failed to address it properly. They also ignored warnings from engineers about the dangers of launching on such a cold day and had failed to adequately report these technical concerns to their superiors. The Rogers Commission offered NASA nine recommendations that were to be implemented before shuttle flights resumed.

- O Rings (are made from special polymer) placed between sections of rocket booster are distorted by cold and ice (see picture) and one of them lead to fuel leak on right booster.

As a result of Rogers Commission investigation NASA implemented IV&V Quality Assurance Standard (see its description below).

NASA's Independent Verification and Validation Facility (IV&V) was established in 1993 and is located in Fairmont, West Virginia. The IV&V Facility was founded under the NASA Office of Safety and Mission Assurance in the aftermath of the Space Shuttle Challenger disaster. The IV&V Facility houses over 150 full-time employees.

Its purpose is to provide a higher level of safety and efficiency for "mission-critical software". Using a rigorous software engineering approach both ground and in-flight software systems are independently evaluated during planning, coding and testing in an effort to circumvent mission failure, the loss of equipment or personnel, and to meet time and cost constraints. All phases of software development are examined: concept, requirements, design, coding, testing and operation. Human operated software, robotic software, instrument software and data analysis software may all be assigned to IV&V by NASA.

The IV&V Facility's efforts have contributed to NASA's improved safety record since the Facility's inception.

Many safety regulations were signed in blood, many accidents with human (and financial) casualties lead to strong safety regulations, also in Software Quality Assurance.

IV&V - Independent Verification and Validation (link to PDF file)

IEEE Standard for Software Verification and Validation

IEEE Std 1012-1998

• Validation is concerned with checking that the software meets the user's needs, and Verification is concerned with checking that the system is well engineered.

• This is sometimes expressed as "Are we building the right system?" and "Are we building the system right?".

• Independent Verification and Validation (IV&V) is Verification and Validation activities performed by team that is not under the control of the team that is developing the software.

• Personnel: Requires the effort to utilize personnel who are not involved in the development of the software.
“Outsiders” approach : This personnel required to formulate its own understanding of the problem and how the proposed system is solving the problem. Technical independence (“fresh viewpoint”) is an important method to detect subtle errors overlooked by those too close to the solution.

• Software tools: Requires to develop set of test and analysis tools separate from the developer’s tools.
Sharing of tools is allowable for computer support environments (e.g., compilers, assemblers, utilities) or for system simulations where an independent version would be too costly.
For shared tools, requires qualification tests on tools to ensure that the common tools do not contain errors that may mask errors in the software being analyzed and tested.

Disaster example from European Space Agency:

“Ariane-5” rocket blew up 37 seconds after lift-off

Cost: 500,000,000 $

Reason: An attempt was made to convert a 64-bit integer into a 16-bit unsigned integer

The ADA exception handler was omitted

The on-board computers crashed, and so did the rocket

Computations on the inertial reference system can stop 9 seconds
before lift-off

But if there is a subsequent hold in the countdown, it takes several
hours to reset the inertial reference system

Computations therefore continue 50 seconds into the flight

Ten years before, it was mathematically proved that overflow was
impossible - on the “Ariane-4”

Because of performance constraints, conversions that could not lead to
overflow were left unprotected

The software was used, unchanged and untested, on the “Ariane-5”

However, the assumptions for the “Ariane-4” did not hold for the “Ariane-5”

Lesson:

Software developed in one context needs to be retested when integrated
into another context, no V&V processes are integrated as well.

  • Independent Team: Method requires that the responsibility for the IV&V effort be vested in an organization separate from the development and program management organizations.

• Independent Testing Plan: Managerial independence also means that the IV&V effort independently selects the segments of the software and system to analyze and test, chooses the IV&V techniques, defines the schedule of IV&V activities, and selects the specific technical issues and problems to act upon.

• JIT Tests: The IV&V effort provides its findings in a timely fashion simultaneously to both the development and program management organizations.

• Freedom of Action: The IV&V effort must be allowed to submit to program management the IV&V results, anomalies, and findings without any restrictions (e.g., without requiring prior approval from the development group) or adverse pressures, direct or indirect, from the development group.

Managerial Independence (?)

This requires that control of the IV&V budget be vested in an organization independent of the development organization.

- This independence prevents situations where the IV&V effort cannot complete its analysis or test or deliver timely results because funds have been diverted or adverse financial pressures or influences have been exerted.

Defined by four kinds:

• Embodies all three independence parameters.

• Responsibility is vested in an organization that is separate from the development organization.

• Uses a close working relationship with the development organization to ensure that IV&V findings and recommendations are integrated rapidly back into the development process.

• Performed by one organization (e.g., supplier) and the development is performed by a separate organization (i.e., another vendor).

• Generally required for software integrity level 4 (i.e., loss of life, loss of mission, significant social or financial loss) through regulations and standards imposed on the system development.

Modified IV&V

• Used in many large programs where the system prime integrator is selected to manage the entire system development including the IV&V.

• Prime integrator selects organizations to assist in the development of the system and to perform the IV&V.

• Acquirer reduces its own acquisition time by passing this responsibility to the prime integrator. Since the prime integrator performs all or some of the development, the managerial independence is compromised by having the IV&V effort report to the prime integrator.

• Technical independence is preserved since the IV&V effort formulates an unbiased opinion of the system solution and uses an independent staff to perform the IV&V.

• Financial independence is preserved since a separate budget is set aside for the IV&V effort. Can appropriate for systems with software integrity level 3 (i.e., an important mission and purpose).

Internal IV&V

• Exists when the developer conducts the IV&V with personnel from within its own organization, although not necessarily those personnel involved directly in the development effort.

• Technical, managerial, and financial independence are compromised:

Technical independence is compromised because the IV&V analysis and test is vulnerable to overlooking errors by using the same assumptions or development environment that masked the error from the developers.

Managerial independence is compromised because the internal IV&V effort uses the same common tools and corporate analysis procedures as the development group. Peer pressure from the development group may adversely influence how aggressively the software is analyzed and tested by the IV&V effort.

Financial independence is compromised because the development group controls the IV&V budget. IV&V funds, resources, and schedules may be reduced as development pressures and needs redirect the IV&V funds into solving development problems.

• The benefit of an internal IV&V effort is access to staff who know the system and its software. This form of IV&V is used when the degree of independence is not explicitly stated and the benefits of preexisting staff knowledge outweigh the benefits of objectivity.

Embedded V&V

• Similar to Internal IV&V in that it uses personnel from the development organization who should preferably not be involved directly in the development effort.

• Focused on ensuring compliance with the development procedures and processes.

• Works side by side with the development organization and attends the same inspections, walkthroughs, and reviews as the development staff (i.e., compromise of technical independence).

• Not tasked specifically to independently assess the original solution or conduct independent tests (i.e., compromise of managerial independence).

• Allows rapid feedback of V&V results into the development process but compromises the technical, managerial, and financial independence of the V&V organization.

• Financial independence is compromised because the IV&V staff resource assignments are controlled by the development group.

Forms of Independence: