product-line verification of safety-critical software

Product-Line Verification of Safety-Critical Software

Robyn Lutz, JPL & ISU

This research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, and at NASA Ames Research Center, under a contract with the National Aeronautics and Space Administration. The work was sponsored by the NASA Office of Safety and Mission Assurance under the Software Assurance Research Program led by the NASA Software IV&V Facility. This activity is managed locally at JPL through the Assurance and Technology Program Office

Technical PresentationNASA OSMA Software Assurance Symposium

September 8-12, 2008

NASA, GRAILElliott, JPL: MSAP

1SAS08_Product_Line_Verif_Safety_Lutz

National Aeronautics and Space Administration

SAS08_Product_Line_Verif_Safety_Lutz 2

Product-Line Verification of Safety-Critical Software Problem

• Industrial experience shows that it is the verifiable conformance of each system to the product-line specifications that makes or breaks the product-line practice

• Verification that the software for each project satisfies its intended product-line constraints is thus essential

• How should we verify that delivered software conforms to the product-line and how do we document that conformance?

• How should we verify that safety-critical software built using product-line assets is safe?



Product-Line Verification of Safety-Critical SoftwareApproach

• Identify candidate suite of product-line verification techniques, focusing on safety

• Tailor as appropriate to NASA needs • Apply, evaluate, and report results in

collaboration with project • Produce examples and data for training and

transfer of techniques to projects



A software product line is defined to be “a set of software-intensive systems sharing a common, managed set of features that satisfy the particular needs of a specific market segment or mission and that are developed from a common set of core assets in a prescribed way” [Clements and Northrop, 2002].

Product-Line Verification of Safety-Critical SoftwareDefinition of product line


• MSAP Project: Worked this year with Multi-Mission System Architecture Platform project at JPL – to identify (1) “negative lessons learned” from industrial use of product lines

(risks & challenges) and (2) relevant mitigation strategies – findings requested by MSAP managers to use in product-line business case– MSAP work slowed when key personnel moved off project

• GRAIL Project: Now working with Gravity Recovery and Interior Lab (Discovery Lunar mission, 2011; twin spacecraft; inheritance from GRACE’s earth-study techniques and from Lockheed Martin’s product-line spacecraft, especially MRO)– to perform trend-tracking for GRAIL of software changes and anomalies

across product line (beginning with MRO): on-going in FY09– to evaluate use of DDP to customize product-line risks/mitigations to

GRAIL: on-going in FY09; collaborative with M. Feather

SAS08_Product_Line_Verif_Safety_Lutz5

Product-Line Verification of Safety-Critical SoftwareOverview


• GRAIL spacecraft is the latest in a LMA product line: high degree of software inheritance with some key differences (e.g., platform, lunar mission)

• Industrial product-line verification techniques help predict quality/defects in a new system in a PL (such as GRAIL) based on similarities/differences with earlier systems in that PL (MRO, Phoenix, etc.)

• Application to GRAIL: what can PL approach to GRAIL tell us about what to expect in terms of quality/defects/concerns for software verification?




Planned Application on GRAIL

• Report : “Survey of Product-Line Verification and Validation Techniques” (2007)– Best practices extracted from reports of industrial experience – Verification challenges & verification enablers identified for

NASA product lines based on experts’ recommendations– List of resources assembled (conferences, industrial and

defense industry experiences, annotated bibliography)• Report: “Tool-Support Survey for Product-Line

Verification and Validation Techniques” (2007)– Commercial and academic tools for product-line development– Configuration management and change management tools for

product lines– Tool-supported testing in product lines (both domain testing and

application testing) – Advice from product-line experts regarding tool support


Product-Line Verification of Safety-Critical Software Quality of data allows generalization of recommendations



Product-Line Verification of Safety-Critical SoftwareAccomplishments this year

• Report : “An Evaluation Report for Three Product-Line Tools ”, Simon Chong (6/08)– Widespread agreement that better tool support for product lines is

needed for project applications– Report evaluated the functionality, usability and performance of 3

candidate product-line tools (2 commercial, 1 academic) identified in Task 2

– FORM (from Pohang University of Science & Technology), Pure::Variants (from PureSystems, GmbH), and Gears (from BigLever)

– Downloaded and experimented with each tool – Emphasis on practical use: e.g., setup, support, user interface,

support for commercial tools, documentation– Recommendation for application plan based on measured

criteria: Gears



Accomplishments this year

• Paper : “Enabling Verifiable Conformance for Product Lines” , R. Lutz– To be presented at 12th Int’l Software Product Line Conference,

Sept. 8-12, 2008– Recommendations for process characteristics that will make it easier

to show that the system being built conforms to the product-line specifications

– Draws on information from four product lines or product-line-like sets of systems (3 JPL, 1 NASA)

• a set of interferometer (spaceborne telescope) software systems with reused architecture, documentation, and code

• a proposed NASA mission consisting of a cluster of microsatellites• a multi-mission hardware and software architecture• a ground data system providing services for multiple spacecraft missions



– Paper describes three challenges for verifying conformance in NASA product lines:

• Many will be safety or mission critical • Many will be long-lived • Many will be highly autonomous

– Recommends process characteristics to handle challenges in the appropriate development phases

• To establish verifiable conformance of safety requirements in the new product need rigorous traceability to the product-line safety requirements

• To maintain conformance for long-lived missions need to accurately capture the requirements delta information

• To verify conformance of highly autonomous product lines need good documentation of the product-line behavior, preferably as a machine-readable model , and use small configuration units


Product-Line Verification of Safety-Critical Software Accomplishments this year


• Collaborated Summer, 2007 with Jairus Hihn (JPL) & Dan Baker (WVU grad student withTim Menzies); Dan developed prototype of Maintenance Effort Estimation Tool

• Served on program committees for Software Product Line Testing workshop, 2007, and Software Product Line Conference, 2008


Product-Line Verification of Safety-Critical Software Synergistic activities


• Rigorous traceability – Key to being able to perform verification on product lines

• Controlling requirements changes in baseline assets– Having this information in the application requirements

specification allows scoping of features and code that will have to be developed beyond the product-line assets

• Sufficient documentation is essential. – Verification that the new system conforms to the product-line

requirements, architecture, and test suite requires good documentation of the new system.

• Access to source code – For thorough testing of the product-line components, need source

code. Code and compilation and linking instructions need to be available in order to adequately verify the code [Pohl]



Findings: Enabling Verification of Inherited Software



• Rigorous traceability – Key to being able to perform verification on product lines [Bergey].

• Capturing the requirements delta– Having this information in the application requirements specification

allows scoping of features and code that will have to be developed beyond the product-line assets [Pohl] .

• Sufficient documentation is essential. – Verification that the new system conforms to the product-line

requirements, architecture, and test suite requires good documentation of the new system.

• Access to source code – For thorough testing of the product-line components, need source

code. Code and compilation and linking instructions need to be available in order to adequately verify the code [Pohl].


Findings: Enabling PL Verification



• Start small– Successful product-line engineering was first applied on a

legacy system “in which there is frequent change occurring at relatively high cost. Such a domain is often an isolatable section of the system where the changes can be encapsulated and where a single group of software developers is responsible for making the changes” [Weiss and Lai].

• Use small configuration units– Rockwell Collin described the tradeoff in their recent experience

building a large product line: “Having more items leads to a finer grained decomposition of functionality, which, in turn, leads to higher levels of reuse and systems that are easier to test and get through airworthiness qualifications. However, having more items also means there are more configuration items to keep track of and maintain, more complexity, and a steeper integration curve” [Clements and Bergey].

Product-Line Verification of Safety-Critical SoftwareFindings: Enabling PL Verification (cont.)



• Build family-specific tools – “Invest in family-specific tools to make the production of family

members rapid using the production process” [Weiss & Lai]. They recommend domain-specific modeling languages and code compilation or composition from the models.

• Demand tool interoperability – Clements and Northrop emphasize that it is the integrated tool

suite rather than the standalone tools that provide the needed development environment. “The interoperability of a chosen set of tools is critical for the automated production of products in a product line from core assets.”

Product-Line Verification of Safety-Critical Software Findings: Tool Support for PL Verification



• Plan verification tasks to reuse test suites – Testing in a product line begins early as the domain-engineering

product-line assets (the reusable components) are built. Then, each time those assets are assembled into a new product-line application, more testing occurs. “During application engineering, assets from this test infrastructure should be reused to speed up testing and thus overall product development” [Knauber and Hettrick].

• Make tools for configuration management (including variability management and change management) a high priority – The payoff for using a product line is safe and efficient reuse.

To enable and control this reuse, product-line developers agree that tool-supported configuration management is required. The challenge is that, “Traditional CM tools are meant to handle variation over time, but they are not good at handling simultaneously supported variations.” [Clements and Northrop]

Product-Line Verification of Safety-Critical SoftwareFindings: Tool Support for PL Verification


Product-Line Verification of Safety-Critical SoftwareTechnical challenges of approach


Categories Challenges Techniques Organizational Ownership of assets unclear

Structure doesn’t match project needs Divide into subdomains Consider separate integration

organization Rigorous two-way traceability

Cultural Reluctance to invest for other projects Feedback from user to owner of assets fails

Create conscious PL culture Learn from experience and problem

reports

Requirements Products needs new features Time can erode product line

Establish strong change board to judge tradeoffs

Capture requirements delta early for new projects

Invest in tool support for change management

Contractor/supplier Low incentive for conformance to product-line specifications

Less control than in-house

Motivate/reward conformance to product-line specifications

Include source code in supplier agreements

Challenges and techniques for verifiable conformance of a product line


• Organizational considerations– External ownership of product line development can make access to

requirements documents, data, code, more difficult• Reluctance to invest for other projects

– Projects that are willing to use other’s products but are reluctant to make investments for other projects can impede product-line development and verification.

• Difficulty controlling the delta– CCB usually first line of defense in deciding whether new features, alternative

design mechanisms, or different coding standards will move systems away from the product-line standard set of verified components.

• Balancing commonality and common sense– Folding changes in one product back into the product line can lead to

degradation of the product line and the erosion of commonality. On the other hand, propagating each change to the product line can make re-verification costs unmanageable.



Challenges for work to be more widely applicable



• TRL 3– “Analytical and experimental critical function and/or

characteristic proof of concept”– MSAP provided limited proof-of-concept of usefulness of

product-line engineering verification techniques for inherited software

– GRAIL application goal for next year: validation in relevant environment (TRL 5)

Product-Line Verification of Safety-Critical Software Technology Readiness Level



• NASA is reusing more product-line, inherited, & reused software assets

• We need better ways to verify compliance of delivered software against the constraints levied on all systems in that product line

• We need better ways to verify that problems with similar, earlier systems won’t recur in our new system

• Work focuses on identifying product-line verification techniques that can help solve problems now for current missions

Product-Line Verification of Safety-Critical SoftwareRelevance to NASA


Product-Line Verification of Safety-Critical Software Planned capability of research

Customize industrial, product-line techniques to verify correctness of a new system’s software inheritance, and apply to NASA systems:

• GRAIL (FY09 focus)• JUNO (same heritage)• Future NASA spacecraft with same LMA product-line heritage • Exploration systems reusing same baseline and inherited

software



product-line verification of safety-critical software

Documents

software product line

productline assets

productline risksmitigations

productline specifications

safetycritical software

line verification techniques

set of software

safety tailor