validating requirements

Validating Requirements

Determining Completeness and Correctness of Requirements Using

the System Reference Model

IV&V Workshop16 September 2009

Overview• Validation Purpose and Definitions• A Correct and Complete SRM, and the Three

Questions• SRM Correlation Mapping• Analysis of Correlation Results• Correct, Complete, Incorrect, and Incomplete

Examples• Best practices, lessons learned, and challenges

IV&V’s Validation Definition• The process of evaluating artifacts to ensure that the

right behaviors have been defined in the artifacts. • The right behaviors are those that adequately describe

– what the system is supposed to do– what the system is not supposed to do, and – what the system is supposed to do under adverse conditions.

• Validation ensures that the software system performs to the user’s needs under operational conditions.– Contains the desired capabilities to accomplish the mission

goals – Does not contain unspecified limitation that impedes the

capabilities

Validation Goal• To ensure that

– The right behaviors have been defined• Adequately describe

– What the system is supposed to do– What the system is not supposed to do– What the system is supposed to do under adverse

conditions• Correct and Complete

– The requirement specifications are of high quality• Unambiguous, Consistent, and Verifiable

Correct (IVV 09-1)• Applicable requirement(s) meet all or part of the goals

and behaviors of the system– Note: not all requirements can be evaluated in isolation; it may require a

set of requirements to be evaluated together in order to determine that a particular goal or behavior is being met).

• The requirements are an accurate elaboration of the defined objectives or goals– e.g., the use of temporal modal operators like “next”, “until”, “always”,

and “eventually”, are appropriately used to reflect the desired behavior• The requirements adequately refine the higher-level

requirements• Design or implementation-specific information is

specified as constraints to the behaviors captured in the requirements

Complete (IVV 09-1)• All the needed information to completely specify a

desired behavior is identified (i.e., all preconditions, postconditions, and invariants are specified for the described behavior).

• Threads of behavior are represented by more than one requirement, versus one compound requirement that attempts to capture the entire thread (i.e., that each requirement specifies only one “thing”).

• The use of conjunctions (e.g., “and”, “or”) are restricted to preconditions, postconditions, and invariants.

How?“This goal is achieved through the development and application of a system reference model (SRM) that will include a formal specification. The SRM can then be used to show (e.g., validate) that the right system behaviors are specified and the associated requirements are unambiguous, correct, complete, consistent, and verifiable. The SRM can also be used to validate (or develop) a test design that will demonstrate that the software products meet the specification and the operational need.” – IVV 09-1

The System Reference Model• Includes sets of Modeling Artifacts

– Use cases– Activity Diagrams, Sequences Diagrams– Statecharts– Domain Models (Class Diagrams, Communication Diagrams)– Statechart Assertions– JUnit Test Cases

• A concise description of the IV&V team’s understanding of the problem – Analysis tool– Communication tool

• Captures expected system behaviors– 3 Questions

What’s In the SRM?Validation and Verification involves answering the following

three questions:1. Will the System do what it is supposed to do?2. Will the System not do what it is not supposed to do?3. Will the System maintain operations under adverse conditions?

Note, in order to answer these questions, we must first have an independent understanding of:– What the system is supposed to do– What the system is not supposed to do– What the system is supposed to do to maintain operations under

adverse conditionsThis information can be found within different areas of our

model.

SRM Product Dependencies

System Goals

System Goals

Constraints, Actors, and Environment

Text Use Cases

Use Case Example

What the system is supposed to do

All parts within the Main Success Scenarios

describe the actions that must take place to

accomplish the goal(s).

Adverse ConditionsExtension Scenarios show

how the system should react to adverse conditions to get back on the success

path or transition to safe mode.

Activity Diagram Example

Flight System

Goal: Flight System precesses and damps nutation to point the High Gain Antenna at earth for communication and GRAV science

Precondition: Engineering Instruments are calibrated

Flight SystemTurn on IMUs

Turn on Precession catalyst bed heaters

Turn on X-band Transmitters

Passive Nutation Damping

Return Errors

<<Main Success Scenario>>Precess to Earth Point

<<Extension Scenarios>>Precess to Earth Point Fails

Select High Gain Antenna

Select Forward Low Gain Antenna

Pulse RCS Thrusters

Turn off IMU

Turn off cat bed heaters

Use Other Thrusters

[IMU Does not turn on or malfunctions]

[Cat bed heaters

do not turn on]

[turning to

earth point][not turning to earth point]

[thrusters do not fire]

What the system is supposed to

doAll parts within the

Main Success Scenarios describe

the actions that must take place to

accomplish the goal(s).

Adverse ConditionsExtension Scenarios show

how the system should react to adverse conditions to get back on the success

path or transition to safe mode.

Sequence Diagrams Example

What the system is supposed to do

The interactions of the Sequence

Diagram describe the steps involved to accomplish the

goal(s).

18

Adverse ConditionsA sequence diagram

can also be developed to show

how the system should react to

adverse condition.

SRM Validation Scenarios

getEstimatedStateVector()

estimateStateVector()

AssertTrue()

AssertFalse()

estimateStateVector()

AssertTrue()

getEstimatedStateVector()

getEstimatedStateVector()

public void testScenario2() {st.getEstimatedStateVector();st.estimateStateVector();assertTrue (st.isSuccess());st.estimateStateVector();assertTrue (st.isSuccess());st.getEstimatedStateVector();st.getEstimatedStateVector();assertFalse (st.isSuccess()); }

What the system is NOT supposed to do

This information can be found in our Assertions

and the Validation scenarios we create to test

against them.

Validation WBS (IVV 09-1) 1.0 Validation

1.1 Obtain/Develop a System Reference Model (SRM)

1.2 Validate System Requirements1.3 Validate Test Design

Validate System Requirements• For each level of system decomposition, we need to

determine– Sufficiency of the requirements

• Is there a corresponding requirement for every SRM behavior and Statechart assertion at that level?

– Quality of the requirements• Assess the quality of each requirement that has a

corresponding SRM behavior or Statechart assertion at that level

– Sufficiency of the SRM• Is there any mission-critical, safety-critical requirement not

covered by an SRM behavior or Statechart assertion at that level?

• A “correlation map” is built to capture these relationships

Validation Possibilities

SRM Behaviors Requirements

SRM Correct?Requirements Complete?

SRM Correct?Requirements In

Scope and Valid?

ValidatedRequirements

Unambiguous, Correct,

Consistent, & Verifiable

Requirements

An Example Behavior

Extensions – Q2 & Q3

Preconditions

Main Success Scenario – Q1 & Q2

Constraints

Post-conditions

Goal

References

Requirement Proxies

Requirement Proxies

Requirement Proxies

Requirement Proxies

Other Analysis Tools

Subject Requirement

Child Requirements

Parent Requirements

Validation Findings

Correlation Mapping

Requirement Data

Subject Requirement

Child Requirements

Parent Requirements

Validation Findings

Correlation Mapping

Subject Requirement

Parent Traces

Subject Requirement

Child Requirements

Parent Requirements

Validation Findings

Correlation Mapping

Parent Requirements

Child Traces

Subject Requirement

Parent Requirements

Validation Findings

Correlation Mapping

Child Requirements

Correlation Mapping

Subject Requirement

Child Requirements

Parent Requirements

Validation Findings

Correlation Mapping

Validation Findings

Subject Requirement

Child Requirements

Parent Requirements

Correlation Mapping

Validation Findings

Requirement Proxies

Correlation Mapping & Requirement Evaluation

Correlation Map

Correlation MapM

odel

Ele

men

tsR

equi

rem

ent M

appi

ngFi

ndin

gs/Is

sues

Correct (IVV 09-1)• Applicable requirement(s) meet all or part of the goals

and behaviors of the system– Note: not all requirements can be evaluated in isolation; it may require a

set of requirements to be evaluated together in order to determine that a particular goal or behavior is being met).

• The requirements are an accurate elaboration of the defined objectives or goals– e.g., the use of temporal modal operators like “next”, “until”, “always”,

and “eventually”, are appropriately used to reflect the desired behavior• The requirements adequately refine the higher-level

requirements• Design or implementation-specific information is

specified as constraints to the behaviors captured in the requirements

Coverage of Model

Consistency with Model

Separation of Information

Complete (IVV 09-1)• All the needed information to completely specify a

desired behavior is identified (i.e., all preconditions, postconditions, and invariants are specified for the described behavior).

• Threads of behavior are represented by more than one requirement, versus one compound requirement that attempts to capture the entire thread (i.e., that each requirement specifies only one “thing”).

• The use of conjunctions (e.g., “and”, “or”) are restricted to preconditions, postconditions, and invariants.

Coverage of Model

Correlation Mapping & Requirement Evaluation

Correlation Mapping & Requirement Evaluation

Identifying Issues• L2 Rqmnt - The Project shall generate, route, transport, store

and execute a sequence containing any of the following types of time-tagged commands:  absolute time, time relative to a sequence-external time value stored on-board, time relative to the execution of the previous command in the sequence.

• L3 Rqmnt - The sequence time tags shall be either an absolute execution time, time relative to a sequence external time value, or a time relative to the execution of the previous command or command file.

• L4 Rqmnt - The flight software shall provide the means for running onboard relative and absolute time, relative to a sequence external time value,  tagged sequences.

Identifying IssuesM

odel

Ele

men

tsR

equi

rem

ent M

appi

ngFi

ndin

gs/Is

sues

Model Requirements Findings

Lessons Learned & Best Practices

Challenges• Varying levels of detail between the SRM and

requirements being validated• What vs. How – “Requirements Model” vs. “Design

Model”• Terminology differences between SRM behaviors and

requirements• Lack of adequate tools, work instructions, product

descriptions/templates – particularly MKS Artifact Mapping capability

Questions?