requirements' quality improvement: a successful case study

47
Jornada sobre Innovación y Tendencias en la Gestión de Requisitos 9 de mayo Madrid Sede Madrid Network organizan: #gestionrequisitos2016 Mejora en la calidad de requisitos: Un caso de éxito José M. Fuentes 1

Upload: the-reuse-company

Post on 12-Apr-2017

254 views

Category:

Engineering


2 download

TRANSCRIPT

Jornada sobre

Innovación y Tendencias

en la Gestión de Requisitos

9 de mayo – MadridSede Madrid Network

organizan:

#gestionrequisitos2016

Mejora en la calidad de requisitos:Un caso de éxito

José M. Fuentes

1

Presenter profile

• José M. Fuentes

[email protected]

• Co-founder of The REUSE Company

• Current Chief Commercial Officer of The REUSE Company

José M. FuentesCCO

• Member of AEIS Board (Spanish Chapter of INCOSE)

• Member of INCOSE Requirements Engineering WG

• Contributor to INCOSE Guide for Writing Requirements

• PM for TRC in EU Research Projects:

– AUTOSoft project

– CRYSTAL Project

– AMASS Project

– REVaMP22

The REUSE Company is a tool

manufacturer providing Knowledge-Centric

solutions to critical systems engineering

such as requirements quality management

or systems knowledge reuse.

The Requirements Quality Suite (RQS), a

tool to manage the requirements V&V

activity, is the most well-known product.

TRC is a SME based in Madrid (Spain)

Parque Tecnológico Legatec.

The REUSE Company profile

3

Contents

• Introduction

– The impact of poor quality in our projects

– Requirements Quality Analysis

• Practical case study

– Goals, inputs and expected outputs

– Tools benchmark

– The PoC Process

– PoC results

4

The impact of poor quality

The impact of poor quality projects

6

Why Requirements Quality Analysis?

7

Why Requirements Quality Analysis?

Doing the right thing right (verification)

http://www.theguardian.com/world/2014/may/21/french-railway-operator-sncf-orders-trains-

too-big

http://elpais.com/elpais/2015/02/04/inenglish/1423052376_326956.html8

The FOCUS:

Requirements Quality Analysis

Why Requirements Quality Analysis?

Source : INCOSE SE Handbook V3.2

95%

85%

70%

Time

Cu

mu

lati

ve p

erce

nta

ge

Life

cylc

e C

ost

Operations through Disposal

100%Production

and test

50%

8%Design

15% 20%Concept

Commited Costs

3-6x

500-1000x

20-100x

Development

10

Systems and Requirements

Engineering life-cycles

CONOPS

Stakeholders

Requirements

System

Requirements

System

Design

Equipment

Requirements

Equipment

Design Equipment

Verification

System

Equipment

System Verification

Product

Product Verification

11

Systems and Requirements

Engineering life-cycles

Elicitation Analysis Specification Validation

close gapsclarify

rewrite

re-evaluate

confirm and correct

Source: Karl Wiegers

12

Systems and Requirements

Engineering life-cycles

CONOPS

Stakeholders

Requirements

System

Requirements

System

Design

Equipment

Requirements

Equipment

Design Equipment

Verification

System

Equipment

System Verification

Product

Product Verification

Requirements

Verification

Requirements

Verification

Requirements

Verification

Design

ValidationDesign

Verification

Requirements

Validation

13

Practical Application at

• Objectives

– Perform correctness, completeness and consistency analyses of requirements (individually and collectively) to improve the Quality of requirements specifications

– Assess the computer-aided requirements authoring feature to accelerate the learning curve of new practitioners (or improve the capability of current practitioners) in requirements development

• Goal

– Exonerate engineers from format concerns (structure) and allow them to concentrate on content (essence of requirements): technical data useful for design

Quick Proof of Concept on Requirements

Quality Improvement

15

• External audits results

– “… Requirements Characterization is not complete: Derived/uncovered requirements justification, Contribution, Categories (technical vs non-technical), V&V Methods…

– …V&V Plan is not complete: Verification activities, or agreed alternate practices (waivers) and associated deliverables…”

• CMMI for Development

– Requirement Development process area – SG 3 Analyze and Validate Requirements

• “… Analyze requirements to determine whether they satisfy the of higher level requirements.Analyze requirements to ensure that they are complete, feasible, realizable, and verifiable…”

– Verification process area – SG 2 Perform Peer Reviews

• “… Establish and maintain checklists to ensure that the work products are reviewed consistently...Rules of construction , Completeness, Correctness…”

Also, a means to improve current practices

16

Previous results

Most common requirements defects. Source: Gauthier Fanmuy - the RAMP project: - AFIS

17

Requirements quality characteristics vs

quality metrics• Well-known requirements quality characteristics

• IEEE Std. 830:– Correct

– Unambiguous

– Complete

– Consistent

– Ranked

– Verifiable

– Modifiable

– Traceable

• ESA PSS-05,

ISO/IEC 29148, others:

– Pretty much the same characteristics

• SMART:– Specific

– Measurable

– Achievable

– Relevant

– Traceable

"I believe that this nation should commit

itself to achieving the goal, before this

decade is out, of landing a man on the

Moon and returning him safely to Earth"

18

Quality characteristics to measure

• How to… Perform CCC

Define Metrics for the Rules proposed in the

INCOSE Guide + Others

INCOSE Requirements Working Group

Guide for Writing Requirements V4

Characteristic Cxx – Characteristic name

Rationale: xxxx

Strategy: xxxx

Rules that help establish this characteristic:

Rxx - /Section/Rule nameAvoid xxxxRyy - /Section/Rule nameAvoid yyy

19

• Quality of individual requirements

– Correctness Requirement statement is understandable by humans (TRC)

Requirement statement’s structure is agreed with the SE dept. (TRC)

Requirement Statement corresponds to user request (ISO TR 24766)

The requirement contains all the information necessary for design and implementation (Alstom)

• Quality of requirements sets

– Consistency Absence of conflict among a set of requirements (ISO TR 24766)

A set of requirements is consistent if each necessity (requirement) is expressed in one and only one requirement (TRC)

– Completeness The set of requirements represents a complete definition of the product

(ISO TR 24766)

Completeness by comparing project content vs project content (TRC)

Managing Requirements Quality: CCC approach

20

Requirements Quality Analysis Benchmark

INITIAL SOURCE:

• AFIS « RAMP » project 2010-2012

• Evaluations by AIRBUS 2012

21

Requirements Quality Suite

The Requirements Quality Suite (RQS) intends to tackle requirements quality

management by offering a set of tools and processes

Automatic measurement of requirements quality metric

Support to Requirements Authoring

RQS models requirements quality metrics using the CCC approach (Correctness,

Consistency and Completeness)

Requirements Quality Analyzer (RQA):

to setup, check and manage the quality of a requirements specification.

Requirement Authoring Tool (RAT):

to assist authors while they are creating or editing requirements.

Knowledge Manager (KM):

to manage knowledge around a requirements specification: the ontology it is based on, the structure of the requirements to be used in the project, the communication between authors and domain architects.22

RQS – Requirements Quality Suite: languages

• RQS is highly dependent of the language of the

requirements

• Languages supported so far:

24

• Correctness metrics are quantitative

• Correctness metric values are calculated counting items

– Example: In the Metric Text length in words the metric counts the number of words; then the QF transforms it into a quantitative value

• The process is simplified by using interval (step, discrete) quality functions

• Metrics use one of the following quality functions:

Managing the metrics Quality Functions (QF)

25

textLength()

QHigh

Med

Low

1 5 10 30 50

Quality Management Definition: Maturity

lifecycle

WHITE Belt

Metrics

YELLOW Belt

Metrics

ORANGE Belt

Metrics

BLUE Belt

Metrics

BROWN Belt

Metrics

BLACK Belt

Metrics

GREEN Belt

Metrics

DEMANDINGLEVEL

textLength()

QHigh

Med

Low

1 5 10 30 50

textLength()

QHigh

Med

Low

1 4 6 2526

Specification quality assessment w.r.t. the

Maturity lifecycle

DEMANDING LEVEL

WHITE BeltMetrics

YELLOW BeltMetrics

ORANGE BeltMetrics

GREEN BeltMetrics

BROWN BeltMetrics

BLACK BeltMetrics

BLUE BeltMetrics

CORRELATED LEVELS OF RESULTS

27

Specification quality assessment: The Goal

WHITE BeltMetrics

YELLOW BeltMetrics

ORANGE BeltMetrics

GREEN BeltMetrics

BROWN BeltMetrics

BLACK BeltMetrics

BLUE BeltMetrics

DEMANDING LEVEL

CORRELATED LEVELS OF RESULTS

28

Specification quality assessment: The Path

TIME

WHITE BeltMetrics

YELLOW BeltMetrics

ORANGE BeltMetrics

GREEN BeltMetrics

BROWN BeltMetrics

BLACK BeltMetrics

BLUE BeltMetrics

29

Specification quality assessment: Maturity level by depts.

or Teams

WHITE BeltMetrics

YELLOW BeltMetrics

ORANGE BeltMetrics

GREEN BeltMetrics

BROWN BeltMetrics

BLACK BeltMetrics

BLUE BeltMetrics

TIME

30

The Knowledge Base

31

Terminology layer

Valid terms, forbidden terms, other NL terms, Syntactic clustering types, everything as concepts

Thesaurus layer

Relationships among concepts (hierarchies, associations, synonyms…) as well as semantic clusters and relationship typesPatterns layer

Matching Patterns

Formalization layer

Semantic formalization

Inference layer

For decision making (e.g. consistency, completeness)

• Patterns:

– Represents the structures every correct requirement should

meet

– Different types of requirements different patterns (templates)

– Customizable for every domain, customer and content of each

requirements document

– Libraries with sets of patterns

– Represented as a sequential set of restrictions: placeholders

Examples of requirements metrics: patterns

32

When <Event> <Component> Shall <Action> <Object> Time_constraint

Knowledge Base: Example

33

A380 A350 System Operate Temperature Environment PressureControlled

Vocabulary

A380 A350

<<Aircraft>> “ Greater than (>) “

Operate Work

<<Operation>>

<<Aircraft>> Shall <Operation> <<Minimum>>At Environment Of[MEASUREMENT

UNIT]NUMBER

temperature“ Greater than (>) “

ºC-70

Patterns

Temperature Pressure

Environment

Temperature [-60ºC , +60ºC]“ Operation Range “

Inference

Rules NUMBER “ Lower than (<) “ -60º NUMBER “ Greater than (>) “ +60º||

Thesaurus

Formalizations The aircraft shall be able to operate

at a minimum temperature of -70º C

If ºC ºC

“ Lower than (<) “

Shall

At a minimum

<<Minimum>>

At a minimum Of

• The REUSE Company has developed IT solutions that attempt to

understand, formalize, represent, reason-about and search-for all

kinds of knowledge assets

• Using: Terminology Control, Patterns, Graphs and Natural language

Processing

Knowledge Base: Requirements Patterns

matching

34

UR044 : The Radar shall identify hits at a minimum rate of 10 units per second

The <DEVICE> Shall <ACTION> <ITEMS> <MINIMUM>At Rate of<RATE

VALUE>[NUMBER]

Radar Hits

<<Detect>>

10 Units per Second

<<Minimum Value>>

Hits

• To promote requirements reuse

• To detect duplicates

• To provide quick access to related requirements

Knowledge base: semantic search engine

35

The PoC Process

Process: Work with one Requirements

Specification

37

OriginalReqs.

Specification

INITIAL QUALITY ASSESSMENT

OriginalReqs.

Specification

QUALITY METRICS

DEFINITION

OrganizationKnowledge Base

V1

SPECIFICATION UPDATE

Reqs.Specification

Rev 1

OriginalReqs.

Specification

OrganizationKB V1

OrganizationKB V2

QualityResults

OrganizationKB V0

1st QUALITY ASSESSMENT WITH

BELTS

OriginalReqs.

Specification

OrganizationKB V1

2nd QUALITY ASSESSMENT WITH BELTS

Reqs.Specification

Rev 1

OrganizationKB V2

The PoC Results

Results: Initial Quality Assessment without belts

39

Original Specification

All out of the box metrics enabled 32 Metrics (INCOSE + TRC)

329 Requirements

% of Requirements

Effort centred in metrics for Correctness

Results: Developed Colored Belts

40

WHITE Belt

YELLOW Belt ORANGE Belt

20 Metrics correctness 31 Metrics correctness 51 Metrics correctness

+

2 Metrics completeness

Results: Developed Ontology

Terminology

Patterns

1 381 unclassified terms after first indexation

70 new organization specific patterns

41

Results: First Quality Assessment with belts

42

Original

Specification329 Requirements

% of Requirements

WHITE Belt YELLOW Belt ORANGE Belt

Results: Specification modification (by experts)

43

WHITE Belt

Metrics

329 Requirements 438 Requirements

66.26% Modified Reqs.

Final Specification

% of Requirements % of Requirements

Original Specification

Results: Comparison original vs modified

specification

44

Final Specification

Original Specification

White Belt Original vs. Final Spec.

• Achieved results

– A tool-supported requirements quality process

– A gradual set of quality metrics (belts)

– A Formal Reusable Knowledge Base

– ~70 new Patterns

– One improved Requirements Specification

– An Alstom “Guide for Requirements Authoring”

• Resources

– 2 participants from Alstom, 2 participants from TRC

– 1.5 effective months in calendar: 3.37 PM

Learning and mastering of the tool suiteDefinition of Metrics, Generation of OntologyModifications of requirements

Conclusion

45

0

50

100

150

200

250

White belt Yellow belt Orange belt

PoC Hours

REQUIREMENTSQUALITY

ASSESSMENT

Specificationof Type 2

CustomerKB V2

QualityResults

REQUIREMENTSQUALITY

ASSESSMENT

Specificationof Type 3

CustomerKB V2

QualityResults

• Decision to apply the white belt maturity to

analyze the quality of other specifications

– Fine-tune the metrics

• Decision to enhance the current Requirement

Engineering Process by including

requirements verification activities

• Decision to launch a real-scale pilot project:

real industrial project and staff, IT

infrastructure, measurement of performances

Conclusions of the PoC

46

Detalles de Contacto

José M. Fuentes

[email protected]

+34 912 17 25 96

@ReuseCompany

Mejora en la calidad de requisitos

47