risk management: achieving higher maturity & capability levels through the lego approach

www.eng.it

26°International Workshop on Software

Measurement (IWSM) and 11th International

Conference on Software Process and Product

Measurement (MENSURA)

Berlin (Germany) - October 5-7, 2016

Luigi Buglione

Alain Abran

Christiane Gresse von Wangenheim

Fergal McCaffery

Jean C.R. Hauck

Achieving Higher Maturity & Capability Levels through the LEGO Approach

Risk Management

http://iwsm-mensura.org/2016/Program











http://ivs.cs.uni-magdeburg.de/sw-eng/us/giak/

http://www.dasma.org/

www.eng.it 2 IWSM-MENSURA 2016 – October 6, 2016

© 2016 Buglione; Abran, Gresse von Wangenheim, McCaffery, Hauck

Goals of the presentation

1. Discuss the impact an organization can suffer or achieve from the way risk is managed

2. Look at the ‘big picture’ in order to convert Risks into Critical Success Factors (CSFs) when dealing with risky events looking at best practices from several frameworks on Risk Management

3. Present a LEGO (Living EnGineering prOcess) example with the Risk Management process

Risk Mgmt and LEGO





ETS - GELOG At a glance

www.etsmtl.ca



http://gelog.etsmtl.ca/

http://www.eng.it/

http://en.etsmtl.ca/en/home?lang=en-ca

http://en.etsmtl.ca/en/home?lang=en-ca



DKIT At a glance

Dundalk Institute of

Technology is a 90 acre

campus situated between

Dublin and Belfast (each

approximately 50 miles

away).

The Institute consists of 4

Schools:

1. Business & Humanities

2. Informatics & Creative

Arts

3. Engineering

4. Health & Science

The Regulated Software Research Group is part of

LERO (the Irish Software Engineering Research

Centre) at the School of Informatics & Creative

Media





UFSC At a glance

Federal University of Santa Catarina Florianópolis/Brazil [http://www.ufsc.br]

• 25,737 Undergraduate students

• 8,543 Graduate students

• 34,280 Students

INCoD an institute for excellence in research, validation and dissemination

to support digital convergence. [http://www.incod.ufsc.br]

The Software Quality Group focuses on scientific research, development and

transfer of SE models, methods & tools. [http://www.gqs.ufsc.br]

[http://www.youtube.com/watch?v=V6E1Z5DEuvk]



http://www.youtube.com/watch?v=V6E1Z5DEuvk



Engineering At a glance

www.eng.it

ISSRE 2014 – Naples (Italy), Nov 5, 2014



http://www.eng.it/

http://www.eng.it/

http://issre.net/home



Risk Mgmt and LEGO Let’s Social...ize!

If you want to share comments/notes/pics…

@IWSMMensura

@lbu_measure

#LEGO

#MCM

#Risk

#RiskManagement

…





Agenda

• Introduction

– A couple of examples about (non) Risk Management…

– Some questions…

• MCMs (Maturity & Capability Models) – Representations & Dimensions

– Why do we need to choose a MCM?

– Coverage & classification of MCMs

• MCMs & Risk Management in Horizontal MCMs (H-MCMs)

– CMMI-DEV/SVC and ISO 15504-2

– Other Sources

• LEGO and Risk Management

– The LEGO approach

– Applying LEGO to Risk Management Elements of Interest (EoI)

– Suggested Improvements

• Conclusions & Prospects

• Q & A

Risk Mgmt and LEGO





Example: latest earthquake in Italy (Sept 2016) Introduction

• 6.2 Richter scale • 290+ people died • 2000+ people without home right now • Did somebody consider such risk in the

past within Italy? How was risk managed? Did the Government invest over this past few years in reducing the chances of such events happening?

Amatrice



https://en.wikipedia.org/wiki/Richter_magnitude_scale



http://edition.cnn.com/2016/08/23/europe/italy-earthquake/



Example: Apple ‘Antenna Gate’ (2010) Introduction

• At the iPhone 4 launch (June 2010) [https://en.wikipedia.org/wiki/IPhone_4#Antenna] • Placed in the wrong place, the signal was lower and the iPhone less performant • The ‘AntennaGate’ was estimated to impact for 20% of Apple sales for iPhone 4

(http://fortune.com/2010/09/08/antennagate-cost-apple-20-of-sales/) • Did (Apple) they managed such risk during the Design phase? How? How much?



https://en.wikipedia.org/wiki/IPhone_4#Antenna










http://fortune.com/2010/09/08/antennagate-cost-apple-20-of-sales/
























Some (important) questions... Introduction

What is risk and what is a damage?

E.g...what are the differences between CMMI and SPICE manage risks?

Are there further frameworks helping to better deal with risks? Do we have a risk catalogue?

How much value could we achieve converting risks into a CSF?





Agenda

• Introduction








– Other Sources






• Q & A

Risk Mgmt and LEGO





Why do we need choosing a MCMs? MCMs





Representations - Staged MCMs

• ML: 5

• PA: 24

• N.min PA : ML1 (0)

• N.max PA : ML3 (13)

ML Focus Id. PA Title

5 Optimizing OPM Organizational Performance Management

CAR Causal Analysis & Resolution

4 Predictable OPP Organizational Process Performance

QPM Quantitative Project Management

3 Defined RD Requirement Development

TS Technical Solution

PI Product Integration

VAL Validation

VER Verification

OPD Organizational Process Definition

OPF Organizational Process Focus

OT Organizational Training

IPM Integrated Project Management

RSKM Risk Management

DAR Decision Analysis & Resolution

2 Managed REQM Requirement Management

PP Project Planning

PMC Project Monitoring & Control

SAM Supplier Agreement Management

MA Measurement & Analysis

PPQA Process & Product Quality Assurance





Representations - Continuous MCMs

• PA categories: 4

• PA: 24 22

• N.min PA per Category : Process Management (5)

• N.max PA per Category: Project Management (7)

Process Categories

Maturity Levels

Process Management

Project Management

Engineering Support

Optimizing OPM CAR

Predictable OPP QPM

Defined OPF

OPD

OT

IPM

RKSM

RD

TS

PI

VER

VAL

DAR

Managed PP

PMC

SAM

REQM CM

MA

PPQA

Initial Ad-hoc processes





Representations – Continuous (example) MCMs

Special cause

(GP.2.2 @ OT)

Common cause (GP.2.9 @

+PA) • Source: SQI Appraisall Assistant - http://goo.gl/i6IvI



http://goo.gl/i6IvI

http://goo.gl/i6IvI



MCMs Classifying MCMs by Dimension

• Horizontal: MMs going through the whole supply chain SwEng: ISO 15504, CMMI, FAA i-CMM, …

• Vertical: MMs focusing on a single perspective/group of processes Test Mgmt: TMM, TPI, …

Project Mgmt: PM-MM, OPM3, …

Requirement Mgmt: ....

• Diagonal: MMs focused on Organizational/Support processes People CMM, TSP, PSP, …Risk Management S

ou

rce

: B

ug

lion

e L

., A

n E

co

logic

al

Vie

w o

n P

rocess I

mp

rove

me

nt:

So

me

Tho

ugh

ts f

or

Imp

rovin

g

Pro

cess

App

rais

als

, 4W

CS

Q,

4th

W

orl

d

Con

gre

ss

on

So

ftw

are

Q

ua

lity,

Wash

ing

ton

D.C

. (U

SA

), 1

5-1

8 S

ep

tem

be

r 2

00

8



http://www.asq.org/conferences/wcsq

http://www.asq.org/conferences/wcsq



Agenda

• Introduction








– Other Sources






• Q & A

Risk Mgmt and LEGO





CMMI-DEV and ISO 15504 – Risk Mgmt ref’s MCMs and Risk Mgmt

Model CMMI-DEV/SVC ISO 15504-12207

Domain Sw-SE Sw-SE

PRM (source) CMMI-DEV v1.3 ISO 12207

PRM (# Processes) 22 47

Process Categories RSKM (Risk Management) – ML3 (Staged representation)

MAN.5 (Risk Management)

Risk Mgmt-related process(es)

SCAMPI v1.3 ISO 15504-2 ISO 15504-5

PAM ext. Appraisals PP-SP-2.2 (Identify Project Risks) PMC-SP-1.3 (Monitor Project Risks)

ACQ.1, ACQ.3, ACQ.4, OPE.1, ENG.1, ENG.2, SUP.10, MAN.3, MAN.5, PIM.3, PA2.1, PA4.1, GP5.1.4, GP5.2.2. related BP (Base Practices)

PAM Risk-related issues

Sw-SE Sw-SE





MCMs and Risk Mgmt

Model/ Framework Repr. Type

ML (#) Architect-

Type Comments/Notes

Project Risk Maturity Model (PRMM)

Staged 4 [1-4] Level-based • 6 perspectives

IACCM CMM Staged 5 [1-5] Level-based • 9 dimensions (#7: Risk Management)

MMGRseg Continuous 5 [1-5] Level-based • Aligned with ISO/IEC 27005 [32]; 43 Control Objectives into 6 groups; Final Risk Scorecard

MPS RMMM Staged 6 [1-6] Matrix-based • 6 drivers for assessing on an ordinal scale business risks

RIMS RMM for Enterprise Risk Management (ERM)

Staged 6 [0-5] Matrix-based • 7 process attributes; for each one, a series of Key Drivers defined

IS RMM Staged 5 [1-5] Level-based • 9 control elements, each one with a variable number of components

INCOSE RMM Staged 4 [1-4] Matrix-based • 5 Drivers

Risk Analysis (WBS) + RBS --- --- WBS -based • Creation of a Risk Breakdown Structure according to the project WBS and quantification of risks by each WBS task (calculation)

Choosing Risk Mgmt MCMs - Results





Agenda

• Introduction








– Other Sources






• Q & A

Risk Mgmt and LEGO





LEGO and SvcMgmt The LEGO Approach

1. MCM Repository 2. Process

Architecture

4. Appraisal Method 3. Mappings &

Comparisons

1.

Identify goals

2.

Query

MCM

repository

3.

Include

new

elements

4.

Adapt

& Adopt

Source: Buglione L., Gresse von Wangenheim C., Hauck J.C.R., Mc Caffery F., The

LEGO Maturity & Capability Model Approach, Proceedings of 5WCSQ, 5th World

Congress on Software Quality, Shanghai (China), Oct 31- Nov 4 2011



http://srv.gqs.ufsc.br:8080/mcm/





Applying LEGO to Risk Mgmt Experiencing LEGO...

The LEGO steps & related activities & outcomes:

1. Identify Goals Improve the internal Risk Management (RM) capability in order to generate more value to

our organization over time (product+service)

Assume the target BPM (Business Process Model) to improve is generically the ISO 15504 MAN.5 process

2. Query the MCM repository Filter the list of available KM-based MCMs from the MCM repository

Next table (EoI – Element of Interest) is a filter of the elements by each of the KM MCMs considered

3. Include new elements into the target BPM Next table (Suggested Improvements) lists the possible EoI matched with the requested

MCMs (both SPs and GPs)

4. Adapt & Adopt Map each practice of the improved process to the related internal QMS process(es)

Validate the mapping results before using them in the daily activities





Step 2 - EoI: Elements of Interest (1/4)

Model/ Framework Elements of Interest (EoI)

Project Risk Maturity Model (PRMM)

Six (6) perspectives (Stakeholders; Risk Identification; Risk Analysis; Risk Responses; Project Management; Culture)

Paid attention to: o The ‘Culture’ perspective is interesting because it deals with people attitude

towards risk o The ‘Stakeholders’ analysis can allow to catch all possible threats and

vulnerabilities in terms of missing items to be discussed and analyzed for possible contingencies to the project plan. The PRMM process considers their engagement for initiating the risk management process

o ‘Risk Response’ is what in other models/frameworks could be the list of ‘countermeasures’ in a ‘Risk Catalogue’

IACCM CMM Quantitative approach (from SixSigma practices) with 9 dimensions (1. leadership; 2. customer/supplier experience; 3. execution and delivery; 4. solution requirements management; 5. financial; 6.information systems/knowledge management; 7. risk management; 8. strategy; 9. people development)

Interesting the eventual inclusion of o ‘Solution Requirements management’ o ‘IS/Knowledge Management’, o ‘People development’, as in the SEI’s People-CMM

LEGO and Risk Mgmt







MMGRseg Alignment with security issues (ISO 27005 [32]) Refinement of the maturity levels into three stages (immaturity, maturity,

excellence) 6 Control Objectives (CO) – processes - each one with a series of practices

o CD1 Context Definition; AA1 Risk Analysis/Assessment; RT1 Risk Treatment; RA1 Risk Acceptance; RC1 Risk Communication; MA1 Monitoring & Critical Analysis

Paid attention to: o CD1.9 (Collect and Store information); AA1.7 (Avoid Rework); AA1.8 (Revise

the process of risk estimation); RT1.4 (Define how to measure the effectiveness of controls); RT1.5 (Calculate Residual Risks); RC1.x (all practices); MA1.3 (Standardize the Monitoring and Critical Analysis activity)

Assessment representation with Kiviat graphs, possible to use also a questionnaire (as in the old Sw-CMM) or also a NPLF ordinal scale using the typical MCM appraisal approach

MPS RMMM ML grow with a larger environment to control (the larger the environment, the higher the ML)

This MCM is about Police Security and cross a series of organizational structures that should be in place, according to their org model

Two dimensions in the matrix-grid: Maturity Level by Maturity Elements Ordinal scale (No, Minimal, Partial, Yes, Significant; Substantial, Full) for rating

each crossed cell in the matrix

LEGO and Risk Mgmt







RIMS RMM 7 process attributes (Adoption of ERM-based discipline; ERM process management; Risk appetite management; Root-cause discipline; Uncovering risk; Performance Management; Business Resiliency and Sustainability), for each one, a series of Key Drivers defined

In each process attribute, there is a definition for matching a certain level (from Non-Existent till Level 5)

Particular attention could be devoted to those aspects: o PA#4 (Root-Cause Discipline) historicize data, classify risk, understanding

the why’s o PA#5 (Uncovering Risks) formalizing risk indicators/measures;

transforming risks into opportunities (CSF’s) o PA#7 (Business Resiliency and Sustainability) understanding of

consequences of action or inaction

IS RMM 9 control elements (Participants; Technologies; Information; Work Practices; Products & Services; Customers ; Infrastructure; Environment ; Strategies)

Based on ISO 31000 Risk Management Process [31], refining the process activities into ‘Control Objectives’: EC (Establishment of the Context); AP (Risk Assessment); TR (Risk Treatment); CR (Communication); SR (Monitoring & Review)

To pay attention eventually to: o EC.3 (Define a normalized method for the definition of the context) o EC.4 (Define a method of appreciation of the risks) o EC.7 (Define a plan of communication) o EC.9 (Define the level of tolerance or acceptance of the risks) o AP.6 + TR.6 + CR.3 + SR.4 (Collect and Store information about…) o SR.1 (Monitor Risk Management Indicators)

LEGO and Risk Mgmt







INCOSE RMM 5 Drivers (Definition; Culture; Process; Experience; Application) Checklist (matrix-based) crossing Levels from 1 (Ad-hoc) to 4

(Managed) with the drivers, asin Crosby’s Quality Management Maturity Grid (QMMG) [2]

To pay attention eventually to: o Definition towards a proactive use of risk management o Culture + Experience learn from experiences, knowledge management

for risk management o Application use of quali-quantitative tools helping to deal with risks as an

opportunity when planning and estimate a new activity/project

LEGO and Risk Mgmt





Step 3 - Suggestions for Improvement (1/2)

ISO 15504 MAN.5 process Suggested Improvements

BP 01 – Establish Risk Management scope

Add practices/notes for collecting information about the Context for the project to be analysed (scope management)

Fundamental a proper definition of events and related risks in a Risk Catalogue Add practices about the need to consider the right stakeholders for eliciting

requirements and consequently potential risks form multiple viewpoints. It can help to better define the scope for the project and its related risks

BP 02 - Define Risk Management strategy

Add practices/notes about the strategic need to be resilient as a way to ‘genetically’ manage risks in a proactive way. Define a method for evaluating risks for a proper (proactive) management.

Communication needs to be part of a risk strategy: people not aware about what is a risk couldn’t work for excellence neither for obtain good results (wouldn’t be a lean organization, at least!)

Culture and Experience from teams is fundamental to avoid and learn by experience, sharing information by a ‘Risk Catalogue’ (as well as in IT Service Management models, ITSM personnel use a ‘Service Catalogue’)

BP 03 – Identify risks Add practices/notes about the need for a ‘risk catalogue’, querying it for any risk analysis in order to find yet classified/managed risks, with possible countermeasures.

Any uncovered risk should be recorded as a new item into the risk catalogue, updating the organization risk history as a basis for any further improvement

LEGO and Risk Mgmt

In the following tables, there is a list of ‘suggested improvements’ to the target process (in this example MAN.5 from ISO 15504) that could be added in its next revision by BP (Base Practice), kept from the EoI previously analysed and listed.





Step 3 - Suggestions for Improvement (2/2)

ISO 15504 MAN.5 process Suggested Improvements

BP 04 - Analyze risks Add practices/notes about the opportunity to have a yet-ready list of possible countermeasures from a Risk Catalogue, properly updated over time from the whole organization’s teams

BP 05 – Define and perform risk treatment actions

Add practices for specifying how to measure the effectiveness of controls and calculate residual risks.

Another fundamental issue will be the definition of thresholds and criteria based on historical data for their dynamic revision over time, choosing the proper updating frequency for any kind/family of risk issues.

BP 06 - Monitor risks Add in order to standardize the monitoring of risks along time. Need to formalize risk indicators/measures and transforming risks into opportunities

(CSF’s).

BP 07 - Take preventive or corrective actions

Add practices/notes about the need for RCA (Root-Cause Analysis) as the basic TQM technique to use for determining the best choice from your own historical project/organizational data.

Communication is not only part of the strategy but – as an action – also the closing step for a corrective/preventive action, checking that the target audience will have properly received and acted against the requested action.

Tools could help in making easier the identification of recurring risk patterns and suggest possible countermeasures

LEGO and Risk Mgmt





Agenda

• Introduction








– Other Sources






• Q & A

Risk Mgmt and LEGO





Conclusions & Future Works

• Risks as threats or opportunities? A risk should be known, analyzed and managed: having a ‘risk catalogue’ (as a service catalogue)

can help organizations to manage a threat and possibly convert it into an improvement opportunity Contigencies should be evaluated but not spent directly into a Gantt chart if not still happened Risk Management is not part of Project Management, but it’s a separated, supporting process Possibly risks should be measured, not only evaluated Look at Value as the final goal to achieve in order to really improve our activities

• Models and Methods Many models, taxonomies and frameworks can be valid for managing risks The value when better managing risks can lead to a lower TCO for projects E.g. ISO 31000 is not the solely source to consider, but also CMMI/SPICE risk-related process could

be considered

• LEGO’ (Living EnGineering prOcess) approach • http://slideshare.re/nssLR8 [5WCSQ, Shangai, Nov 2011] • Choose and integrate the ‘pieces of the puzzle’ you need for your goals the target is your QMS,

not the model(s) you are using

Next Steps Identify further ‘silver bullets’ for leveraging the joint view of products and services, also from a

business viewpoint Hybridize more models and techniques between the two communities for benchmarking purposes

All models are wrong. Some models are useful.

(George Box, Mathematician, 1919-2013)

Risk Mgmt and LEGO



http://slideshare.re/nssLR8







http://en.wikiquote.org/wiki/George_E._P._Box



Lessons Learned...

UR

L:

ww

w.d

ilbe

rt.c

om

Risk Mgmt and LEGO



http://www.dilbert.com/







Q & A

Danke für Ihre Aufmerksamkeit!

Thanks for your attention!

Risk Mgmt and LEGO





Our Contact Data Risk Mgmt and LEGO

Luigi

Buglione Engineering Ing. Inf. /ETS

[email protected]

Fergal

McCaffery

DKIT [email protected]

C. Gresse von

Wangenheim

UFSC [email protected]

Alain

Abran

ETS [email protected]

Jean Carlo R.

Hauck

UFSC

[email protected]



http://www.eng.it/