bi-directional gap analyses of cmmi ... the process 1. become familiar with cmmi, aspice, and...
TRANSCRIPT
BI-DIRECTIONAL GAP ANALYSES OFCMMI, AUTOMOTIVE SPICE, AND INCOSE
Travis Foust, Matthew Ramirez, Regina Dorow
October 2nd, 2017
2
BACKGROUNDCMMI for Development is a process reference model developed by the Software Engineering Institute (SEI) to be used by organizations developing systems (software or otherwise) to measure their maturity and/or capability.
Automotive SPICE, or ASPICE, is a process reference model developed by organizations within the automotive industry to create a more automotive-focused reference model compared to SPICE or CMMI. ASPICE is derived from SPICE.
INCOSE Systems Engineering Handbook is a handbook developed by the International Council of Systems Engineering to be a practical guide to Systems Engineering. INCOSE standards are be used to evaluate individual engineers while CMMI and ASPICE are used to evaluate organizations. The handbook adheres to ISO15288.
Objective - While CMMI and ASPICE are the two dominant sets of standards in the automotive industry, this study will attempt to understand the differences between all three of these standards. The results will be a valuable resource in determining what standard(s) should be mapped and adopted as best practice, required of suppliers, and/or used for internal appraisals.
3
CMMIProject Management Engineering
SupportProcess Management
REQM (2)
Requirements
Management
PP (2)
Project
Planning
PMC (2)
Project Monitoring
and Control
SAM (2)
Supplier Agreement
Management
IPM (3)
Integrated Project
Management
RSKM (2)
Risk
Management
QPM (4)
Quantitative Project
Management
RD (3)
Requirements
Development
TS (3)
Technical
Solution
PI (3)
Product
Integration
VER (3)
Verification
VAL (3)
Validation
MA (2)
Measurement
and Analysis
DAR (3)
Decision Analysis
and Resolution
PPQA (2)
Process and Product
Quality Assurance
CM (2)
Configuration
Management
CAR (5)
Causal Analysis
and Resolution
OPF (3)
Organizational
Process Focus
OPD (3)
Organizational
Process Definition
OT (3)
Organizational
Training
OPP (4)
Organizational
Process Performance
OPM (5)
Organizational
Performance Management
7
THE PROCESS
1. Become familiar with CMMI, ASPICE, and INCOSE.
2. Conduct literature review to understand similar studies in the field.
3. Define gap analysis scope and technique.
4. Execute gap analyses.
5. Review gap analyses.
6. Analyze results.
7. Determine and document conclusions.
8. Develop appropriate proposed actions.
6 Analyses (bi-directional x3)
CMMI – 167 Specific Practices
ASPICE – 261 Base Practices
INCOSE – 110 Process Activities
8
ANALYSES VOLUME
CMMI
ASPICE INCOSE
9
EXAMPLE ANALYSIS – ASPICE TO CMMI
ASPICE
System Engineering
Process Group (SYS)
SYS.1 Requirements
Elicitation
SYS.1.BP1: Obtain
stakeholder
requirements and
requests.
Search CMMI
specific practices
for match to
SYS.1.BP1.
CMMI
Engineering
Requirements
Development (RD)
RD 1.1: Elicit
stakeholder needs,
expectations,
constraints, and
interfaces for all
phases of the
product lifecycle.
ASPICE SYS.1
Requirements Elicitation
ASPICE BP CMMI SP
SYS.1.BP1 RD 1.1
SYS.1.BP2
SYS.1.BP3
SYS.1.BP4
SYS.1.BP5
SYS.1.BP6
Coverage 16.7%
“ASPICE to CMMI” may also be
interpreted as CMMI coverage of
ASPICE.
1 0
CHALLENGES
ASPICE
Develop software detailed
design.
CMMI
Develop a design for the
product or product
component.
Gap
Match
ASPICE
Identify relevant
regulations.
INCOSE
Use expert to help
determine relevant
regulations if needed.
Gap
1. Related practices may make assumptions and/or omit key actions…
2. Similar practices may vary in specificity…
3. Identical practices may have different subjects. E.g. “Identify risks” in System Design does not map to
“Identify risks” in Acquisition.
• Major Gap Areas
• Acquisition
• Supply
• Documentation
• Reuse
• Consistent gaps found in most processes for ensuring consistency of process and communicating results to impacted groups.
1 2
ASPICE TO CMMIContract Agreement 57.1%
Supplier Monitoring 100.0%
Technical Requirements 70.0%
Legal and Administrative Requirements 0.0%
Project Requirements 33.3%
Request for Proposals 0.0%
Supplier Qualification 60.0%
Supplier Tendering 0.0%
Product Release 46.2%
Requirements Elicitation 66.7%
System Requirements Analysis 62.5%
System Architectural Design 75.0%
System Integration and Integration Test 55.6%
System Qualification Test 71.4%
Software Requirements Analysis 62.5%
Software Architectural Design 66.7%
Software Detailed Design and Unit Construction 62.5%
Software Unit Verification 71.4%
Software Integration and Integration Test 55.6%
Software Qualification Test 71.4%
Quality Assurance 100.0%
Verification 80.0%
Joint Review 50.0%
Documentation 12.5%
Configuration Management 88.9%
Problem Resolution Management 55.6%
Change Request Management 87.5%
Project Management 90.0%
Risk Management 100.0%
Measurement 81.8%
Reuse Process Group Reuse Program Management 37.5% 37.5%
Process Improvement
Process Group
Process Improvement
87.5% 87.5%
Total 59.1%
Management Process
Group
ASPICE to CMMI Gap Analysis
40.0%
28.6%
89.3%
66.0%
64.6%
65.8%
Acquisition Process Group
Supply Process Group
System Engineering Process
Group
Software Engineering
Process Group
Supporting Process Group
• Major Gap Areas• Project Monitoring, Integrated Project Management, and Quantitative Project
Management• Verification and Validation• Decision Analysis• Process Management
• Some gaps in Engineering were found as a result of ASPICE’s higher level of specificity.
1 3
CMMI TO ASPICE
Maturity Total
Requirements Management 100.0% Measurement and Analysis 62.5%
Supplier Agreement Management 66.7% Process and Product Quality Assurance 75.0%
Project Planning 57.1% Configuration Management 100.0%
Project Monitoring and Control 10.0%
Integrated Project Management 0.0% Requirements Development 50.0% Decision Analysis and Resolution 0.0% Organizational Process Focus 0.0%
Risk Management 85.7% Technical Solution 50.0% Organizational Process Definition 0.0%
Product Integration 66.7% Organizational Training 0.0%
Validation 40.0%
Verification 25.0%
Level 4 Quantitative Project Management 0.0% Organizational Process Performance 0.0% 0.0%
Level 5 Causal Analysis and Resolution 100.0% Organizational Performance Management 30.0% 53.3%
Total 39.5%
61.1%
CMMI to ASPICE Gap Analysis
40.7% 47.5% 7.9%66.7%
29.1%
Level 2
Project Management Process ManagementEngineering Support
Level 3
• Major Gap Areas
• Architecture Definition
• System Analysis
• Operation
• Maintenance
• Disposal
• Supply
• Portfolio Management
• INCOSE to CMMI has the highest coverage results in this study.
1 4
INCOSE TO CMMIBusiness or Mission Analysis 100.0%
Stakeholder Needs and Requirements Definition 100.0%
System Requirements Definition 100.0%
Architecture Definition 33.3%
Design Definition 100.0%
System Analysis 0.0%
Implementation 100.0%
Integration 100.0%
Verification 100.0%
Transition 100.0%
Validation 100.0%
Operation 0.0%
Maintenance 0.0%
Disposal 0.0%
Project Planning 100.0%
Project Assessment and Control 66.7%
Decision Management 100.0%
Risk Management 100.0%
Configuration Management 100.0%
Information Management 100.0%
Measurement 100.0%
Quality Assurance 100.0%
Acquisition 80.0%
Supply 0.0%
Life-Cycle Model Management 100.0%
Infrastructure Management 100.0%
Portfolio Management 33.3%
Human Resource Management 66.7%
Quality Management 66.7%
Knowledge Management 100.0%
Total 73.6%
77.8%
40.0%
96.4%
66.7%
INCOSE to CMMI Gap Analysis
Technical Processes
Technical Management
Processes
Agreement Processes
Organizational Project-
Enabling Processes
• Major Gap Areas• Project Monitoring, Integrated Project Management, and Quantitative
Project Management• Causal Analysis and Resolution• Organizational Process Performance and Performance Management
• CMMI to INCOSE gaps mostly involve self-evaluation and process improvement.
1 5
CMMI TO INCOSE
Maturity Total
Requirements Management 80.0% Measurement and Analysis 100.0%
Supplier Agreement Management 100.0% Process and Product Quality Assurance 100.0%
Project Planning 50.0% Configuration Management 100.0%
Project Monitoring and Control 30.0%
Integrated Project Management 20.0% Requirements Development 90.0% Decision Analysis and Resolution 100.0% Organizational Process Focus 55.56%
Risk Management 85.7% Technical Solution 100.0% Organizational Process Definition 57.14%
Product Integration 77.8% Organizational Training 42.86%
Validation 100.0%
Verification 62.5%
Level 4 Quantitative Project Management 28.6% Organizational Process Performance 0.00% 16.7%
Level 5 Causal Analysis and Resolution 0.0% Organizational Performance Management 30.00% 20.0%
Total 62.3%
72.2%
CMMI to INCOSE Gap Analysis
50.8% 85.0% 39.5%83.3%
69.8%
Level 2
Project Management Process ManagementEngineering Support
Level 3
• Major Gap Areas
• Acquisition and Supply
• Software Engineering
• Joint Review
• Problem Resolution
• Change Management
• Reuse
• Most of the ASPICE to INCOSE engineering gaps were found due to INCOSE not covering engineering standards specific to software.
1 6
ASPICE TO INCOSEContract Agreement 100.0%
Supplier Monitoring 80.0%
Technical Requirements 40.0%
Legal and Administrative Requirements 40.0%
Project Requirements 20.0%
Request for Proposals 25.0%
Supplier Qualification 80.0%
Supplier Tendering 12.5%
Product Release 23.1%
Requirements Elicitation 83.3%
System Requirements Analysis 87.5%
System Architectural Design 50.0%
System Integration and Integration Test 88.9%
System Qualification Test 71.4%
Software Requirements Analysis 62.5%
Software Architectural Design 50.0%
Software Detailed Design and Unit Construction 0.0%
Software Unit Verification 0.0%
Software Integration and Integration Test 0.0%
Software Qualification Test 0.0%
Quality Assurance 100.0%
Verification 100.0%
Joint Review 0.0%
Documentation 62.5%
Configuration Management 66.7%
Problem Resolution Management 0.0%
Change Request Management 0.0%
Project Management 70.0%
Risk Management 83.3%
Measurement 54.5%
Reuse Process Group Reuse Program Management 12.5% 12.5%
Process Improvement
Process Group
Process Improvement
87.5% 87.5%
Total
System Engineering Process
Group
76.3%
ASPICE to INCOSE Gap Analysis
Acquisition Process Group
46.7%
Supply Process Group
19.0%
45.0%
Software Engineering
Process Group
19.1%
Supporting Process Group
41.5%
Management Process
Group
66.7%
• INCOSE to ASPICE has the lowest coverage results of this study.
• The large quantity of major gap areas is mostly due to INCOSE using broad, systems engineering standards while ASPICE uses specific, automotive engineering standards.
1 7
INCOSE TO ASPICEBusiness or Mission Analysis 40.0%
Stakeholder Needs and Requirements Definition 66.7%
System Requirements Definition 75.0%
Architecture Definition 33.3%
Design Definition 50.0%
System Analysis 0.0%
Implementation 0.0%
Integration 66.7%
Verification 100.0%
Transition 0.0%
Validation 0.0%
Operation 0.0%
Maintenance 0.0%
Disposal 0.0%
Project Planning 0.0%
Project Assessment and Control 100.0%
Decision Management 0.0%
Risk Management 80.0%
Configuration Management 80.0%
Information Management 0.0%
Measurement 100.0%
Quality Assurance 80.0%
Acquisition 0.0%
Supply 0.0%
Life-Cycle Model Management 0.0%
Infrastructure Management 0.0%
Portfolio Management 0.0%
Human Resource Management 0.0%
Quality Management 0.0%
Knowledge Management 0.0%
Total
INCOSE to ASPICE Gap Analysis
Technical Processes
Technical Management
Processes
Agreement Processes
Organizational Project-
Enabling Processes
31.8%
0.0%
0.0%
60.7%
33.3%
CMMI ASPICE INCOSE
CMMI 39.5% 62.3%
ASPICE 59.1% 45.0%
INCOSE 73.6% 31.8%
Results Summary - Row to Column
• CMMI has the highest average coverage of the other standards (66.4%).
• ASPICE has the lowest coverage of other standards due to its higher specificity.
• INCOSE has weaker coverage of other standards due its lack of software-specific processes.
1 8
RESULTS SUMMARY
2 0
GAP ANALYSIS SUMMARY – CMMI, ASPICE, INCOSE
• Supply Management
(outgoing)
• Legal and Administrative
Requirements
• Documentation
Management
• Decision Analysis and
Resolution
• Life-Cycle Model
Management
• Human Resource
Management
• Knowledge Management
• Causal Analysis and
Resolution
• Joint Review
• Process Performance
• Integrated Project
Management
• Quantitative Project
Management
• Request for Proposals
• Reuse Program Management
• Software-specific engineering
• System Analysis
• Operation
• Maintenance
• Disposal
CMMI
Unique Inclusions
Unique Gaps
ASPICE INCOSE
2 1
WHICH GAPS ARE IMPORTANT?
Weighing the importance of each major gap identified will lead us to determining which standard(s) should be mapped in Stages, adopted as best practice, required of suppliers, and/or used for internal appraisals.
Some identified gaps may have little significance in our business objective while others are critical.
Note that the weight of some gaps may differ for OEMs and suppliers. (E.g. supply management)
2 2
NEXT STEPS
1. Weigh the Gaps
Weigh each of the items identified in the Gap Analysis Summary. This should include all relevant engineering process leaders. These weights should be used to help determine action items regarding which standard(s) should be mapped in Stages, adopted as best practice, required of suppliers, and/or used for internal appraisals.
2 3
NEXT STEPS
2. Determine Best Standard/Model
It is crucial to pick a process model to adhere to. Without doing so, future processes can only reference back to previous processes, greatly diminishing potential improvement and increasing risk. Process requirements are necessary for process improvement to focus on the “what” rather than the “how.”
Choosing a process model to adhere to may involve using selections from multiple sets of standards in combination. The best option/combination should be selected and reviewed regularly.
2 6
REFERENCES[1] Automotive SPICE PFM PAM v3.0
[2] INCOSE Systems Engineering Handbook 4e 2015 07
[3] CMMI for Development, Version 1.3
[4] Monzón, A. (2010, May). Bi-directional Mapping between CMMI and INCOSE SE Handbook. In Embedded Real-Time Software International Conference, Toulouse.
[5] Sabar, S. (2011). Software Process Improvement and Lifecycle Models in Automotive Industry.
[6] Baldassarre, M. T., Piattini, M., Pino, F. J., & Visaggio, G. (2009, May). Comparing ISO/IEC 12207 and CMMI-Dev: towards a mapping of ISO/IEC 15504-7. In Software Quality, 2009. WOSQ'09. ICSE Workshop on (pp. 59-64). IEEE.
[7] Sassenburg, H., & Kitson, D. (2006). A comparative analysis of CMMI and automotive SPICE. European SEPG, Amsterdam/Netherlands (June 2006).