![Page 1: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/1.jpg)
http://itconfidence2013.wordpress.com
Application Lifecycle Management and process
monitoring through an integrated and low-cost
solution, mainly based on Open Source Software Products
1°International Conference on IT Data collection, Analysis and Benchmarking
Rio de Janeiro (Brazil) - October 3, 2013
N. Bertazzo, D. Gagliardi, N. Bertazzo, D. Gagliardi, S. Oltolina,S. Oltolina, G. Ruffatti G. Ruffatti
Engineering GroupEngineering GroupInsert here a picture
![Page 2: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/2.jpg)
2IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
ALM and…
G1. Measuring company performances through an effective SPI program G2. Adopting a multi-dimensional performance model deployed mostly with 3D instancesG3. Implementing an integrated and low cost – OSS based – solution for the measurement and governance of software product & process quality
Presentation Goals
![Page 3: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/3.jpg)
3IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
Which is users’ & customers’ level of satisfaction?
How productive is myorganization?
Is there REALLY a way to measure performance?
Which is thequality level of my product?
How can I improve the development process?
How can I compare different labs?
Top Top ManagerManager
Is my project on track?
How can I improveperformance?
Project Project ManagerManager
Quality Quality ManagerManager
Which are corporate audits results?
Introduction The problem: Managers’ needs
![Page 4: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/4.jpg)
4IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
Engineering At a glance
www.eng.it
![Page 5: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/5.jpg)
5IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
Technical, Innovation & Research Division Engineering’s Software Labs (ESL)
PRODUCTION
ESL3: Application Management
ESL1-2: Project development
RFPs technical support
MANAGED OPERATIONSInfrastructures & System Services
Architectural design Research&
Development
CompetenceCenters
Resource management
Business Units (BUs) for different market sectors
AccountManagers
Sales Managers
ServiceDesk
Business Analysis
ProjectManagersBusiness
CompetenceCenter
Worldwide Customers
Background Engineering production organization Managers’ needs
![Page 6: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/6.jpg)
6IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
• Continuous Quality Improvement in Engineering's projects• Unified Infrastructure supporting quality processes granting flexibility and
adaptability• CMMI-DEV and ISO certifications, as independent method to validate the
compliance of processes and infrastructures with quality standards• Set-up of Engineering Software Labs (ESLs) to enhance and measure
productivity and improve quality practices
Background Compliance of SPI to quality systems
…
![Page 7: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/7.jpg)
7IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
Interaction & Information sources (quant. & qual. data)
![Page 8: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/8.jpg)
8IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
• Three dimensions of analysis:
1. Economical (EE)2. Social (SS)3. Technical (TT)
QEST model: 3 dimensions
•Performance values for each dimension allow to identify process areas that need improvements
![Page 9: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/9.jpg)
9IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
• Spago4Q, the open source platform to measure, analyze and monitor quality of products, processes and services
http://www.spago4q.org
The model• QEST nD model, a conceptual framework for measuring process
performance based on multiple analysis dimensions
http://www.semq.eu/leng/modtechqlm.htm
The tool
Productivity Intelligence
![Page 10: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/10.jpg)
10IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
The model (1): QEST
Method: Performance is expressed as the combination of the specific ratios selected for each of the 3 dimensions of the quantitative assessment (Productivity - PR) and the perceived product quality level of the qualitative assessment (Quality - Q)
Performance = PR + Q
Model: QEST (Quality factor + Economic, Social & Technical dimensions) is a “structured shell” to be filled according to management objectives in relation to a specific project
Such a model has the ability to handle independent sets of dimensions without predefined ratios and weights - referred to as an open model
![Page 11: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/11.jpg)
11IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
The model (2): geometrical indicators
It is possible to measure performance considering at least 3 distinct geometrical concepts:• Distance between the tetrahedron base center of gravity and the center of the plane section along the tetrahedron height – the greater the distance from 0, the higher the performance level;• Area of the sloped plane section – the smaller the area, the higher the performance level;• Volume of the lowest part of the truncated tetrahedron – the greater the volume, the higher the performance level.
• Target: measuring project performance (p) using 3 distinct viewpoints• Input Data: list of weighted ratios for each dimension and quality questionnaires• Output Data: an integrated normalized value of performance
![Page 12: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/12.jpg)
12IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
• Integrated quantitative and qualitative evaluation from 3 concurrent organisational viewpoints• A 3D geometrical representation at a single project phase (usually after the project is completed) •Use of de facto and de jure standards (e.g. ISO/IEC 9126 for the Quality Factor now ISO/IEC 25010:2011)• Performance Measurement Model to use for consolidating Balanced Scorecard (BSC) measurement outcomes• Extension of the original 3D model to n possible dimensions-perspectives QEST nD through the simplexsimplex as the mechanism to solve the problem from the 4th dimension on
The model (3): key features
![Page 13: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/13.jpg)
13IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
The tool: Spago4Q
![Page 14: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/14.jpg)
14IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
• QEST model is fully supported by Spago4Q
• The procedure is coherent with the PMAI (Plan-Measure-Assess-Improve) cycle: PLAN, defining a set of metrics, based on the GQM approach, and
possible dimensions of analysis (perspectives) characterizing the analysis
MEASURE, including the collection of data, and the computation of metric values and global performance value
ASSESS, presenting results through dashboards and reports IMPROVE, analyzing in detail each value that is less than the
expected thresholds, in order to find possible problems or bottlenecks from a process-based viewpoint
The integrated environment: QEST & Spago4Q
![Page 15: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/15.jpg)
15IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
The ESL model selected goals for each analysis dimension:
1. Economical (EE)E.G1 Reduce the effort of corrective maintenance (corrective + preventive, ISO/IEC14764:2006)E.G2 Improve ESL resource/assets allocation E.G3 Reduce effort due to hardware system unavailability (‘downtime’)E.G4 Reduce rework (Analysis/Design SLC phases)E.G5 Improve productivity (note: different ‘sizing’ units)
2. Social (S) S.G1 Reduce the number of non-conformity issues (QA inspection)S.G2 Improve artifacts reuse (functional reuse)S.G3 Evaluate training skills for organizational resourcesS.G4 Improve customer satisfaction (e.g. Customers/Prospects, Business Units, Developers) S.G5 Improve knowledge sharing (“social 2.0”, communities)
3.Technical (TT)T.G1 Reduce the resolution time for defects and technical issuesT.G2 Reduce the number of pre-delivery defects (as in ODC analysis) T.G3 Improve delivery time for deliverablesT.G4 Improve code qualityT.G5 Improve the testing process (e.g. coverages, # req’s, # tests, ...)
3D analysis: main goals
![Page 16: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/16.jpg)
16IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
Metric ID Metric Desc Formula SourceE.M1.1 Incidence of corrective maintenance effort Corrective Maintenance Effort/Development Effort ALM & prj registry
E.M2.1 Allocation of ESL resources Nr. of Res (hours) allocated on prj/Tot of Res (hours) ALM & Corp. Systems
E.M3.1 Hardware System Availability Percentage System Availability System Monitoring
E.M4.1 Incidence of rework Rework Effort / Development Effort ALM & prj registry
E.M5.1 Development capability FP/Effort ALM & prj registry
S.M1.1 n. Of Non Conformity issue % of NC for project ALM & QA Registry
S.M2.1 Incidence of artifact reuse Nr downloads/tot nr of artifacts stored Component repo
S.M3.1 Skill improvement % new (or modify) skills for resource Skill management tool
S.M4.1 Customer Satisfaction Results of survey Survey tool
S.M5.1 Knowledge sharing improvement % of interaction with collab. tools Collaboration tools
T.M1.1 Incidence of defects % nr. of defects (errors + defects) for project ALM
T.M2.1 Defects Mean Resolution Time Tot. resolution time/Tot. defects ALM
T.M3.1 Incidence of delayed deliverables % nr. delayed deliv. / Tot. deliverables ALM
T.M4.1 Code Quality Results of automatic static test Code analysis tool
T.M5.1 Testing process improvement Test coverage ALM
3D analysis: metrics
![Page 17: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/17.jpg)
17IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
ESL Chief Manager
ESL Lab Manager
Level 1
Level 2
Level 3
Project Manager (PM)
3D view: drill-down through the organizational levels
PRJ n
ESL
ESL 1 ESL 2 ESL 3
PRJ 1
PRJ n
PRJ 1PRJ n
PRJ 1
Project developmentProject development Application maintenanceApplication maintenanceProject developmentProject development
Engineering’s Software LabsEngineering’s Software Labs
Top Manager (TM)
![Page 18: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/18.jpg)
18IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
3D view: Level 1 - TM dashboard – sample #1
Unified view on Engineering Software Labs
Global performance indicator
Performance comparison (time, labs)
![Page 19: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/19.jpg)
19IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
3D view: Level 1 - TM dashboard – sample #2
![Page 20: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/20.jpg)
20IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
Metrics
Reqs & Bugs
RisksTasks & Issues
Docs
Detailed view
Tracking and trends
A single dimension view: Level 3 - PM dashboard
![Page 21: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/21.jpg)
21IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
A single dimension view: Level 3 - PM ALM tool
![Page 22: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/22.jpg)
22IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
A single dimension view: Levels 1/2 - QA audits
![Page 23: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/23.jpg)
23IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
Users & customersfeedbacks arenow integrated
with corporate data
Finally I know how productive myorganization is!
Finally we can REALLY measure performance!
I can monitor thequality level of my product
The development processis under control and
I can improve it!Now I can compare Labs performance!
Top Top ManagerManager
I know if my project is on track & I can
identify issues
Conclusions
Productivity Intelligence enables performance
improvement
Project Project ManagerManager
Quality Quality ManagerManager
Through audit dashboards, corporate QA
is under control
![Page 24: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/24.jpg)
24IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
• Improve reports and KPIs Introduce/improve reports for new/modified information needs Dynamical update for thresholds
• Integrate tools to collect soft factors measures and indicators from two categories of ESL customers using on-line surveys: “external customer”: feedback on the perceived product quality “internal customer”: feedback on the actual process quality from
project managers of the various Business Units• Build new KPIs on data coming from the Infrastructure
Enhancing Project that collects issues and suggestions by the ALM users e.g. in the BSC ‘Learning & Growth’ perspective
Next steps
![Page 25: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/25.jpg)
25IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
25
Buglione L., Misurare il Software. Quantità, qualità, standards e miglioramento di processo nell’Information & Communication Technology, FrancoAngeli, 3/ed, 2008, ISBN 978-88-464-9271-5
Buglione L. & Abran A., QEST nD: n-dimensional extension and generalisation of a Software Performance Measurement Model, International Journal of Advances in Engineering Software, Elsevier Science Publisher, Vol. 33, No. 1, January 2002, pp.1-7
Spago4Q website and resources: www.spago4q.org
Contacts & Info: [email protected]
References
![Page 26: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/26.jpg)
26IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
Q && A
Muito obrigado pela vossa atençaoMuito obrigado pela vossa atençao!!Thanks for your attentionThanks for your attention!!
ALM and...
![Page 27: Application Lifecycle Management and process monitoring ... · • CMMI-DEV and ISO certifications, as independent method to validate the ... •Use of de facto and de jure standards](https://reader030.vdocuments.site/reader030/viewer/2022013009/5e7b92286257e940c97e5452/html5/thumbnails/27.jpg)
27IT Confidence 2013 – October 3, 2013 http://itconfidence2013.wordpress.com
Our Contact Data
NicolaBertazzo
DanieleGagliardi
GabrieleRuffatti
Sergio Oltolina
ALM and...