the sqale method: presentation

Source Code Quality Evaluation:The SQALE method

December 2011

Author: Jean-Louis Letouzey

SQALE: Software Quality Assessment based on Lifecycle Expectations

Agenda

The needs for a Source Code evaluation method

Issues with current aggregation rules

The SQALE method structure

The SQALE Quality Model

The SQALE Analysis Model

The SQALE indices and indicators

SQALE in practice

2

The SQALE Method: Summary

Has been developed by experts, independent of any tool vendor

Focus on the diagnostic objectivity (precision, no false-positive)

Easy to understand, to implement and to deploy

Avoids practices that damage measurement results

Aggregation with averages that generates compensation effects

Notation on a delimited interval that generates threshold effect

Promotes simple principles

Source code quality is a non functional requirement that should be specified, then verified

Evaluating quality is measuring the remaining needed workload to fix all non-conformities

Is a robust method for identifying and managing the Technical Debt

Software Quality Assessment based on Lifecycle Expectations

5

inspearit and source code evaluation

inspearit is not a tool vendor, inspearit is an independantcompany

inspearit thinks that the method come first, then the toolsto support it

inspearit assist large accounts to implement source code analysis with SQALE

Our customers reported us issues with current methods and tools

Difficulties to understand the meaning and usage of indicators

Too much false positive

No support for remediation decision (what are the priorities?)

6

The needs for a source code evaluation method

Ability to objectively evaluate and monitor software development products in order to anticipate issues

Aligned with best measurement practices

Ability to compare

Source code versions

Different products with different usage/history

Development teams or subcontractors performance

Capacity to provide useful inputs to an improvement plan

Capacity to support decisions: Ex. Two teams working on two similar projects

The first one, delivered 3 weeks in advance but with 100 coding practice issues

The second, delivered 1 week in advance but with 15 coding practice issues

Which project is the most efficient and effective?

7

High level requirements for an evaluation method

The SQALE method has been developped as a solution to all these requirements

Quantified, Objective, Precise, Sensitive

Implementable by automated static analysis tools

Reproducible by the implementation of one tool to another (produce the same findings based on non ambigous definitions, rules…)

Provide guidance for tailoring this standard model to:

Any language

Different severity levels (business critical, life critical etc.)

Agenda







SQALE in practice

8

9

A Hierarchy of Qualitychar. and sub-char.

= Quality Model

QH

Q

Measure/rule

Sub-characteristic

CharacteristicMaintainability

A Hierarchyof Artifacts

A

Component

AppliA

AH

Summary of the challenge for an evaluation method

Challenge / Need:Provide a quality related “measure” or “score” for each couple {A,Q} of the 2 hierarchies

Two hierarchies:

File Get a measure, a score which characterizes, represe nts the evaluated concept

Portfolio

Domain

Quality

10

Measurement basics: The representation condition

“The condition that, if one software entity is less than another entity in terms of a selected attribute, then any software measure for that attribute must associate a smaller number to the first

entity than it does to the second entity” [1]

Real World Mathematical World

Aa

Ab

Mes(Aa)

Mes(Ab)

Measurement function

Impact on :- Measure/rule choice- Normalization functions- Aggregation rules

Ac

Attribute

Mes(Ac)Art

ifact

s

Mes(Aa) > Mes(Ab) Mes(Ab) > Mes(Ac )

A representative measure keep the

relationship established in the

real world Aa >> Ab >> Ac

[1] N.E. Fenton and S. L. Pfleeger, Software Metric s: A rigourous & Practical Approach, second edition, ISBN 053495425-1,PWS Publishing Comp any, Boston, 1997

11

The representation condition applied to aggregation

Issues:

The aggregation should represent the basic findings

Issues should be reported up to the highest level of the hierarchy

Aggregationrule

?

ImprovementVersion a Version b

AgScore_a AgScore_b The aggregate score should report the improvement

Version b >> Version aAgScore_b > AgScore_a

We have identified some effects that violate this c ondition

12

The masking effect

The masking effect appears when the aggregate value is not sensitive to the variation of one of the base values

Example:

File 1

File 2

File 3

File 4

File 5

Min Max Media

n

MyAppli Va A A C E E A E C

MyAppli Vb A A C D E A E C

An improvement from Va to Vb on file 4 does not impact the aggregate score

�� Aggregations by Min, Max and Median violate the rep resentation condition

13

The compensation effect

The compensation effect appears on aggregation functions such as: mean, weighted mean, median

Example:

Comment ratio (target for the project : >30% per file)

File 1

File

2File

3File

4File

5

File 6

File 7

File 8

File 9

File 10

MyAppli Va 5% 34% 48% 47% 31% 37% 33% 35% 4% 39%

Avera

ge

31%

While 2 files do have “maintainability” issue, the average is OK

In real life, lack of comment in files 1 & 9 won’t be compensated by abundance of comments in file 3 & 4

That ‘s one reason why most Quality Dashboards are not precise

�� Aggregations by average (weighted or not) violate t he representation condition

14

The type of scale and allowed aggregations

The measurement theory is precise about allowed aggregation [1]

Scale Valid Transformation Main Valid AgregationNominal 1 to 1 mapping NoneOrdinal Monotonic increasing

functionMin, Max, Median

Interval M' = aM + B (a>0) Min, Max, Median, Average

Ratio M' = aM (a>0) Min, Max, Median, Mean, Average, Sum, Distance (Euclidian or other)

Absolute M' = M All

Due to the representation condition, some combinati ons (scale type,aggregation) should be rejected

[1] N.E. Fenton and S. L. Pfleeger, Software Metric s: A rigourous & Practical Approach, second edition, ISBN 053495425-1,PWS Publishing Comp any, Boston, 1997

More C

hoice

15

Synthesis of our analysis

Synthesis of allowed operations and aggregation issues depending on scale type

Within SQALE, we choose to normalize all measures o n a ratio scale andto aggregate the normalized values by summation

Scale Min, Max, medianAverage, Weighted

averageSum, Distance

NominalNot allowed Not allowed Not allowed

OrdinalPotentially not representative

Not allowed Not allowed

IntervalPotentially not representative

Potentially not representative

Not allowed

RatioPotentially not representative


Representative

AbsolutePotentially not representative


Representative

Agenda







SQALE in practice

16

17

The SQALE method: Structure

Implementation/Tools

8 Fundamental Principles

4 concepts

Tailoring

Measurement theory and representativity

18

The SQALE 8 Fundamental Principles

1. The quality of the source code is a non-functional requirement

2. The requirements in relation to the quality of the source code have to be formalised according to the same quality criteria such as any other functional requirement

3. Assessing the quality of a source code is in essence assessing the distance between its state and its expected quality objective

4. The SQALE Method assesses the distance to the conformity with the requirements by considering the necessary remediation cost of bringing the source code to conformity

5. The SQALE Method respects the representation condition

6. The SQALE Method uses addition for aggregating the remediation costs and for calculating its quality indicators

7. The SQALE Method’s Quality Model is orthogonal

8. The SQALE Method’s Quality Model takes the software’s lifecycle into account

19

The 4 main concepts of the SQALE method

Source Code

Static analysis tools

FindingsTable

Remediationcosts table Aggregation

rules

SQI

STISRI…

SQID…

1 Quality Model 2 Analysis Model Indicators43 Indices

List of source code related requirements

Rem

ediationfunctions

Testabilité Fiabilité Evolutivité Efficacité Maintenabi lité

Maintenabilité 589

Efficacité 248 248

Evolutivité 1 480 1 480 1 480

Fiabilité 548 548 548 548

Testabilité 6 535 6 535 6 535 6 535 6 535

6 535 7 083 8 563 8 811 9 400

Agenda







SQALE in practice

20

21

Back to the fundamentals of Quality

You buy a new car

• How will you feel, if the delivered car has only 5 cylinders and 290 hp?

22

Back to the fundamentals of Quality

In 1979, Philip Crosby in his famous book “Quality is free” established the 4 principles of Quality:

the definition of quality is conformance to requirements

the system of quality is prevention

the performance standard is zero defects

the measurement of quality is the price of nonconformance

Since that time, the vision and definition of quality has been extended to a much wider scope including customer satisfaction

But anyway, quality is still at least “Conformance to requirements”

23

The SQALE Quality Model: source code requirements

An organized set of expectations (requirements)based on lifecycle needs

Reuse

Maintain

Deliver

Evolve

Test

Code

Testability

Reliability

Changeability

Efficiency

Maintainability

Reusability

Security

Portability

Architecture related reliabilityArchitecture related reliability

Fault toleranceFault tolerance

Logic related reliabilityLogic related reliability

Instruction related reliabilityInstruction related reliability

Data related reliabilityData related reliability

UnderstandabilityUnderstandability

ReadabilityReadability

Ram related efficiencyRam related efficiency

Rom related efficiencyRom related efficiency

CPU related efficiency CPU related efficiency

Archi. related changeabilityArchi. related changeability

Logic related changeabilityLogic related changeability

Data related changeabilityData related changeability

Unit Testing testabilityUnit Testing testability

Integration Testing testabilityIntegration Testing testability

TestabilityTestability

ReliabilityReliability

ChangeabilityChangeability

EfficiencyEfficiency

SecuritySecurity

MaintainabilityMaintainability

PortabilityPortability

ReusabilityReusability

Number of derived class <=10no public data

…

…

…

Characteristic

Sub-characteristic

Requirement

Requirements , appear only once within the Quality Model, when they are first needed. They are checked with relevant static analysis tools

Requirements: type of issues

It is important to use a SQALE Quality Model that covers all the types of code issues

24

Copy and Paste, internal structure of methodsCopy and Paste, internal structure of methods

Potential logic errors, exception management,

test coverage

Potential logic errors, exception management,

test coverage

Excessive coupling, Hard coded dataExcessive coupling, Hard coded data

Useless code,un-optimized codeUseless code,un-optimized code

Presentation, structurnessPresentation, structurness

When deployed, the SQALE Quality Model contains from 30 to 100+ requirements tailored to the organization context






Agenda







SQALE in practice

25

26

Rem

ediationfunction

Remediation cost table


Quality Indexes represent the remediation effort needed to refactor artifacts in order to comply with the Quality Model

Tool

Analysis

4.1“Understandability index” for the selected file

“Characteristic indexes” are aggregated by adding “ Subcharacteristic indexes”

Non conformity table

Part level indexesPart level indexes are aggregated by adding all fil e indexes

Σ

Sub characteristic

indexes4.1

Σ

27

Req_1 Req_2 Req_3 Req_4 Req_5 Req_6 … … … … Req_99File_1File_2

File_3File_4……………File_99999

Remediation costs table


For a given couple {A,Q}, SQALE provides a simple rule to calculate the associated score

A

Component

AppliA

File

Portfolio

Domain

Q

Measure/rule

Sub-characteristic

CharacteristicMaintainability

Quality

The positions into the 2 hierarchies define the per imeter of remediation costs to be added

Σ

28

The SQALE Analysis Model: remediation factors

How findings are transformed into costs?

• SQALE use « Remediation Functions » that are associated to types of Non Conformity. The standard SQALE Analysis Model contains 5 types which correspond to different « remediation lifecycles »

• These Types and values are proposed by default. I it is recommended to extend /taylor them at Organization/Project/Application level

Estimated cost for fixing one Non Conformity of Typ e4 is: 5 Work Units

Agenda







SQALE in practice

29

30

3° concept: SQALE indices

The SQALE characteristic indices:

- SQALE Testability Index: STI- SQALE Reliability Index: SRI- SQALE Changeability Index: SCI- SQALE Security Index: SSI- SQALE Efficiency Index: SEI- SQALE Maintainability Index: SMI- SQALE Portability Index: SPI- SQALE Reusability Index: SRuI

The global SQALE Quality Index: SQI

- This is the Technical Debt

SQALE index densities: SQID, STID etc

Artifact remediation workload for all requirements associated to testability

Example: Density by KLOC(1,000 lines of code)

31

4° concept: The 3 SQALE indicators

SQALE Indices

SQI

STISRI…

SQID…

Rating Grid

SQALE Kiviat

Testabilité Fiabilité Evolutivité Efficacité Maintenabi lité

Maintenabilité 589

Efficacité 248 248

Evolutivité 1 480 1 480 1 480

Fiabilité 548 548 548 548

Testabilité 6 535 6 535 6 535 6 535 6 535

6 535 7 083 8 563 8 811 9 400

Based on the ratio (in %)Remediation cost / Development cost

Example of Testability ratingRemediation cost (STI): 4.36 hoursDevelopment cost: 250 hoursRatio: 1.7 % �� Rating: “C”

32






The SQALE Pyramid: A two points of view indicator

An analytic view provided by orthogonal characteristicsOne understands impact of each Non Conformity and improvement on quality characteristic and life cycle issues.

© Copyright inspearit 2007-20011

Σ

Σ

Σ

An external view that represents the percieved quality evaluated by consolidation of the hierarchy of characteristics

Σ

Interpreting the SQALE indices

33

The perceived benefits

As SQALE requires to specify the quality of the code, the quality measure is objective

The SQALE quality index represents a workload, a cost. It is the concrete “Technical debt” of the project

It is easy to monitor simultaneously:

the remaining workload associated to functionalities

the debt associated to code quality

and update accordingly the project‘s planning

Technical debt may be aggregated at any portfolio granularity

Technical debt density allows to compare versions, applications, subcontractors…

As SQALE does not violate the representation condit ion, SQALE thus provides usable meanings to source code measurement s

Agenda







SQALE in practice

34

35

Using SQALE: Tools

PRIVATE, METRIXWARE:

SQALE index distribution analysis

36

Using SQALE: Tools

SQUORE: Dashboard

37

Using SQALE: Tools

SONAR: Monitoring the source code quality(Technical Debt)

38

Using SQALE: Tools

SONAR: Tailored Dashboard

39

Using SQALE: Tools

SONAR: Portfolio Management

40

Using SQALE: Deployment

0 Initialization 1 Tailoring Deployment32 Implementation

Planification Stake holders Perimeter …

Method trainingDevelopment of Tailored SQALE models Specify the tool solution

Choose and Implementation ofthe solution

Tool Deployment, training coaching Monitor and improve

Identify the sponsor and stake holders, define the roadmap

Define the most usefull use cases of source code analysis:

Build the « Source code analysis » Project team

Perform awarenesssession

Coach and support the users

Monitor the solution and define an improvementplan (identification and implementation of new requirements)

Choose the solution to be implemented withinthe organization

Implementation of the tailored models withinthe selected solution

Implementation of the selected indicators and reports within the selected solution

Update process, associated deliverablesand training to preparedeployment

On day training for the team: The SQALE Method

Develop a tailoredSQALE Quality Model

Develop a tailoredSQALE Analysis Model

Validation of bothmodels trough a pilot project

Specify the toolsolution includingrecommandedindicators

Training,Workshop preparation and animation

Support SeminarsTrainingCoaching Support

Support

41

The SQALE Discovery Kit

Discover the fundamentals principles and benefits of the SQALE method with the“SQALE Discovery Kit”. This package contains:

A one day training session on the SQALE Method

The identification (through dedicated interviews) of your main “use cases” of source code analysis within your organisation’s context

The development (through dedicated Workshops) of your own quality and analysis models

These models will be tailored to your environment and will be the basis for defining and evaluating the quality of your source code (for one of the following language: Java, C, C++, Cobol)

The concrete assessment of one of your application using the SQALE method and your tailored quality and analysis models including a detailed assessment report

Workshop on how to interpret and use the results

At the end you will get:

Your tailored models for one of your development language

An evaluation report

Direction for actions

Total duration: about 20 days

Thanks

To know more about SQALE: http: /www.sqale.orgTo do more with SQALE, contact inspearit

Saturday, December 17, 2011

© inspearit - Author - Place

42

just sqale it

the sqale method: presentation

Technology