defining quality models for agile projects

FP7-ICT-2011-8. Project #: 318082

Defining Quality Models for Agile Projects

FP7-ICT2011-8. Project #: 318082

Contents

• A few words about speakers’ biographies and companies• The U-QASAR Project • What to Measure in Agile• Practical Workshop: defining a Quality Model for one Agile

company

FP7-ICT2011-8. Project #: 318082

The U-QASAR Project (1)

• U-QASAR. An EU funded research project under FP7.

• 8 organizations from Finland, Norway, Germany, Luxembourg and Spain: Experts in SW Quality, Ontology modeling and implementation, SW Engineering methods, and SW Measurement.

FP7-ICT2011-8. Project #: 318082


• Objective: Create a solution to provide project managers, software engineers, testers and other actors with an objective measurement of quality for:

– Software Development processes– Software Products

FP7-ICT2011-8. Project #: 318082

Methodology + Platform:


FP7-ICT2011-8. Project #: 318082

“If you can not measure it, you can not improve it…”

Lord Kelvin

What to Measure in Agile (1)

FP7-ICT2011-8. Project #: 318082


Defect count, test success rate, test failure rate technical debt in categories, technical debt in effort

velocity, completed web pages, schedule performance index, business value delivered,

customer satisfaction, defect count after testing, violations of static code analysis, build status…

FP7-ICT2011-8. Project #: 318082


• What Metrics are Agile teams using in practice?:– Metrics proposed by Agile literature + Custom Metrics.– Measuring Products and Processes.– Reasons for and effects of using metrics:

• Iteration and Project Planning• Iteration and Project Progress Tracking• Software Quality Measurement• Fixing Software Process Problems• Motivating People

FP7-ICT2011-8. Project #: 318082


Iteration and Project Planning• Why

– Prioritization of tasks– Scoping the Iteration/Release– Effort prediction

• Metrics– E.g Effort estimate, actual velocity of previous

iteration, team’s available hours

FP7-ICT2011-8. Project #: 318082


Iteration and Project Progress Tracking• Why

– Monitoring progress– Increase Visibility for all stakeholders– Achieving goals (Are we going to make it?)

• Reduce scope, add more resources

– Balance workflow (prevent overloading people)

• Metrics– Velocity / Burndown, completed work items

FP7-ICT2011-8. Project #: 318082


Understanding and Improving Quality• Why

– Understand and increase quality of the product– Prevent defects reaching customer– Predict and Evaluate post release quality– Ensure the level of testing

• Metrics– Technical debt, maintenance effort, defect deferred, N

of change requests, test coverage, critical defects sent by customer

FP7-ICT2011-8. Project #: 318082


Fixing Software Process Problems• Why

– Identify and predict problems– Identify phases where no value is added

• Metrics– Lead time, processing time, cost types, story

percentage completed, fix times of failed build, violation of static code analysis, queue time, check-ins per day

FP7-ICT2011-8. Project #: 318082


Motivating People• Why

– Motivate people to react faster and improve• E.g Fix build faster, fix bugs faster, increase unit testing

• Metrics– E.g Fix build faster, fix bugs faster, increase unit

testing, defect trend, work in progress

FP7-ICT2011-8. Project #: 318082


Entities AttributesProducts Internal External

Products running tested features, build statuscustomer satisfaction, progress as working code

Test plans number of test cases

Codetechnical debt in categories , technical debt in effort, violations of static code analysis

Featurestask's expected end date, task done, effort estimate, story complete percentage

business value delivered

Requirements requirement's cost types, percentage of stories prepared for Iteration

Defectsdefect trend indicator, predicted number of defect

N. E. Fenton and S. L. Peeger. Software metrics: a rigorous and practicalapproach. PWS Publishing Co., 1998.

Categorisation of Metrics by Fenton and Pfleeger

FP7-ICT2011-8. Project #: 318082


Entities AttributesProcesses Internal External

Testingdefect count, test success rate, test failure rate, defects deferred, test coverage, test growth ratio

number of bounce backs, fault slips

Implementation

velocity, number of unit tests, completed web pages, cost performance index, schedule performance index, planned velocity, common tempo time, check-ins per day, fix time of failed build

story flow percentage

Reqs.engineering velocity of elaborating features

WholeDevelopment

Cycle

cycle time, lead time, processing time, queuetime, maintenance effort, work in progress, variance in handovers, throughput, queue, implemented vs wasted requirements

ResourcesTeam team effectiveness

Customer revenue per customer

N. E. Fenton and S. L. Peeger. Software metrics: a rigorous and practicalapproach. PWS Publishing Co., 1998.

Categorisation of Metrics by Fenton and Pfleeger

FP7-ICT2011-8. Project #: 318082

Practical Workshop:

Defining a Quality Model for anAgile company

FP7-ICT2011-8. Project #: 318082

Defining a Quality Model (1)

• Create a Quality Model– It can be based on existing standards or models.– It can be an organization’s self-made model.– It can deal with process, products, or both.

Creation of a model consists of identifying expected values, weights and thresholds for:

• Quality Objectives, or aggregation of indicators to evaluate the achievement of an objective.

• Quality Indicators, or aggregation of metrics used to evaluate• Quality Metrics, or raw measurements obtained automatically or

manually

FP7-ICT2011-8. Project #: 318082


• Create Quality Objectives (QOs) or Select existing ones.

• Select or create Quality Indicators (QIs) by aggregating metrics.

CONTEXT AWARENESS & NEGOTIATION• Weights, Thresholds and Values are negotiated among the

participants of the QA process• QOs’ and QIs’ are adapted along the development LC (e.g. a feature

is identified as critical for the customer in the 3rd Iteration).

FP7-ICT2011-8. Project #: 318082

Iteration 1

Iteration 2

Iteration N


QM 1.0

QM 1.1

QM v.r

Adapted along the development LC (e.g. a feature is identified as critical for the customer in the 3rd Iteration).

All actors negotiate QM, Weights, Thresholds and Values

FP7-ICT2011-8. Project #: 318082

Example of a Quality Model selected for an specific Project:


FP7-ICT2011-8. Project #: 318082

Case Study (1)

• Spanish company founded in 2014• Mobile applications development• Agile methods• SME: 10 employees

• We need to manage Quality of our:- Processes- Products

FP7-ICT2011-8. Project #: 318082

Case Study (2)

Help us to define our Quality Model!!

FP7-ICT2011-8. Project #: 318082

How: Methodology

• Method 1: Quality Objectives Settings Method

• Method 2: Planning a Measurement Program

• Method 3: Defining the Metrics: Goal/Question/Metric

• Method 4: Data Collection• Method 5: Reliability and Validity of

the Measures

Iteration 1

Iteration 2

Iteration N

FP7-ICT2011-8. Project #: 318082

Method 1: QO Setting Method (1)• Inputs :

– Stakeholders & ExpertiseManagers-decision-makers, Developers,Quality assurance, Customers, Project managers…

– Quality Model:ISO 9126 ISO/IEC 25000

– Existing measurement data:Raw data of result of test cases in Excel documents

FP7-ICT2011-8. Project #: 318082

Why all the stakeholders….

managers-decision-makers => holistic vision of the Company

developers => are the requested goals measurable?

Project managers => knowledge of historical projects

Customers => we look at customer satisfaction

Method 1: QO Setting Method (2)

FP7-ICT2011-8. Project #: 318082

Everybody is committed to achieve the Quality Goals

Method 1: QO Setting Method (3)

FP7-ICT2011-8. Project #: 318082

Method 1: QOSM• Step 1: Preparation and pre-assignment

– Setting the Context (Example study case)– Setting the Roles (Roles involved in the process)– Scheduling a workshop (Now)– Pre-assignment to participants (5mins)

Test process conformance

Usability

FP7-ICT2011-8. Project #: 318082

Method 1: QOSM• Step 2: Brainstorming

– Group brainstorming (20 mins)– Forming objectives on the wall (15 mins)

FP7-ICT2011-8. Project #: 318082

Method 1: QOSM• Step 3: Prioritizing

– Voting (10mins)

FP7-ICT2011-8. Project #: 318082

Method 1: QOSM• Step 4: Elaborating

– Top-5 selection– Detailed description (1 Quality Objective):

Name Short name

Description Description on higher level

Rationale Motivates why the objective is important. Gains and risks; E.g., costs, business benefits, customer viewpoint

Related factors Tentative ideas of what contributes, or prevents achieving this objective

Votes Number of votes in prioritization

One or more quality indicators Measures that can be used indicate if this objective is achievedFor each quality indicator:- Description- Current value of the measure- Target level to achieve the objective- Deadline - Continuous or stepwise

Easiness of collecting We have it / Requires some work / Impossible

FP7-ICT2011-8. Project #: 318082

Method 1: QOSM• Step 4: QO detailed description

Name SW easy to learnDescription How easy is for the end user to learn how to use the application

Rationale End user experience. Customer satisfaction

Related factors Knowledge of the end user. Accessibility. Device

Votes 8

One or more quality indicators Effectiveness of ‘help’: easiness of finding help functions- Not measured yet- Satisfaction > 85%- 5 months- Stepwise: Per each prototype delivered to the ownerTime to learn: How long it takes to learn the application- Not measured yet- Time < 5 days- 5 months - Stepwise: Per each project finished

Easiness of collecting Requires some work (For example, usability surveys)

FP7-ICT2011-8. Project #: 318082

Method 1: QOSM

• Step 5: Post-Workshop QO– Completion of the description of the Quality

Objective– Digging out the existing data on current level– Communicate the Quality Objectives

FP7-ICT2011-8. Project #: 318082

Method 2: Planning measurement program

• Identifying Quality Indicator types:– Purpose of the Measurements– Measurability (numbers, human assessment, not

measurable)– Automation of data collection– Target of the measurement– Frequency of the Feedback– Organizational Level of Information Use

• Planning the use of data

FP7-ICT2011-8. Project #: 318082

• Quality Indicator definitionName Effectiveness of help

Purpose Evaluate the easiness of finding the help functions

Measurability The indicator it is going to be measured by both: human assessment and quantitative numbers. It could be evaluated by means of:- Satisfaction of the end user (Survey) - Time to find the help functions

Degree of automation None

Target Product

Frequency of feedback Whenever a SW project is delivered to client

Organizational level of information use

Developer Manager

Method 2: Planning measurement program

FP7-ICT2011-8. Project #: 318082

Method 3: Defining the Metrics

• Method 3: Defining the Metrics: Goal/question/metric

FP7-ICT2011-8. Project #: 318082

Next Steps: Methods 4 & 5

• Method 4: Data collection• Successful measurement activities• Manual or automatic data collection

• Method 5: Reliability and validity of the measures• Assessing measurement validity• Evaluating proposed metrics

FP7-ICT2011-8. Project #: 318082

Many thanks !!!

facebook.com/UQASAR

twitter.com/UQASAR

linkedin.com/groups/ UQASAR-Project-4682307

www.uqasar.eu

Contact us!