software reliability cmm-dfss
DESCRIPTION
Presentation given during a software reliability seminar organized by Holland Innovative BV - subject was the combination of CMM and DFSS within Philips Consumer LifestyleTRANSCRIPT
Guy Van HooveldCTO Office
Deployment of DFSS in software – experience within Philips Consumer Lifestyle
Guy Van Hooveld 2March , 2008
Objectives of this presentation
• Give insight in two different approaches to apply DFSS in a software environment with Philips Consumer Lifestyle– Bruges TV experience – link between DFSS and CMM L4– Bangalore experience – link between DFSS and
milestone driven software project management• Conclude on the gains and limitations of those
approaches based on the current experience.
Guy Van Hooveld 4March , 2008
CMM model ______ KPA _____ ( Key Process Area )
CMM Level 1
CMM Level 2 RM Requirements Management SLC Software lifecycle PP Project Planning PTO Project Tracking and Oversight SM Subcontract Management QA/SCV Quality Assurance/Software Compliance Verification SCM Software Configuration Management
CMM Level 3 OPF Organisation Process Focus
OPD Organisation Process Definition TP Training Program IM Integrated Management PE Product Engineering IC Intergroup Coordination PR Peer Reviews
CMM Level 4 QPM Quantitative Process Management
SQM Software Quality Management CMM Level 5 DP Defect Prevention
TCM Technology Change Management PCM Process Change Management
Guy Van Hooveld 5March , 2008
CMM model
Level 4 SQM : definition
The purpose of Software Quality Management is to develop a quantitative understanding of the quality of the project's software products and achieve specific quality goals.
Software Quality Management involves defining quality goals for the software products, establishing plans to achieve these goals, and monitoring and adjusting the software plans, software work products, activities, and quality goals to satisfy the needs and desires of the customer and end user for high quality products.
Guy Van Hooveld 6March , 2008
Quality goals to satisfy the needs and desires of the customer and end user for high quality products.• The reference framework in the software is CMM (L4)• Theoretically no actual DFSS deployment as such in the software, but:• Use of VOC to derive requirements• Use of derived CTQs to measure software quality• Focus on non-functional requirements• CMM philosophy applied + classical software reliability engineering applied
– Fault prevention• Reviews• Requirements management• FMEAs / risk management• Architecture
– Fault tolerance• Reboots / watchdogs• …
– Fault detection/correction• Verification/validation• Root cause analysis with feedback loop etc…
• In fact a lot of commonality with DFSS, close to an actual DFSS deployment.
Guy Van Hooveld 7March , 2008
What is SW Quality ? (choices made for a specific project)
– Stability– Critical resources : e.g.
• Memory Budget tracking• CPU load, • latency, • bandwidth
– Performance : e.g. startup time, zapping time, UI
– SDE warnings : Koala, compiler– QAC warnings– Asserts– Debugging– NFR conformity (validation)– Reviews : Defect densities
Guy Van Hooveld 8March , 2008
SQM in a TV project
Step 1. - create plan :
SW_Quality_attribute_baseline
Step 2. - deploy the planmeasurements & reporting ( by product, subsystems, suppliers)
Step 3. - corrective actions on deviations
Guy Van Hooveld 9March , 2008
Plan for a TV project : overview of CTQ’s
CTQ metrics on the Plan Product Platform
Subsystem
Stability x
Critical res. : Memory Budget tracking
x x
Critical res.: CPU load, latency, bandwidth
x x
Performance : startup, zapping, UI
x x
SDE warnings : Koala, compiler x x
QAC warnings x x
Asserts x x
Debugger warnings x
NFR conformity (validation) x
Defect densities x
Guy Van Hooveld 11March , 2008
TV project : quality parameters Reporting
SDE Warnings
0
10
20
30
40
412 413 414 415 416 417 418 419 420 421Koala Warnings Compiler Warnings
Target Koala Target Compiler
Example :
Guy Van Hooveld 12March , 2008
Right design vs design right
• Right design (what does the consumer want ?)– DFSS in requirements process– Requirements process – functionality vs quality– Support from development to marketing
• Design right (how do we implement it correctly ?)– Software processes– DFSS in development
Guy Van Hooveld 13March , 2008
Lessons learned
• The CMM L4/DFSS approach has worked to improve quality• Approach made non-functional requirements more visible• When marketing is not capable of providing exact requirements (logical
for technical constraints for instance), development can help suggesting CTQs to start discussion.
• FMEAs not efficient enough in the software (risks can be listed but more difficult to make things measurable/w tolerance cf mechanical/electrical)
• A number of measurements reflect mainly process quality, not product quality (defects density)
• Supplier management: – improve control over critical parameters from our suppliers : Stability, NFR,
robustness.– for some SW-suppliers this is handled as well as possible through provisions in
requirements (NFR) and work agreements (measurement & reporting)– for others : there are still examples where this is difficult to achieve.
• SW CTQ's (and NFR's) are twofold :– those that contribute mainly to end-product (Stability, performance)– those that contribute mainly to development trajectory (warnings…)– Outside SW, only CTQ's contributing to the end-product are considered
Guy Van Hooveld 15March , 2008
Approach in Bangalore
• More classical DFSS deployment (cf Bruges example)• Alignment between DFSS DIDOV phases and project
milestones• Translation of DFSS to software tasks• This has been applied in numerous projects (wireless audio,
DVD-recorder, …)
• Clear results with respect to customer satisfaction, reduced
cost of non-quality, first time right design.
Guy Van Hooveld 16March , 2008
DFSS Approach in Software Life cycle CONQ reduction
Listen to Consumer needs (Voice of consumer)
Translate needs into CTQs with targets and specification limits
Flow down the top-level CTQ (Y) into lower level inputs Xs
Select Best design to meet the CTQ
Predict Capability on “Y” based on current Xs
Optimize gap between predicted and desired
Verify CTQs on target, Factory
Monitor CTQs in Field
FMEA &
Mistake proofing
Req.Mgt
Arch,Des
&Imp
Int.
Test
Maint
Sw Life-cycle DD
II
DD
OO
vv
MM
Guy Van Hooveld 17March , 2008
Requirements
Architecture / Top Level Design
Detail Design
Implementation
Module testing
Integration Testing
Alpha testing
DFSS mapping to V-model and SPEED
Feasibility Beta testing Maintenance
VPH CS PRS DR CR MPRProgramming Conception Development Maturity Maintenance
DefineIdentify
DesignOptimize
VerifyMonitor
Guy Van Hooveld 18March , 2008
Quality process - Software Mapping
Quality process based on 4 main principles
1. Identify the Voice of the Customer (VOC)• Obtain & prioritize customer requirements, Derive CTQs
2. Manage Critical parameters• Define targets for CTQs, Flow-down CTQs• Flow-up CTQs through predictions, Establish control plan
3. Establish Robust & Reliable Design• Identify sources of variation, de-sensitize design• Optimize Performance, Verify & Validate robustness
4. Manage risks & problems• Anticipate risk, counter or mitigate them• Detect and solve problems
Guy Van Hooveld 19March , 2008
Quality - Software Mapping
Kano
Risk Benefit
Brand Positioning
Marketing Concept
Consumer Insight
Benefits Reasons to Believe
Discriminator
TargetComp.Environment
“When we watch digital TV or DVD movies, we want a system that can playback and record in excellent audio and video quality.”
Selectives/ Classics 25-49 years old/Married w children or coupled Young Potentials Achieving while balancing family life
Takes time and effort to research on the market or product he is interested in Sensible consumers who value quality, functionality and ease of use.
Consumer InsightTargetCompetitive Environment
• Turn up your experience with Hi-definition Audio & Video
DiscriminatorReasons to BelieveBenefits
•Recording gets simpler, yet the quality of the picture gets better•The picture quality enhancements allow for a cinematic viewing experience, completely immersing us into the movie.
• Ease of recording with EPG•Picture quality improvements happen automatically•User-friendly user-interface
•Best Picture Quality
•There's always my favorite content waiting to be watched on the hard drive
AdvancedEasy To ExperienceDesigned Around You
Marketing Concept: (fixed: the essence of the marketing concept)
ValueHouse
DVDR5xxxH/7xxxH series
Emotional:•A rewarding experience that brings you
closer to the action.•Always know something you want to
watch is waiting for you. • Your recordings bring you closer to the
action than ever before.Functional:•Watch recordings in the best picture quality,
even better than broadcast TV.•Simple operation and use, facilitated by EPG
• Best viewing experience:HDMI , 1080i upsampling, Digital TV (DVB-T )
• Best sound:HDMI , 5 .1 channel recording from STB
• Ease of Experience:More than 400hrs of recordings (250Gb), EPG, user- friendly UI .
C brands entering Q3 at 499 Euros HDD capacity increased substantially HDMI being offered by others (Toshiba,
LG, Samsung) There are more recorder models with
DTT introduced by Sony, Panasonic, Samsung in Europe, especially in UK & Germany. •H
VPH
Extremely important
Very important Less important Not important
Playback music of a disc Adapting the set recording to the time that the program is broadcast. (VPS/PDC)
Playback of a recorded program, while the recorder is still recording the program.
Playback photos of a disc
Changing subtitle language while watching DVD
Recording 1. Timer recording 2. Direct recording 1. Show View 2. EPG (TV
guide)
Copying from hard disk to DVD disc (archiving)
Repeated recording of daily/weekly broadcasted program Changing audio
language while watching DVD
Playback of a recorded program and record another live tv program in the same time
Repeated playback of a recording or a part of a recording
Replay a soundtrack
Using a recording device: 1. Playback a movie 2. Watching a TV program 3. Jumping to the next chapter
Zoom in on an object in a movie/TV programme
For editing, most often used: 1. Deleting recording 2. Change title recording. Playback in slow-
motion Watching TV by switching channels through recorder tuner Pause live TV
(for currently watching channel)
Connecting an extra product: 1. VCR, for copying videocassettes to DVD discs 2. Digital camera, for watching pictures on TV screen and copying photo’s to DVD discs 3. Camcorder, for watching video on TV screen and copying video’s to DVD discs 4. DVD player, for copying DVD discs
Pausing during recording, so that parts of the broadcast (e.g. commercials) are not recorded Changing camera
angle in a movie
CRS
CTQs , CFs
HECE, UIS, FRS
1- Identify the Voice of customer (VOC)
Guy Van Hooveld 20March , 2008
CTQ flow downCTQ – Targets & Limits
CTQ flow-up & predictions
Control Plan - Dashboard
Quality - Software Mapping 2- Manage Critical Parameters
Guy Van Hooveld 21March , 2008
Arch/Des choices
CTQ Trade-offsCTQ optimisation, Noise de-sensitization
Transfer functions
3- Establish robust & reliable design
Quality - Software Mapping
Guy Van Hooveld 22March , 2008
User/CEC tests
Test coverage
Stress TestsField Tests
FMEA
PMI, PR/CR tracking
Verify CTQs
Quality - Software Mapping 4 – Manage Risks & Problems
Guy Van Hooveld 23March , 2008
Gains• Software engineering (addressing the pain areas)
– Better Requirements Management in terms of specifications, prioritization, traceability– Focus on non-functional aspects of usability, interoperability, performance, robustness– Importance of Execution architecture (State-transitions, CPU loading, Memory)– Increase in Design effort, upfront design discussions automatically reduce rework– Early involvement of test team – CTQ measurement method– CTQs as leading indicators of product quality as against PMI
• Cross-functional approach– Reduction in communication barrier – way of working with marketing
• e.g. challenging specifications, Kano, risk-benefit analysis– Common language of CTQs with other disciplines for synergy e.g. hardware, electrical
• End-user perspective– Development community sensitive to VOC E.g. needs versus wants– Measurement focus (variation rather than average), best case-worst case spectrum
• System understanding (Product perspective)– Transfer functions triggers to understand the system better (may expose competency)– Understanding of noise and its effect on the system (usage environment)– Trigger on “what could fail” (FMEA, mistake-proofing) improves robustness
• Soft Aspects – Mindset of “first time right” instead of “build-test-fix”
Guy Van Hooveld 24March , 2008
Issues– Everything is linked to CTQs so there is a chance of completely missing important
ones– Does not compensate for domain competency (CTQ flow down, FMEA etc)– Many requirements in software fall under Yes/No, Pass/Fail category so limit
setting limits, target becomes fuzzy (Critical factors rather than CTQs)– Out of limits need not strictly mean defective as in case of hardware e.g. start-up
time, hence predicting DPMO (defects per million opportunities) may not reflect reality
– Random failures due to only software are rare due to which concept like MTBF for software alone is questionable (however makes sense at product level)
– No concept of samples – the same piece of code is corrected and used, so hard-core statistical concepts cannot be applied at software level only (however can be done at product level)
– Configuration Management has to be addressed in the traditional way– No direct tools available to control regression and side-effects in software– Additional effort needs to be budgeted in the initial phases (confidence of Project)– With more and more projects going the ODM way (Black-box model) – not clear on
how to select/manage suppliers on software CTQs– Marketing community still needs to be educated to be able to specify CTQs in a
quantifiable way and as part of the requirements specification itself