agile and medical device software€¦ · “agile methods could be non-compliant because of the...
TRANSCRIPT
Agile and Medical Device Software
Incompatible? or Harmonious Union?
Brian Pate
Agenda
• Why Agile?
• Agile and Medical Device Software
• Lean and Compliant
Copyright SoftwareCPR®
Agenda
• Why Agile?
• Agile and Medical Device Software
• Lean and Compliant
Copyright SoftwareCPR®
Copyright SoftwareCPR®
We are uncovering better ways of developing software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
MANIFESTO FOR AGILE SOFTWARE DEVELOPMENT
© 2001, the above authors this declaration may be freely copied in any form, but only in its entirety through this notice
Kent Beck James Grenning Robert C. Martin
Mike Beedle Jim Highsmith Steve Mellor
Arie van Bennekum Andrew Hunt Ken Schwaber
Alistair Cockburn Ron Jeffries Jeff Sutherland
Ward Cunningham Jon Kern Dave Thomas
Martin Fowler Brian Marick
• Continuous planning and
feedback ensure value is
maximized.
• Continuously aligning the
delivered software with
business needs (adapting).
• Continuous testing provides
accurate visibility.
• Focusing on the rapid
delivery of business value
reducing the overall risk.
©2012, VersionOne, Inc. All rights reserved www.versionone.com
Agile & Lean Deliver on 9 Critical ROI Metrics
Copyright © Gear Stream, Inc.
• Improved Efficiency due to Optimized Process
• Higher Productivity due to Improved Morale
• Cost Control due to Superior On-time Performance
• Cost Reduction due to Improved Maintainability
• Higher Quality due to Improved Defect Removal
• Benefit Improvement due to Integral Customer Feedback
• Benefit Acceleration due to Incremental Delivery
• Benefit Recovery due to Superior On-time Performance
• Opportunity Recovery due to Superior On-time Performance
Questions
Agile methods for medical
device software…
• can it be compliant?
• can it be safe?
Copyright SoftwareCPR®
Questions
“Agile methods could be non-compliant because of the lack of
formal requirements.”
“Agile methods are not suited for medical device software
development because there is no formal verification and
validation.”
“Agile methods are not suited for medical device software
development because there is no formal process for ensuring all
hazards are properly mitigated.”
Copyright SoftwareCPR®
Resources
• Soon to be released Technical Information
Report from AAMI titled “Guidance on the use
of agile practices in the development of
medical device software”
• IEC 62304:2006 “Medical device software --
Software life cycle processes”
Copyright SoftwareCPR®
Copyright SoftwareCPR® Copyright SoftwareCPR®
Agenda
• Why Agile?
• Agile and Medical Device Software
• Lean and Compliant
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
TIR Content:
• Perspectives
• Concepts
• Practices
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
Guidance on the use of Agile practices in the
development of medical device software
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
Why Publish a TIR?
• Information is valuable
• Immediate need
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
TIR Purpose
• Provide motivation for the use of Agile
• Clarify misconceptions about the suitability of
Agile
• Provide direction on the application of Agile
to meet quality system requirements
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
What the TIR is not intended as:
• Educational tool for Agile development
practices
• Education tool for quality management
regulations
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
How should the TIR be used?
• Resource for creation of SW Dev process
– Expert point of view
– Can present different points of view
• Assist with communication within
organization
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
When will this TIR release?
• Soon …
• AAMI Software committee is reviewing …
hoping for June 1, 2012 meeting
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
TIR Content:
• Perspectives
• Concepts
• Practices
Copyright SoftwareCPR®
Agile and Medical Device Software
TIR Perspectives
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
Guidance on the use of Agile practices in the
development of medical device software
Copyright SoftwareCPR®
TIR - Perspectives
The Regulatory Perspective:
• Values safety and performance
• Values process and methods to ensure safety
and efficacy
• Value demonstrable evidence that processes and
methods were used
Copyright SoftwareCPR®
TIR - Perspectives
Key regulatory principles:
• Established (controlled) processes
• No specific development methodology, but should be methodic
and contain essential activities
• Quality systems depend upon management control, professional
training, and professional work
• Documentation (evidence) necessary to demonstrate compliance
• Planning is important to ensure processes are executed effectively
Copyright SoftwareCPR®
Regulations/Standards • Governmental Laws (e.g. FDA, MDD, etc)
• Standards (e.g. ANSI/AAMI/ISO 14971:2000, EN
62304:2006) ANSI/AAMI/ISO 14971:2000, Medical
devices—Application of risk management to medical
devices
• ISO 13485:2003, Medical devices -- Quality management
systems -- Requirements for regulatory purposes
• EN 62304:2006
Copyright SoftwareCPR®
FDA
FDA regulates software in the following areas:
• Medical Devices
• Automation of Production Systems
• Automation of Quality Systems
• Computers used in Clinical Investigations
• Electronic Record & Electronic Signatures
Copyright SoftwareCPR®
TIR - Perspectives
IEC 62304:2006
• A framework – processes, activities and tasks
– Process is the top level, a process has activities and an activity has tasks. Specific
requirements in IEC 62304 are generally at the task level.
• Identifies requirements for what needs to be done and what
needs to be documented
• Specifies a software safety classification scheme
– Additional requirements apply as safety becomes more important
– Much more significant than a simple minor/moderate distinction
Copyright SoftwareCPR®
TIR - Perspectives
Do the goals of regulations and
standard align with the goals Agile?
• Agile and regulatory bodies both value
high quality software
• Keep balanced emphasis
Copyright SoftwareCPR®
TIR - Perspectives
Do the values of regulations and standard
align with the goals Agile?
• The Agile Manifesto is a clear, powerful, and
internationally recognized
• Regulatory value is inferred from regulations
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
TIR Content:
• Perspectives
• Concepts
• Practices
Copyright SoftwareCPR®
Agile and Medical Device Software
TIR Aligning on Concepts
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
Guidance on the use of Agile practices in the
development of medical device software
Copyright SoftwareCPR®
TIR - Concepts
Is Agile an Incremental or an Evolutionary
Lifecycle?
• Yes
Copyright SoftwareCPR®
TIR - Concepts
Agile is Incremental …
• Software system broken into small pieces
• Uses backlog
• Pieces are realized in short increments
Copyright SoftwareCPR®
TIR - Concepts
But Agile is also Evolutionary …
• Allows product definition to evolve over time
• Detailed definition “emerges”
• Highly overlapping and dynamic
Copyright SoftwareCPR®
TIR - Concepts
Software Quality Assurance …
• Performed on small pieces
• Continuously applied
• Continuously integrated to demonstrate “sum of
parts” is consistent and a correct whole
Copyright SoftwareCPR®
TIR - Concepts
Regulations, standards, and guidance
applicable to medical device software neither
mandate nor prefer any particular lifecycle
model.
• Manufacturer to choose
• Should be appropriate to their situation/application
Copyright SoftwareCPR®
Process Flow, Timing
62304 shows as linear sequence: Requirements
analysis, Architectural design, Detailed design,
• Design Planning
• Design Review documentation
• “…contain historical versions of key DMR
documents to show how design evolved”
Copyright SoftwareCPR®
TIR - Concepts
Address these in your process:
• Process activities to be executed
• The timing and sequencing of those activities
• The inputs to and outputs of an activity
• The process artifacts to be produced
• How lifecycle integrates with risk management
• How change control and CM will be handled
Copyright SoftwareCPR®
Layers of Abstraction • Project
– What’s needed to deliver finished product
• Release
– What’s needed to deliver a usable product
• Increment
– What’s needed to deliver a set of useful functionality
• Story
– What’s needed to deliver a small piece of functionality
Copyright SoftwareCPR®
Copyright SoftwareCPR®
Configuration Management
Agile approaches to software development
require significantly more discipline in
managing the software system configuration
work products change constantly and frequently
Copyright SoftwareCPR®
Configuration Management
Benefit of
responding to
customer needs
[requirement]
changes
Pain of added CM
discipline
Copyright SoftwareCPR®
› greater than!
Defining Done
DONEness criteria should also address:
• Process requirements that must be satisfied
• Reviews to be performed
• Approvals to be obtained
• Product acceptance criteria used to evaluate work product
• Documentation requirements that must be satisfied
Copyright SoftwareCPR®
On-going Impact Analysis
Work that occurs in any iteration could affect
work previously done in an earlier iteration
• Use the concepts of impact analysis on each iteration
• Practices such as continuous integration, automated
regression testing, design review, and customer demos
provide necessary feedback (non-exhaustive list)
Copyright SoftwareCPR®
Inputs
Iterations only need to consider a limited set of
inputs
Use the concepts of impact analysis on each iteration
• Practices such as continuous integration, automated
regression testing, design review, and customer demos
provide necessary feedback (non-exhaustive list)
Copyright SoftwareCPR®
Inputs
Agile’s evolutionary nature allows for inputs to
evolve as the outputs are created
• Common at the start of an iteration to say “the input is
sufficient to begin”
• This is highly subjective and depends heavily on the context
in which it applies and on the team’s judgment to apply it.
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
TIR Content:
• Perspectives
• Concepts
• Practices
Copyright SoftwareCPR®
Agile and Medical Device Software
TIR Aligning on Practices
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
Guidance on the use of Agile practices in the
development of medical device software
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
TIR Content:
• Perspectives
• Concepts
• Practices: Topics related to planning
Copyright SoftwareCPR®
Planning
Is agile too undisciplined to meet planning
requirements?
• Agile approaches are sometimes perceived to
lack disciplined processes and process
execution
This perception is incorrect.
Copyright SoftwareCPR®
Planning
Agile has the concept of layers of planning
• Planning occurs at the story, the increment,
and the release
Copyright SoftwareCPR®
Development Planning
• Story – plan for elaboration and implementation
of a specific piece of functionality
• Increment – plan for the all of the functionality
that will be added in a fixed time window
• Release – plan for all the activities that are
needed to finish the software development
project
Copyright SoftwareCPR®
Planning
• Agile has the concept of the “DONE is DONE”
checklist that is executed and verified.
– This provides feedback on successful completion
of the planned activities.
Copyright SoftwareCPR®
Planning
• So, is agile suitable for medical device
software?
– Agile teams will find they do need more than just
a Backlog and a Release strategy to address all
regulatory requirements.
Copyright SoftwareCPR®
Planning
IEC 62304 5.1 Software Development Planning
requires:
• The processes to be used
• The deliverables (included documentation) of the activities and tasks
• Traceability between system reqm’ts, sw reqm’ts, test, and risk control
measures
• Software configuration management and change management (including
SOUP)
• Software problem resolution at each stage of the life cycle
Copyright SoftwareCPR®
Planning
• Is shippable software, after every increment,
a realistic expectation?
• This would seem to be rarely practical for medical
device software.
• Release hardening and formal verification &
validation could be cost prohibitive.
Copyright SoftwareCPR®
Planning
• Is the “shippable” practice valuable?
• Act as though it were shippable (minus a few
activities)
• Continuous Integration, Integration testing (for the
increment might be less formal than for the release)
• Regression testing schemes
Copyright SoftwareCPR®
Verification Planning
Agile’s “Done is Done” concept is core to creating a
verification plan
• Each story needs: elaboration (requirements analysis),
architectural and detailed design, sw unit coding and
verification, sw integration testing, and system testing
• Verification becomes part of the “Done is Done” checklist
Copyright SoftwareCPR®
Continuous Integration
Continuous Integration typically requires that
configuration management practices are
defined and perfected very early in the project,
and then relentlessly followed.
• Provides immediate feedback if things go
differently than planned.
Copyright SoftwareCPR®
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
TIR Content:
• Perspectives
• Concepts
• Practices: Topics related to team structure
and collaboration
Copyright SoftwareCPR®
Pairing
Pairing is the practice of putting two developers on one
activity.
• Could be applied to requirements, design, test and coding
(commonly applied to coding, i.e. “pair programming”)
• Second person can assume reviewer role
• Perceived to bring extra costs but this is usually offset by
lower defect removal costs later
Copyright SoftwareCPR®
Pairing
Benefits:
• Improved design: different perspectives
• Effective training/mentoring
• Knowledge retention: if developers leave
• Improve quality: review, conformance
• Form of continuous verification
Copyright SoftwareCPR®
Pairing
When used to fulfill regulatory or compliance
requirement for review:
• Establish acceptance criteria to ensure the design
output meets design input requirements
• Define the qualifications of the reviewers appropriate
to the task
• Provide appropriate evidence of the review
Copyright SoftwareCPR®
Pairing
What about independence of review?
• Certain critical, complex, or safety related
software may need additional perspectives to
provide diversity of knowledge and experience to
ensure adequate review.
• Could be accomplished by changing pairing
assignments or supplemental reviews.
Copyright SoftwareCPR®
Stop the line!
Stop the line is the practice of monitoring processes to
identify abnormalities that require attention.
• Monitor the development process: burndown charts,
regression results wave, tests written/passed charts
• Make information available to team (to everyone!): helps
to ensure full engagement to own and solve problems,
connect remote teams
• Act on the information quickly
Copyright SoftwareCPR®
Retrospectives/Reflections
Retrospectives and reflections are the practices that encourage a
team to regularly inspect their performance and take action to be
more effective.
• Retrospectives are event driven (e.g. end of iteration, release) and
focus on the preceding iteration/release and explore what could
have been done better.
• Reflections are done periodically and focus on how the team is
functioning.
• Continuous improvement
Copyright SoftwareCPR®
Collective Ownership
Collective ownership is about the team taking
ownership and responsibility for the quality of
the delivered software product.
• The team owns it all, not just one aspect.
• Equal voice: any team member can address
design, implementation, or test
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
TIR Content:
• Perspectives
• Concepts
• Practices: Topics related to product definition
and requirements documentation
Copyright SoftwareCPR®
Stories
When does a story have enough definition to begin
work?
• Story is a high level description of a piece of
functionality
• Must be detailed enough to allow prioritization
• Must be detailed enough to allow developer to
effectively elaborate and implement the story
Copyright SoftwareCPR®
Stories
• Effective prioritization helps ensure that the
highest value stories are done early in
development.
The presence of an embedded customer/
product owner can facilitate the elaboration
of the story with less documentation.
Copyright SoftwareCPR®
Acceptance Tests
Can the acceptance tests for stories be used for
final requirements?
• Important to produce requirements documentation that is useful
to both the team and to outsiders such as regulators, auditors, and
even other projects that might build on or interface to this project.
• Stories are generally at a high level
• If stories are updated with the results of the elaboration, could
possibly be used in requirements for the product.
Copyright SoftwareCPR®
Executable Requirements
Can Executable Requirements be a valid part of the
requirements definition and documentation?
• Executable requirement is an agile technique where requirements are
written in a formal language such that the test(s) for the requirement are
automatically generated.
• Challenge is to provide reviewable requirements documentation to
regulators. Ensure readability by outsider, Use English-like words for your
definition language.
Copyright SoftwareCPR®
V & V
How does agile help with requirements
verification and validation?
• Agile promotes the active involvement of the customer
• Pair programming allows refinement of requirements as
story is elaborated and implemented
• Demonstrations allow feedback from users on the
adequacy of requirements
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
TIR Content:
• Perspectives
• Concepts
• Practices: Topics related to software
architecture
Copyright SoftwareCPR®
Evolving Architecture
Project Layer
• Big picture architecture activities are performed “just
enough” to plan releases, increments, and some stories
Story Layer
• Impacts on architecture are addressed and adjusted as
needed
Underlying process is risk management: Architecture
plays a huge role in effective risk control.
Copyright SoftwareCPR®
Architecture Verification
• Continuous integration and test can expose
architectural weaknesses early.
• When refactoring happens, the ease or
difficulty of doing it is a reflection of the
maturity and correctness of the architecture.
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
TIR Content:
• Perspectives
• Concepts
• Practices: Topics related to detailed design
Copyright SoftwareCPR®
Detailed Design
• Not as distinct an activity as linear-flow
models might suggest.
• Planning documentation or procedures must
define how detailed design is to be
performed.
Copyright SoftwareCPR®
Emergent Design
• Agile encourages “Simple design” principle -> simple
enough to deliver the functionality of the story.
– Discourages complex designs intended to provide options
for future functionality that is not yet in the product.
• Refactoring is the process by which designs are re-
worked as new functionality is added. Better designs
emerge.
Copyright SoftwareCPR®
Design Documentation
• Well-written code along with well-written test
cases can itself be good documentation of the
detailed design – separate documentation may
not be necessary.
• Use the “hand-off test” – what verbal
information does a developer say to another
developer when handing off development?
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
TIR Content:
• Perspectives
• Concepts
• Practices: Topics related to implementation
and unit verification
Copyright SoftwareCPR®
Test Driven Development
• Unit test is created first, then coding is done
to satisfy the test.
• With pair programming, authors can “test-
drive” each others software.
Copyright SoftwareCPR®
• Agile practices such as test-driven
development, pair programming, continuous
integration, and continuous testing align to
deliver high quality software.
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
TIR Content:
• Perspectives
• Concepts
• Practices: Topics related to integration and
integration testing; system testing
Copyright SoftwareCPR®
Integration
• Continuous integration forces integration from
beginning.
• Techniques such as stubs, software simulations,
hardware emulators and test fixtures allow early
integration and progressive development.
• Integration testing at Story, increment, and
release layers.
Copyright SoftwareCPR®
System Test
• Agile has the potential to have test results for
many different versions, so it is important to
plan and clearly describe what testing was
done on what version, and how final testing
demonstrates that the final version has been
thoroughly tested.
Copyright SoftwareCPR®
Regression Testing
Agile has the disadvantage that system testing done on
the story and increment layers may not be repeated on
the final version.
– Concern: a later change may have introduced a defect.
With any incremental model, regression testing is
necessary to demonstrate that changes do not
introduce unintended consequences in software that
has already been tested.
Copyright SoftwareCPR®
SOUP
SOUP items may be updated throughout the
project due to normal updates, bug fixes,
security patches, etc.
SOUP updates will drive additional regression
testing.
Copyright SoftwareCPR®
System Testing
• System tests should be developed as the code
is developed, preferably in advance.
• Tests may not fully pass if future stories are
required.
• If code is changed in future story work, tests
must be changed accordingly.
Copyright SoftwareCPR®
Documenting System Testing
• Regulations and standards require the results of
system testing to be documented.
• If the team is tempted to skip the documentation
work, this should be an indication that the
documentation is too costly or lacking value –
find ways to address both of these issues (e.g.
automated testing).
Copyright SoftwareCPR®
Traceability
• The value of using TDD practices is that
requirements, verification, and test execution
are very closely linked, thus giving you built-in
traceability.
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
TIR Content:
• Perspectives
• Concepts
• Practices: Topics related to software release
Copyright SoftwareCPR®
Software Release
• Short release cycles
• Maybe not “shippable” but “near shippable”
– Usability testing
– Controlled clinical testing
Copyright SoftwareCPR®
Software Release
Layered Approach
• Story - most documentation is created
• Increment - named/controlled release; v&v
activities; doc package
• Release Layer – additional validation
activities; doc package completed
Copyright SoftwareCPR®
Software Release
• All known residual anomalies must be
documented.
– While agile doesn’t specifically address this, the
short release cycles and continuous testing will
make anomalies visible early.
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
TIR Content:
• Perspectives
• Concepts
• Practices: Topics related to configuration
management and change management
Copyright SoftwareCPR®
SW Configuration Identification
• Standards require a team to document the set
of configuration items and versions that
comprise the software system.
– Continuous integration
Establish software configuration baselines at
Increment boundaries.
Copyright SoftwareCPR®
SOUP CM
• Acquired software must be captured and
archived in a suitable manner.
• If SOUP items are updated, these updates
must be captured.
Establish software configuration baselines at
Increment boundaries.
Copyright SoftwareCPR®
Change Control
• Agile embraces change … welcomes change.
• Backlog can be management mechanism for
change control.
• Backlog enforces same prioritization as used
for new features.
Copyright SoftwareCPR®
AAMI/TIR SW1:2012
TIR Take-Away:
• Expectation for Tailoring … use what makes sense for your organization
• Apply the values of Agile in a way that enhances a robust QMS.
• Apply the practices of Agile within the context of an established QMS.
• Set the correct expectations by defining the lifecycle and how regulatory
requirements are fulfilled.
• Establish a robust change management system to manage change and
mitigate risks associated with rapid change.
Copyright SoftwareCPR®
Agenda
• Why Agile?
• Agile and Medical Device Software
• Lean and Compliant
Copyright SoftwareCPR®
Confessions
“Experience is something you don’t get until
just after you need it.” – Steven Wright
Copyright SoftwareCPR®
Discipline Point: “Requirements” are central to everything, BUT don’t need to be at same levels (waste)
Copyright SoftwareCPR®
Requirements detail: Low High
“go
od
en
ou
gh
” sp
ecs
SAFETY
CLINICAL UI
Typ
ical
sit
uat
ion
EVERYTHING
• May get overly focused/weighted toward unit testing
• May omit formal technical reviews with evidence
• May not perform proper regression testing or know when
to perform regression testing
• May fail to capture integration and system level test
cases during sprints
• Self-managed teams may neglect formal testing
altogether!
Copyright SoftwareCPR®
V&V
Each iteration should capture unit testing
As the system grows, each iteration should also capture integration
and system level test cases as appropriate for the system interactions
Integration and system test failures should be captured in a defect
tracking system
Test and spec approvals prior to running formal tests during certain
identified iteration
All test documentation is under configuration management
Copyright SoftwareCPR®
Discipline Points
• While system level hazards may be known, sprint
team may be not identify software-cause hazards
• Team may equate zero-defects with risk-free
• Even if hazards are mitigated, mitigation may not be
formally documented for test
• Subsequent sprints might alter mitigation software
and inadvertently dilute the mitigation effect
Copyright SoftwareCPR®
Risk Mitigation
Each iteration should capture any new hazards or causes
identified and document in risk management file
Each iteration should capture any mitigations developed
in that sprint and capture in requirements documentation
Phase planning should account for reviews of the risk
management documentation
Consider a designated risk team member to participate in
selected iteration or reviews
Copyright SoftwareCPR®
Discipline Points
Final Thoughts
• Inspect and adapt
• Understand the intent of the regulations
• Don’t check your “brain at the door”
Copyright SoftwareCPR®
Questions?
SoftwareCPR® is a software regulatory, safety, compliance, quality, and management consulting firm. We specialize in project and regulatory crisis
recovery in addition to training, preventive action, and continuous improvement.
Our expert staff and partners are industry practitioners – most have 25+ years experience in the medical device industry at technical levels through executive
management. We provide strategic and hands-on assistance to assure the success of your software investments.
Our approach focuses on your business objectives, project and regulatory risk
management, and pragmatic approaches that are tailored to your internal culture. We work closely with your internal staff, external vendors, your FDA counsel and
FDA itself.