business value of ci, cd, & devops sec -...
TRANSCRIPT
Business Value ofCI, CD, & DevOpsSec
Scaling Up to Billion User Global Systems of Systems Using END-TO-END AUTOMATION &
CONTAINERIZED DOCKER UBUNTU IMAGESDr. David F. Rico, PMP, CSEP, FCP, FCT, ACP, CSM, SAFe
Twitter: @dr_david_f_ricoWebsite: http://www.davidfrico.com
LinkedIn: http://www.linkedin.com/in/davidfricoAgile Capabilities: http://davidfrico.com/rico-capability-agile.pdf
Agile Resources: http://www.davidfrico.com/daves-agile-resources.htmAgile Cheat Sheet: http://davidfrico.com/key-agile-theories-ideas-and-principles.pdf
Dave’s NEW Business Agility Video: https://www.youtube.com/watch?v=-wTXqN-OBzADoD Fighter Jets vs. Amazon Web Services: http://davidfrico.com/dod-agile-principles.pdf
Author Background Gov’t contractor with 33+ years of IT experience B.S. Comp. Sci., M.S. Soft. Eng., & D.M. Info. Sys. Large gov’t projects in U.S., Far/Mid-East, & Europe
2
Career systems & software engineering methodologist Lean-Agile, Six Sigma, CMMI, ISO 9001, DoD 5000NASA, USAF, Navy, Army, DISA, & DARPA projects Published seven books & numerous journal articles Intn’l keynote speaker, 160 talks to 13,000+ people Specializes in metrics, models, & cost engineeringCloud Computing, SOA, Web Services, FOSS, etc. Adjunct at six Washington, DC-area universities
Most of world’s population connected to Internet Systems must support billions of simultaneous users New approaches are needed to scale to global market
3Kemp, S. (2016). Digital in 2016: We are social's compendium of global digital, social, and mobile data, trends, and statistics. New York, NY: We Are Social, Inc.
Agile Testing—Global Scaling
Test-ing (tĕst′ĭng) An early, iterative, and automatedV&V of customer requirements; Incremental testing A testing approach embracing principles & values of lean
thinking, product development flow, & agile methods Early, collaborative, and automated form of incremental
development, integration, system, & operational testing Testing method that supports collaboration, teamwork,
iterative development, & responding to change Mult-tiered automated framework for TDD, Continuous
Integration, BDD, Continuous Delivery, & DevOps Maximizes BUSINESS VALUE of organizations, portfolios,
& projects by enabling buyers-suppliers to scale globally
4
Crispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.Crispin, L., & Gregory, J. (2015). More agile testing: Learning journeys for the whole team. Boston, MA: Addison-Wesley.
Agile Testing—What is it?
NetworkComputer
Operating SystemMiddlewareApplications
APIsGUI
Agile requirements implemented in slices vs. layers User needs with higher business value are done first Reduces cost & risk while increasing business success
5Shore, J. (2011). Evolutionary design illustrated. Norwegian Developers Conference, Oslo, Norway.
Agile Traditional1 2 3 Faster
Early ROI
Lower Costs
Fewer Defects
Manageable Risk
Better Performance
Smaller Attack Surface
Late
No Value
Cost Overruns
Very Poor Quality
Uncontrollable Risk
Slowest Performance
More Security Incidents Seven Wastes1. Rework2. Motion3. Waiting4. Inventory5. Transportation6. Overprocessing7. Overproduction
MINIMIZES MAXIMIZES
JIT, Just-enough architecture Early, in-process system V&V Fast continuous improvement Scalable to systems of systems Maximizes successful outcomes
Myth of perfect architecture Late big-bang integration tests Year long improvement cycles Breaks down on large projects Undermines business success
Agile Testing—How it works?
Thousands of TestsContinuously Executed
No More Late BigBang Integration
User needs designed & developed one-at-a-time Changes automatically detected, built, and tested System fully tested and deployed as changes occur
6Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.
BuildIntegration
Server
VersionControlServer
BuildScripts
UsesWatches
BuildStatus
ProvidesDeveloper A
Developer B
Developer C
CommitsChanges
CommitsChanges
CommitsChanges
Builds
Database
Analysis
Testing
Reporting
Documentation
Deployment
Early, Automated, Fast,Efficient, & Repeatable
Constant ReadinessState & CM Control
Lean, Waste Free, Low WIP,No Deadlocked Test Queues
Rapidly & SuccessfullyDev. Complex Systems
Agile Testing—Workflow
7
Traditional vs. Agile Cumulative Flow
Wor
k (S
tory
, Poi
nt, T
ask)
or E
ffor
t (W
eek,
Day
, Hou
r)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Wor
k (S
tory
, Poi
nt, T
ask)
or E
ffor
t (W
eek,
Day
, Hou
r)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
TRADITIONAL Cumulative Flow AGILE Cumulative Flow
Late big bang integration increases WIP backlog Agile testing early and often reduces WIP backlog Improves workflow and reduces WIP & lead times
Anderson, D. J. (2004). Agile management for software engineering. Upper Saddle River, NJ: Pearson Education.Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.
Agile Testing—Workflow—Results
8
Methods to “scope” project, product, or system “Key” is smallest possible scope with highest value Reduces cost, risk, time, failure, & tech. obsolescence
INCREASES TESTABILITY, QUALITY, RELIABILITY, SECURITY, MORALE, MAINTAINABILITY, & SUCCESS
Denne, M., & Cleland-Huang, J. (2004). Software by numbers: Low-risk, high-return development. Santa Clara, CA: Sun Microsystems.Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation. New York, NY: Crown Publishing.Patton, J. (2014). User story mapping: Discover the whole story, build the right product. Sebastopol, CA: O'Reilly Media.Layton, M. C., & Maurer, R. (2011). Agile project management for dummies. Hoboken, NJ: Wiley Publishing.Krause, L. (2014). Microservices: Patterns and applications. Paris, France: Lucas Krause.
MINIMUM
MARKETABLE FEATURE- MMF -
AdvantageDifferenceRevenueProfitSavingsBrandLoyalty
MINIMUMVIABLE PRODUCT
- MVP -
GoalProcessFeaturesPrioritiesStory MapArchitecture
STORY MAPOR IMPACT MAP
- SM or IM -
GoalActors ImpactsDeliverablesMeasuresMilestones
VISIONSTATEMENT
- VS -
For <customer>Who <needs it>The <product> Is a <benefit>That <customer>Unlike <other>Ours <different>
MICRO-SERVICE- MS -
PurposeAutomatedUnique IndependentResilientEcosystemConsumer
Agile Testing—MMF, MVP, MVA, etc.
9
Lightweight, fast, disposable virtual environments Set of isolated processes running on shared kernel Efficient way for building, delivering, & running apps
Monolithic Applications Just-Enough Applications Containerized Apps
Minimal - Typically single process entitiesDeclarative - Built from layered Docker imagesImmutable - Do exactly same thing from run to kill
• Small autonomous services that work together• Self-contained process that provides a unique capability
• Loosely coupled service oriented architecture with bounded contexts• Small independent processes communicating with each other using language-agnostic APIs
• Fined-grained independent services running in their own processes that are developed and deployed independently• Suite of services running in their own process, exposing APIs, and doing one thing well (independently developed and deployable)
• Single app as a suite of small services, each running in its own process and communicating with lightweight mechanisms (HTTP APIs)
Krause, L. (2014). Microservices: Patterns and applications. Paris, France: Lucas Krause.
Agile Testing—Microservices
OS’s HaveUNREPORTED
30-50 CVEs
10
TDD
- 2003 -CI
- 2006 -BDD
- 2008 -CD
- 2011 -DEVOPS
- 2012 -DEVOPSSEC- 2014 -
User Story
Acc Criteria
Dev Unit Test
Run Unit Test
Write SW Unit
Re-Run Unit Test
Refactor Unit
Building
Database
Inspections
Testing
Feedback
Documentation
Deployment
Analyze Feature
Acc Criteria
Dev Feat. Test
Run Feat. Test
Develop Feature
Re-Run Feature
Refactor Feat.
Packaging
Acceptance
Load Test
Performance
Pre-Production
Certification
Deployment
Sys Admin
Config. Mgt.
Host Builds
Virtualization
Containerization
Deployment
Monitor & Supp
Sec. Engineer.
Sec. Containers
Sec. Evaluation
Sec. Deploy.
Runtime Prot.
Sec. Monitoring
Response Mgt.
Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.Barker, K., & Humphries, C. (2008). Foundations of rspec: Behavior driven development with ruby and rails. New York, NY: Apress.Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Huttermann, M. (2012). Devops for developers: Integrate development and operations the agile way. New York, NY: Apress.Bird, J. (2016). Devopssec: Delivering secure software through continuous delivery. Sebastopol, CA: O'Reilly Media.
Numerous models of lean-agile testing emerging Based on principles of lean & agile one piece flow Include software, hardware, system, & port. testing
Agile Testing—Various Models
BASIC—Test Driven Development Term coined by Kent Beck in 2003 Consists of writing all tests before design Ensures all components are verified and validated
11Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.
BASIC—Behavior Driven Develop. Term coined by Dan North in 2006 Consists of writing feature tests before design Ensures all system features are verified and validated
12Smart, J. F. (2014). BDD in action: Behavior-driven development for the whole software lifecycle. Shelter Island, NY: Manning Publications.
ADVANCED—Continuous Integration Term coined by Martin Fowler in 1998 Process of automated build/regression testing Evaluates impact of changes against entire system
13Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
ALL DEVELOPERS RUN PRIVATE BUILDS
DEVELOPERS COMMIT CODE TO VERSION CONTROL
INTEGRATION BUILDS OCCUR SEVERAL TIMES PER DAY
100% OF SYSTEM TESTS MUST PASS FOR EVERY BUILD
A SHIPPABLE PRODUCT RESULTS FROM EVERY BUILD
FIXING BROKEN BUILDS IS OF THE HIGHEST PRIORITY
REPORTS AUTOMATICALLY GENERATED & REVIEWED
Created by Jez Humble of ThoughtWorks in 2011 Includes CM, build, testing, integration, release, etc. Goal is one-touch automation of deployment pipeline
14Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.Ohara, D. (2012). Continuous delivery and the world of devops. San Francisco, CA: GigaOM Pro.
CoQ
• 80% MS Tst• 8/10 No Val• $24B in 90s• Rep by CD• Not Add MLK
ENTERPRISE—Continuous Delivery
Source CodeControl
BuildAutomation
TestAutomation
ContinuousIntegration
ReleaseAutomation
ContinuousDelivery
Created by Patrick Debois of Jedi BVBA in 2007 Collaboration of developers & infrastructure people Goal to automate the deployment to end-user devices
15Bass, L., Weber, I., & Zhu, L. (2015). Devops: A software architect's perspective. Old Tappan, NJ: Pearson Education.Gruver, G., & Mouser, T. (2015). Leading the transformation: Applying agile and devops at scale. Portland, OR: IT Revolution Press.Humble, J., Molesky, J., & O'Reilly, B. (2015). Lean enterprise: How high performance organizations innovate at scale. Sebastopol, CA: O'Reilly Media.
GLOBAL—Development Operations
Collaboration
GLOBAL—Development Ops Sec DevOpsSec coined by Shannon Lietz in 2014 Rugged devops, devsecops, secdevops, devopssec Microservices, security engineering & operations keysSecure Microservices
• Docker App• Docker Bins• Docker Files• Docker Images• Docker Scanning• Docker Registry• Docker Host• Docker Hub• Docker Monitoring
Secure Engineering• Security Champions• Security Planning• Security Training• Security Requirements• Security Architecture• Security Analysis• Security Testing• Security Review• Security Response
Secure Operations• Activity Logging• Event Monitoring• Configuration Mgt.
• Patch Management• User Access Control• Privilege Management
• Vulnerability Mgt.• Response Mgt.• Performance Mgt.
Bird, J. (2016). Devopssec: Delivering secure software through continuous delivery. Sebastopol, CA: O'Reilly Media.16
Agile Testing—Basic Metrics
17Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
Defects DecreaseIntegrations
Increase
IncreaseCoverage
IncreaseAutomation
18
Hewlett-Packard is a major user of CI, CD, & DevOps 400 engineers developed 10 million LOC in 4 years Major gains in testing, deployment, & innovation
Gruver, G., Young, M. & Fulghum, P. (2013). A practical approach to large-scale agile development. Upper Saddle River, NJ: Pearson Education.
TYPE METRIC MANUAL DEVOPS MAJOR GAINS
CYCLE TIME
IMPROVEMENTS
Build Time 40 Hours 3 Hours 13 x
No. Builds 1-2 per Day 10-15 per Day 8 x
Feedback 1 per Day 100 per Day 100 xRegression Testing 240 Hours 24 Hours 10 x
DEVELOPMENT
COST EFFORT
DISTRIBUTION
Integration 10% 2% 5 x
Planning 20% 5% 4 x
Porting 25% 15% 2 x
Support 25% 5% 5 x
Testing 15% 5% 3 x
Innovation 5% 40% 8 x
Agile Testing—CD Statistics
Assembla went from 2 to 45 releases every month 15K Google developers run 120 million tests per day 30K+ Amazon developers deliver 136K releases a day
19Singleton, A. (2014). Unblock: A guide to the new continuous agile. Needham, MA: Assembla, Inc.
62 x FasterU.S. DoD
IT Project
3,645 x FasterU.S. DoD
IT Project
Agile Testing—DevOps Statistics
Simple example of a DevOps reference architecture Includes CM, continuous integration, & deployment Code automatically built/tested/deployed to users
20Morris, B., & Cassatt, C. (2015). Devops for the rest of us. Proceedings of the Agile DC Conference, Washington, DC, USA.Weeks, D. E. (2014). Devops and continuous delivery reference architectures (volume 1 & 2). Fulton, MD: Sonatype.
Agile Testing—DevOps Example
21Juengst, D. (2015). Deliver better software faster: With the cloudbees jenkins platform. San Francisco, CA: CloudBees.Weeks, D. E. (2014). Devops and continuous delivery reference architectures (volume 1 & 2). Fulton, MD: Sonatype.
Agile Testing—DevOps Ecosystem Numerous tools to automate DevOps pipeline People can piece together toolset along with hubs Tools include version control, testing, & deployment
22
DevOps adoption growing fast in-spite of slow start 35% using, 27% thinking about it, & 38% are in-dark DevOps a global industry-wide extinction-level event
22Brown, A. (2016). Devops and the need for speed, quality, and security: Do organizations really have to pick two out of three. Portland, OR: Puppet Labs.
Agile Testing—DevOps Adoption
23Kim, G., Debois, P., Willis, J., & Humble, J. The devops handbook: How to create world-class agility, reliability, and security in technology organizations. Portland, OR: IT Revolution Press.
Everything begins with lean & agile principles Next step is smaller portfolio & simpler designs Final step is modular interfaces & E2E automation
Agile Testing—5 Keys to Success
Eliminates big-bang integration in the 11th hour Creates a repeatable and reliable testing process Evaluates system-wide changes throughout project
24Maeda, M. K. (2009). Agile testing: Early, often, and smart. Arlington, MA: Cutter Consortium.
Agile TestingTraditional Testing
· High project visibility· Greater confidence and morale· Incremental business value· 24x7 deployability to users· Highly quality and reliability
· Lack of deployability· Late big-bang integration· Testing is a bottleneck· Poor customer satisfaction· Outright project failure
Agile Testing—Bottom Line?
Dramatically reduces risks
· Automates manual processes· Instant verification & validation
Late defect discovery· Low quality software· Poor project visibility
··
Thousands of textbooks on agile methods Include requirements, design, coding, test, etc. Continuous Integration, Delivery, & DevOps best
25
Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.Smart, J. F. (2014). BDD in action: Behavior-driven development for the whole software lifecycle. Shelter Island, NY: Manning Publications.Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Kim, G., Debois, P., Willis, J., & Humble, J. The devops handbook: How to create world-class agility, reliability, and security in technology organizations. Portland, OR: IT Revolution Press.
Agile Testing—Foundational Books
Dave’s PROFESSIONAL CAPABILITIES
26
SoftwareQuality
Mgt.
TechnicalProject
Mgt.
SoftwareDevelopment
Methods
Strategy &Roadmapping
SystemsEngineering
Cost Estimates& Scheduling
Acquisition &Contracting
OrganizationChange
Lean, Kanban,& Six Sigma
Modeling &Simulations
Big Data,Cloud, NoSQL
WorkflowAutomation
Metrics,Models, & SPC
BPR, IDEF0,& DoDAF
DoD 5000,TRA, & SRA
PSP, TSP, &Code Reviews
CMMI &ISO 9001
InnovationManagement
Statistics, CFA,EFA, & SEM
ResearchMethods
EvolutionaryDesign
Valuation — Cost-Benefit Analysis, B/CR, ROI, NPV, BEP, Real Options, etc.
Lean-Agile — Scrum, SAFe, Continuous Integration & Delivery, DevOps, etc.
STRENGTHS – Data Mining Gathering & Reporting Performance Data Strategic Planning Executive & Manage-ment Briefs Brownbags & Webinars White Papers Tiger-Teams Short-Fuse Tasking Audits & Reviews Etc.
● Data mining. Metrics, benchmarks, & performance.● Simplification. Refactoring, refinement, & streamlining.● Assessments. Audits, reviews, appraisals, & risk analysis.● Coaching. Diagnosing, debugging, & restarting stalled projects.● Business cases. Cost, benefit, & return-on-investment (ROI) analysis.● Communications. Executive summaries, white papers, & lightning talks.● Strategy & tactics. Program, project, task, & activity scoping, charters, & plans.
PMP, CSEP,FCP, FCT
ACP, CSM,& SAFE
33 YEARSIN IT
INDUSTRY
Backup Slides
Software in U.S. DoD Systems
Kennedy, M. P., & Umphress, D. A. (2011). An agile systems engineering process: The missing link. Crosstalk, 24(3), 16-20.
No. of software-intensive systems is growing 80% of US DoD functions performed in software Major driver of cost, schedule, & tech. performance
28
Requirements Defects & Waste
29Sheldon, F. T. et al. (1992). Reliability measurement: From theory to practice. IEEE Software, 9(4), 13-20Johnson, J. (2002). ROI: It's your job. Extreme Programming 2002 Conference, Alghero, Sardinia, Italy.
Requirements defects are #1 reason projects fail Traditional projects specify too many requirements More than 65% of requirements are never used at all
Other 7%
Requirements47%
Design28%
Implementation18%
Defects
Always 7%
Often 13%
Sometimes16%
Rarely19%
Never45%
Waste
Large Traditional Projects
30
Big projects result in poor quality and scope changes Productivity declines with long queues/wait times Large projects are unsuccessful or canceled
Jones, C. (1991). Applied software measurement: Assuring productivity and quality. New York, NY: McGraw-Hill.
Size vs. Quality
DE
FEC
TS
0.00
3.20
6.40
9.60
12.80
16.00
0 2 6 25 100 400
SIZE
Size vs. Productivity
PR
OD
UC
TIV
ITY
0.00
1.00
2.00
3.00
4.00
5.00
0 2 6 25 100 400
SIZE
Size vs. Change
CH
AN
GE
0%
8%
16%
24%
32%
40%
0 2 6 25 100 400
SIZE
Size vs. SuccessS
UC
CE
SS
0%
12%
24%
36%
48%
60%
0 2 6 25 100 400
SIZE
Global Project Failures
31Standish Group. (2015). Chaos summary 2015. Boston, MA: Author.Sessions, R. (2009). The IT complexity crisis: Danger and opportunity. Houston, TX: Object Watch.
Challenged and failed projects hover at 67% Big projects fail more often, which is 5% to 10% Of $1.7T spent on IT projects, over $858B were lost
$0.0
$0.4
$0.7
$1.1
$1.4
$1.8
2002 2003 2004 2005 2006 2007 2008 2009 2010
Trill
ions
(US
Dolla
rs)
Expenditures Failed Investments
0% 20% 40% 60% 80% 100%
28%
34%
29%
35%
32%
33%
27%
28%
29%
49%
51%
53%
46%
44%
41%
56%
55%
52%
23%
15%
18%
19%
24%
26%
17%
17%
19%
2000
2002
2004
2006
2008
2010
2012
2014
2015
Year
Successful Challenged Failed
32
Capability #1
● Feature 1● Feature 2● Feature 3● Feature 4● Feature 5● Feature 6● Feature 7
Capability #2
● Feature 8● Feature 9● Feature 10● Feature 11● Feature 12● Feature 13● Feature 14
Capability #3
● Feature 15● Feature 16● Feature 17● Feature 18● Feature 19● Feature 20● Feature 21
Capability #4
● Feature 22● Feature 23● Feature 24● Feature 25● Feature 26● Feature 27● Feature 28
Capability #5
● Feature 29● Feature 30● Feature 31● Feature 32● Feature 33● Feature 34● Feature 35
Capability #6
● Feature 36● Feature 37● Feature 38● Feature 39● Feature 40● Feature 41● Feature 42
Capability #7
● Feature 43● Feature 44● Feature 45● Feature 46● Feature 47● Feature 48● Feature 49
1
2 3
4
5 6
7
8 9
10
11 12
13
14 15
16
17 18
19
20 21
Evolving “Unified/Integrated” Enterprise Data Model
“Disparate” LEGACY SYSTEM DATABASES (AND DATA MODELS)
ETL
A A
B C
D E F
G H I J K
A
B C
D E F
A
B C
D E
A
B C
D
A
B C
A
B
“Legacy” MS SQL Server Stovepipes “Inter-Departmental” Linux Blade/Oracle/Java/WebSphere Server
“Leased” DWA/HPC/Cloud Services
Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6 Sprint 7
ETL ETL ETL ETL ETL ETL
Bente, S., Bombosch, U., & Langade, S. (2012). Collaborative enterprise architecture: Enriching EA with lean, agile, and enterprise 2.0 practices. Waltham, MA: Elsevier.
(for example, assume 25 user stories per feature, 175 user stories per capability, and 1,225 user stories total)
Organize needs into capabilities, features, and stories Prioritize features, group releases, and initiate sprints Develop minimum set of features with highest value
Release
Release
Release
ReleaseMMF- or -MVP
Agile Testing—Road/Story Map, Arch
Ladas, C. (2008). Scrumban: Essays on kanban systems for lean software development. Seattle, WA: Modus Cooperandi.Reddy, A. (2016). The scrumban revolution: Getting the most out of agile, scrum, and lean-kanban. New York, NY: Addison-Wesley.
Created by Corey Ladas of Modus Cooperandi (2008) Hybrid of Agile (Scrum) and Lean (Kanban) methods Scrum with one-piece-workflow vs sprints (batches)
33
Agile Testing—Workflow—ScrumBan
Agile BDD consists of seven broad practices Document test criteria, tests, syst. features, etc. Include refactoring, verification, optimization, etc.
34
Practice
Feature
Acc Criteria
Dev Test
Run Test
Dev Feature
Rerun Test
Refac Feature
Description
Read feature, analyze meaning, ask questions, and clarify understanding
Identify, verify, and document acceptance criteria for each feature
Design, develop, code, and verify automated feature test for feature
Run automated feature test to verify that it fails the first time (sanity check)
Design, develop, code, and verify the feature software to satisfy feature
Rerun automated feature test to see if code satisfies automated feature test
Refine, reduce, and simplify code to remove waste and optimize performance
PRACTICES—Behavior Driven Dev.
Smart, J. F. (2014). BDD in action: Behavior-driven development for the whole software lifecycle. Shelter Island, NY: Manning Publications.
Agile CI consists of seven broad practices Automated build, database, inspection, tests, etc. Include reporting, documentation, deployment, etc.
35
Practice
Building
Database
Inspections
Testing
Feedback
Documentation
Deployment
Description
Frequently assembling products and services to ensure delivery readiness
Frequently generating/analyzing database schemas, queries, and forms
Frequently performing automated static analysis of product/service quality
Frequently performing automated dynamic product and service evaluation
Frequently generating automated status reports/messages for all stakeholders
Frequently performing automated technical/customer document generation
Frequently performing automated delivery of products/services to end users
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.
PRACTICES—Continuous Integration
Agile CD consists of seven broad practices Automated acceptance, load, performance, etc. Include packaging, pre-production test, C&A, etc.
36
Practice
Packaging
Acceptance
Load Test
Performance
Pre-Production
Certification
Deployment
Description
Frequently generating system images for pre-production testing & checkout
Frequently performing automated system & user acceptance testing
Frequently performing automated system load, stress, & capacity testing
Frequently performing automated system user & technical performance testing
Frequently performing automated pre-production tests prior to final deployment
Frequently performing automated system certification & accreditation tests
Frequently generating product images for pre-deployment testing & checkout
Mukherjee, J. (2015). Continuous delivery pipeline: Where does it choke. Charleston, SC: CreateSpace.Swartout, P. (2014). Continuous delivery and devops: A quickstart guide. Birmingham, UK: Packt Publishing.
PRACTICES—Continuous Delivery
Agile DevOps consists of seven broad practices Automated sys admin, CM, building, monitor, etc. Include virtualization, containerize, deployment, etc.
37
Practice
Sys Admin
Config. Mgt.
Host Builds
Virtualization
Containerize
Deployment
Monitor & Supp
Description
Frequently performing automated system administration tasks, i.e., scripting
Frequently performing automated infrastructure config. mgt./version control
Frequently performing automated system and server host builds and config.
Frequently performing automated system, server, & net virtualization services
Frequently performing automated software and Microservices containerization
Frequently generating final end-user system & software images for distribution
Frequently performing automated metrics collection & deployment monitoring
Duffy, M. (2015). Devops automation cookbook: Over 120 recipes coverying key automation techniques. Birmingham, UK: Packt Publishing.Farcic, V. (2016). The devops 2.0 toolkit: Automating the continuous deployment pipelines with containerized microservices. Victoria, CA: LeanPub.
PRACTICES—Development Operations
DevOpsSec consists of seven broad practices Automated secure build, analysis, & deployment Includes containerization, engineering & operations
38
Practice
Engineering
Containers
Evaluation
Deployment
Protection
Monitoring
Responses
Description
Frequently performing “baked-in” lean and agile security engineering practices
Frequently performing automated microservices containerization practices
Frequently performing automated static and dynamic vulnerability analysis
Frequently performing automated digitally signed security deployment practices
Frequently performing automated real-time self-security protection practices
Frequently performing automated real-time security monitoring practices
Frequently performing automated trigger-based rollback response practices
Bird, J. (2016). Devopssec: Delivering secure software through continuous delivery. Sebastopol, CA: O'Reilly Media.
PRACTICES—Development Ops Sec
Fewer integrations leave in higher bug counts Frequent, early integrations eliminate most defects Goal is to have as many early integrations as possible
39Lacoste, F. J. (2009). Killing the gatekeeper: Introducing a continuous integration system. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 387-392.
Number ofIntegrations
Less Defects•More Integrations•Early IntegrationsMore Defects
•Few Integrations•Late Integrations
Agile Testing—CI Statistics
Activity Def CoQ DevOps Economics Hours ROIDevelopment Operations 100 0.001 100 Defects x 70% Efficiency x 0.001 Hours 0.070 72,900%
Continuous Delivery 30 0.01 30 Defects x 70% Efficiency x 0.01 Hours 0.210 24,300%
Continuous Integration 9 0.1 9 Defects x 70% Efficiency x 0.1 Hours 0.630 8,100%
Software Inspections 3 1 2.7 Defects x 70% Efficiency x 1 Hours 1.890 2,700%
"Traditional" Testing 0.81 10 0.81 Defects x 70% Efficiency x 10 Hours 5.670 900%
Manual Debugging 0.243 100 0.243 Defects x 70% Efficiency x 100 Hours 17.010 300%
Operations & Maintenance 0.073 1,000 0.0729 Defects x 70% Efficiency x 1,000 Hours 51.030 n/a
40
Agile testing is orders-of-magnitude more efficient Based on millions of automated tests run in seconds One-touch auto-delivery to billions of global end-users
Rico, D. F. (2016). Devops cost of quality (CoQ): Phase-based defect removal model. Retrieved May 10, 2016, from http://davidfrico.com
Agile Testing—DevOps CoQ
Under 4Minutes
41Tesauro, M. (2016). Taking appsec to 11: Appsec pipelines, devops, and making things better. Denver, CO: SnowFROC 2016.Weeks, D. E. (2014). Devops and continuous delivery reference architectures (volume 1 & 2). Fulton, MD: Sonatype.
Agile Testing—DevOps Security Many tools emerging for DevOps application security Begins-ends with microservices—tiny attack surface Includes containers, testing, & real-time monitoring
42XeniaLabs. (2016). Periodic table of devops tools. Retrieved April 11, 2016, from https://xebialabs.com/periodic-table-of-devops-tools.Weeks, D. E. (2014). Devops and continuous delivery reference architectures (volume 1 & 2). Fulton, MD: Sonatype.
Agile Testing—DevOps Periodic Tab.
Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.Fredrick, J. (2008). Accelerate software delivery with continuous integration and testing. Japanese Symposium on Software Testing, Tokyo, Japan.
Most agile testing tools are “free” open source Build server costs no more than a commodity PC 10x more efficient/effective than traditional testing
43
Agile Testing—CI Stats—Cont’d
Traditional testing is a late, manual process Agile testing is an early and automated process Goal to deliver early & often and V&V components
44Rico, D. F. (2012). Agile testing resources. Retrieved Sep. 9, 2012, from http://davidfrico.com/agile-testing-resources.txtCrispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.
AGILE TESTING- Early Incremental Testing -
TRADITIONAL TESTING- Late Big Bang Integration Testing -
Code is Frequently Checked InCode Automatically RetrievedCode Automatically CompiledTests Automatically Executed Instant Feedback & Test Reports
Code Checked In Late in ProjectCode Manually Submitted to TestCode Manually Compiled & BuiltTests Manually Executed LateLate Project Feedback & Reports
Code Automatically DeployedLate Defects Freeze Projects
Agile Testing—vs. Traditional
Test Criteria Accompany StoriesAutomated Tests Written FirstUnits Coded-Tested One at Time
Test Criteria Written After FactManual Tests Written Much LaterUnits Coded Late All at One Time
Agile teams don’t often use TDD, CI, CD & DevOps Implement independent test teams after Sprints done Sprint Waterfalling, Scrummerfalling, & Wagile result
45Heusser, M. (2015). 12 years of agile testing: What do we know now. Proceedings of the Agile Gathering, Grand Rapids, Michigan, USA.
Incorrect• Phased Testing• Separate Teams• Delayed Testing
Correct• Integrated Testing• Integrated Teams• Continuous Testing
Agile Testing—Anti-Patterns
Agile testing slows down with very large systems Slow testing slows integration and increases bugs Agile testing can speed back up with more attention
46Kokko, H. (2009). Increase productivity with large scale continuous integration. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.
MICRO ADJUSTMENTS- Focused Impact Tuning-
MACRO ADJUSTMENTS- Wide Impact Tuning-
Reduce or Remove Test TimeoutsSelect Different TestsRefactor Code & ComponentsTune Network & SoftwareTune Database & Middleware
Remove Process RandomnessUse Faster Code & Test Tools Incremental vs. Big Bang TestsParallelize Build & InstallTune & Optimize Build Process
Agile Testing—Scaling Practices
Add More CPUs & MemoryParallelize System BuildsReplace 3rd Party Test Libraries
In-Memory CompilationParallelize Test RunsPre-Install Test Libraries
Industry very slow in adopting agile testing model Cost, difficulty, and territorialism are common issues Developers must take initiative for disciplined testing
47
Technical BarriersOrganizational BarriersDevelopers don’t want to test
· Infrequently committing code· Committing broken code· Failing to immediately fix builds· Not writing automated tests· Not ensuring 100% of tests pass· Not running private builds· Resorting to traditional testing
Resistance to change· Fear of investment costs· Fear of learning new skills· Test group territorialism· Organizational policy conflicts· Overhead of maintaining CI· Complexity and scaling· Not developing a quality culture
··
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
Agile Testing—Common Barriers
Agile test use is low in spite of its age, i.e., 15 years Many do not understand its utter simplicity and power Failure to use agile testing undermines project success
48Kim, D. (2013). The state of scrum: Benchmarks and guidelines. Indianapolis, IN: Scrum Alliance.
Agile PracticesRetrospectives
Refactoring
Done Definition
Test Tools
Test Driven Dev.
CM Tools
Simplicity
Pair Programming
Technical Debt
Agile Testing 13%
Continuous Integrations
Weekly
Daily
2-3 TimesPer Day
Never
2-3Times
PerIteration
Agile Testing—Usage Statistics
Microsoft created software security life cycle in 2002 Waterfall approach tailored for Scrum sprints in 2009 Uses security req, threat modeling & security testing
49
Microsoft. (2011). Security development lifecycle: SDL Process Guidance (Version 5.1). Redmond, WA: Author.Microsoft. (2010). Security development lifecycle: Simplified implementation of the microsoft SDL. Redmond, WA: Author.Microsoft. (2009). Security development lifecycle: Security development lifecycle for agile development (Version 1.0). Redmond, WA: Author.Bidstrup, E., & Kowalczyk, E. C. (2005). Security development lifecycle. Changing the software development process to build in security from the start. Security Summit West.
SEE DETAILED - SECURITY LIFE CYCLE STEPShttp://davidfrico.com/agile-security-lifecycle.txt
Agile Testing—Security Life Cycle
Agile for EMBEDDED SYSTEMS 1st-generation systems used hardwired logic 2nd-generation systems used PROMS & FPGAs 3rd-generation systems use APP. SW & COTS HW
50Pries, K. H., & Quigley, J. M. (2010). Scrum project management. Boca Raton, FL: CRC Press.Pries, K. H., & Quigley, J. M. (2009). Project management of complex and embedded systems. Boca Raton, FL: Auerbach Publications.Thomke, S. (2003). Experimentation matters: Unlocking the potential of new technologies for innovation. Boston, MA: Harvard Business School Press.
● Short Lead● Least Cost● Lowest Risk● 90% Software● COTS Hardware● Early, Iterative Dev.● Continuous V&V
● Moderate Lead● Moderate Cost● Moderate Risk● 50% Hardware● COTS Components● Midpoint Testing● “Some” Early V&V
● Long Lead● Highest Cost● Highest Risk● 90% Hardware● Custom Hardware● Linear, Staged Dev.● Late Big-Bang I&T
AGILE“Software Model”- MOST FLEXIBLE -
NEO-TRADITIONAL“FPGA Model”
- MALLEABLE -
TRADITIONAL“Hardwired Model”
- LEAST FLEXIBLE -
GOAL – SHIFT FROM LATE HARDWARE TO EARLIER SOFTWARE SOLUTION
RISKEmbeddedSystemsMore HWThan SW
STOPCompeting
With HW
STARTCompeting
With SW
Iter
atio
ns, I
nteg
rati
ons,
& V
alid
atio
ns
Key Agile SCALING POINTERS One must think and act small to accomplish big things Slow down to speed up, speed up ‘til wheels come off Scaling up lowers productivity, quality, & business value
51Rico, D. F. (2014). Dave's Notes: For Scaling with SAFe, DaD, LeSS, RAGE, ScrumPLoP, Enterprise Scrum, etc. Retrieved March 28, 2014 from http://davidfrico.com
EMPOWER WORKFORCE - Allow workers to help establish enterprise business goals and objectives.
ALIGN BUSINESS VALUE - Align and focus agile teams on delivering business value to the enterprise.
PERFORM VISIONING - Frequently communicate portfolio, project, and team vision on continuous basis.
REDUCE SIZE - Reduce sizes of agile portfolios, acquisitions, products, programs, projects, and teams.
ACT SMALL - Get large agile teams to act, behave, collaborate, communicate, and perform like small ones.
BE SMALL - Get small projects to act, behave, and collaborate like small ones instead of trying to act larger.
ACT COLLOCATED - Get virtual distributed teams to act, behave, communicate and perform like collocated ones.
USE SMALL ACQUISITION BATCHES - Organize suppliers to rapidly deliver new capabilities and quickly reprioritize.
USE LEAN-AGILE CONTRACTS - Use collaborative contracts to share responsibility instead of adversarial legal ones.
USE ENTERPRISE AUTOMATION - Automate everything with Continuous Integration, Continuous Delivery, & DevOps.
Patterson, K., et al. (2008). Influencer: The power to change anything: New York, NY: McGraw-Hill.Pink, D. H. (2009). Drive: The surprising truth about what motivates us. New York, NY: Riverhead Books.Heath, C., & Heath, D. (2010). Switch: How to change things when change is hard. New York, NY: Random House.Pink, D. H. (2012). To sell is human: The surprising truth about moving others. New York, NY: Riverhead Books.Heath, C., & Heath, D. (2013). Decisive: How to make better choices in life and work. New York, NY: Random House.
Change, no matter how small or large, is difficult Smaller focused changes help to cross the chasm Validating, simplifying, & incrementalism are keys
52
INFLUENCER
Create new experiences Create new motives
Perfect complex skills Build emotional skills
Recruit public figures Recruit influential leaders
Utilize teamwork Power of social capital
Use incentives wisely Use punishment sparingly
Make it easy Make it unavoidable
MAKE IT DESIRABLE
SURPASS YOUR LIMITS
USE PEER PRESSURE
STRENGTH IN NUMBERS
DESIGN REWARDS
CHANGE ENVIRONMENT
DRIVE
PURPOSE
AUTONOMY
MASTERY
Purpose-profit equality Business& societal benefit Share control of profits Delegate implementation Culture & goal alignment Remake society-globe
Accountable to someone Self-select work tasks Self-directed work tasks Self-selected timelines Self-selected teams Self-selected implement.
Experiment & innovate Align tasks to abilities Continuously improve Learning over profits Create challenging tasks Set high expectations
A-B-C
Reduce Your Power Take Their Perspective Use Strategic Mimicry
Use Interrogative Self-Talk Opt. Positivity Ratios Offer Explanatory Style
Find the Right Problem Find Your Frames Find an Easy Path
ATTUNEMENT
BUOYANCY
CLARITY
SWITCH
Follow the bright spots Script the critical moves Point to the destination
Find the feeling Shrink the change Grow your people
Tweak the environment Build habits Rally the herd
DIRECT THE RIDER
MOTIVATE ELEPHANT
SHAPE PATH
DECISIVE
COMMON ERRORS Narrow framing Confirmation bias Short term emotion Over confidence
WIDEN OPTIONS Avoid a narrow frame Multi-track Find out who solved it
TEST ASSUMPTIONS Consider the opposite Zoom out & zoom in Ooch
ATTAIN DISTANCE Overcome emotion Gather & shift perspective Self-directed work tasks
PREPARE TO BE WRONG Bookend the future Set a tripwire Trust the process
Models of AGILE ORG. CHANGE
Agile vs. Traditional Success Traditional projects succeed at 50% industry avg. Traditional projects are challenged 20% more often Agile projects succeed 3x more and fail 3x less often
Standish Group. (2012). Chaos manifesto. Boston, MA: Author.
53
Agile Traditional
Success42%
Failed9%
Challenged49%
Success14%
Failed29%
Challenged57%
Hoque, F., et al. (2007). Business technology convergence. The role of business technology convergence in innovation and adaptability and its effect on financial performance. Stamford, CT: BTM Corporation. 54
Study of 15 agile vs. non-agile Fortune 500 firms Based on models to measure organizational agility Agile firms out perform non agile firms by up to 36%
Fin. Benefits to ENTERPRISE AGILITY
Suhy, S. (2014). Has the U.S. government moved to agile without telling anyone? Retrieved April 24, 2015, from http://agileingov.comPorter, M. E., & Schwab, K. (2008). The global competitiveness report: 2008 to 2009. Geneva, Switzerland: World Economic Forum. 55
U.S. gov’t agile jobs grew by 13,000% from 2006-2013 Adoption is higher in U.S. DoD than Civilian Agencies GDP of countries with high adoption rates is greater
High
Low
Low HighAGILITY
CO
MP
ET
ITIV
EN
ES
S
GOVERNMENT AGILE JOB GROWTH
PE
RC
EN
TAG
E
13,000%
02006 2013YEARS
GOVERNMENT COMPETITIVENESS
Nat’l Benefits to ENTERPRISE AGILITY