Data-Driven Software Mastery @Open Mastery Austin

Download Data-Driven Software Mastery @Open Mastery Austin

Post on 08-Jan-2017

297 views

Category:

Software

1 download

Embed Size (px)

TRANSCRIPT

<ul><li><p>Janelle Kleinopenmastery.org @janellekz</p><p>Tour Speaker</p><p>Data-Driven Software Mastery</p><p>newiron.com</p><p>http://@openmasteryhttp://newiron.com</p></li><li>, Developer, Consultant, CTO @New IronSpecialized in Statistical Process Control (SPC) and Supply Chain Optimization from Lean Manufacturing (I </li><li><p>LEARN YOUR WAY TO AWESOME.</p><p>Community of Practice for Data-Driven Software Mastery </p><p>Why Are We Here?</p></li><li><p>Why Should You Care?</p><p>Problem #1: Organizational Dysfunction</p></li><li><p>Why Should You Care?</p><p>Problem #2: Solving the Wrong Problem</p></li><li><p>Why Should You Care?</p><p>This talk is about a REALISTIC STRATEGY </p><p>to solve these problems. </p></li><li><p>Im going to show you:</p><p>POINT A</p><p>(Ive been working on this for 5 years)</p></li><li><p>I Need Your Help to Get to:</p><p>POINT AWESOME</p><p>Point A Point AWESOME?</p><p>Iterate!</p></li><li><p>LEARN YOUR WAY TO AWESOME.</p><p>Why Are We Here?</p><p>Learning Takes Work.</p></li><li><p>Lets Make the PAIN Visible!</p><p>#OpenDX</p><p>Our Campaign:</p></li><li><p>This isnt About Me.</p><p>This is About ALL OF US.</p></li><li><p>Five Years Ago I Read This Book:</p><p>How to Build a Learning Organization</p></li><li><p>Five Disciplines of a Learning Organization</p><p>Personal Mastery</p><p>Mental Models</p><p>Shared Vision</p><p>Team Learning</p><p>Systems Thinking</p><p>These disciplines were emergent practice on our team.</p></li><li><p>Five Disciplines of a Learning Organization</p><p>Personal Mastery</p><p>Mental Models</p><p>Shared Vision</p><p>Team Learning</p><p>Systems Thinking</p><p>These disciplines are emergent practice in software development</p></li><li><p>Whats the Gap?</p><p>What was different about our team?</p></li><li><p>My Day Job is Technical Assessment</p><p>GeorgeMe</p><p>What does better quality mean to you?</p></li><li><p>What does better really mean?</p></li><li><p>What does better really mean?</p><p>Where does my better come from?</p></li><li><p>Better</p><p>Fifth Discipline: Shared Vision = Shared Better</p><p>The Key to Team Learning: </p><p>Better</p></li><li><p>Five Disciplines of a Learning Organization</p><p>Personal Mastery</p><p>Mental Models</p><p>Team Learning</p><p>Systems Thinking</p><p>This book is about the art of group problem-solving.</p><p>Shared LearningShared Better</p></li><li><p>Five Years to Put Better into Words</p><p>How to Measure the PAIN in Software Development</p><p>Janelle Klein</p><p>This is the my story.</p></li><li><p>We were trying to do all the right things.</p><p>About 8 Years Ago</p></li><li><p>We were building a factory automation system...</p></li><li><p>We shipped to production...</p></li><li><p>We shipped to production... (again)</p></li><li><p>We couldnt reproduce the problem.</p></li><li><p>We shipped to production... (AGAIN)</p><p>The rollback failed!</p></li><li><p>Project FAILURE</p></li><li><p>The Retrospective</p><p>What are we going to do?!</p><p>Our biggest problem</p><p>Well, we know weve got a quality problem right?</p></li><li><p>The Retrospective</p><p>What are we going to do?!</p><p>Our biggest problem</p><p>The problem is we dont have enough test automation!</p></li><li><p>So the Test Automation Began</p><p>Our regression testing took 3-4 weeks</p><p>Lets automate the tests!</p></li><li><p>20%</p><p>Percent Automated:</p><p>So the Test Automation Began</p></li><li><p>50%</p><p>Maintenance started getting PAINFUL</p><p>Lets refactor the tests!</p><p>Percent Automated:</p><p>So the Test Automation Began</p></li><li><p>80%</p><p>It was still really PAINFUL</p><p>Well, at least our regression cycle is faster, right?</p><p>Our regression cycle still took 3-4 weeks!</p><p>Percent Automated:</p><p>So the Test Automation Began</p></li><li><p>ApplicationTests</p><p>Our Test Suite</p></li><li><p>ApplicationTests</p><p>Our Test Suite</p><p>Any ideas?</p></li><li><p>The Retrospective</p><p>What are we supposed to do?!</p><p>Our biggest problem</p><p>Well, if we dont understand a problem, we should </p><p>collect data.</p></li><li><p>The Retrospective</p><p>Our biggest problem</p><p>What data would help us understand the problem?</p><p>What are we supposed to do?!</p></li><li><p>Technical Debt Mistakes</p><p>I thought the main obstacle was Technical Debt</p></li><li><p>?Most of our mistakes were in the </p><p>most well-written parts of the code.</p><p>Mistakes</p></li><li><p>We made significantly more mistakes in code that we didnt write ourselves.</p><p>Lower Familiarity</p><p>More Mistakes=</p><p>There had to be more to the story...</p></li><li><p>Complex(So*ware(</p><p>PAIN</p><p>This is what I knew...</p><p>What made development feel painful?</p></li><li><p>Unexpected Behavior</p><p>Problem Resolved</p><p>Tracking Painful Interaction with the Code (Friction)</p><p>Troubleshooting</p><p>Progress</p><p>5 hours and 18 minutes of troubleshooting...</p><p>PAINFUL</p></li><li><p>The amount of PAIN was caused by</p><p>Likeliness(of((Unexpected(Behavior(</p><p>Cost(to(Troubleshoot(and(Repair(</p><p>High(Frequency(Low(Impact(</p><p>Low(Frequency(Low(Impact(</p><p>Low(Frequency(High(Impact(</p><p>PAIN(</p></li><li><p>What Causes Unexpected Behavior (likeliness)?</p><p>What Makes Troubleshooting Time-Consuming (impact)?</p><p>Semantic Mistakes </p><p>Stale Memory Mistakes </p><p>Association Mistakes </p><p>Bad Input Assumption </p><p>Tedious Change Mistakes </p><p>Copy-Edit Mistakes </p><p>Transposition Mistakes </p><p>Failed Refactor Mistakes </p><p>False Alarm</p><p>Non-Deterministic Behavior </p><p>Ambiguous Clues </p><p>Lots of Code Changes </p><p>Noisy Output </p><p>Cryptic Output </p><p>Long Execution Time </p><p>Environment Cleanup </p><p>Test Data Creation </p><p>Using Debugger</p><p>Most of the pain was caused by human factors.</p><p>What causes PAIN?</p></li><li><p>What Causes Unexpected Behavior (likeliness)?</p><p>What Makes Troubleshooting Time-Consuming (impact)?</p><p>Non-Deterministic Behavior </p><p>Ambiguous Clues </p><p>Lots of Code Changes </p><p>Noisy Output </p><p>Cryptic Output </p><p>Long Execution Time </p><p>Environment Cleanup </p><p>Test Data Creation </p><p>Using Debugger</p><p>What causes PAIN?</p><p>Most of the pain was caused by human factors.</p><p>Semantic Mistakes </p><p>Stale Memory Mistakes </p><p>Association Mistakes </p><p>Bad Input Assumption </p><p>Tedious Change Mistakes </p><p>Copy-Edit Mistakes </p><p>Transposition Mistakes </p><p>Failed Refactor Mistakes </p><p>False Alarm</p></li><li><p>What Causes Unexpected Behavior (likeliness)?</p><p>What Makes Troubleshooting Time-Consuming (impact)?</p><p>Non-Deterministic Behavior </p><p>Ambiguous Clues </p><p>Lots of Code Changes </p><p>Noisy Output </p><p>Cryptic Output </p><p>Long Execution Time </p><p>Environment Cleanup </p><p>Test Data Creation </p><p>Using Debugger</p><p>What causes PAIN?</p><p>Semantic Mistakes </p><p>Stale Memory Mistakes </p><p>Association Mistakes </p><p>Bad Input Assumption </p><p>Tedious Change Mistakes </p><p>Copy-Edit Mistakes </p><p>Transposition Mistakes </p><p>Failed Refactor Mistakes </p><p>False Alarm</p><p>Most of the pain was caused by human factors.</p></li><li><p>What Causes Unexpected Behavior (likeliness)?</p><p>What Makes Troubleshooting Time-Consuming (impact)?</p><p>Non-Deterministic Behavior </p><p>Ambiguous Clues </p><p>Lots of Code Changes </p><p>Noisy Output </p><p>Cryptic Output </p><p>Long Execution Time </p><p>Environment Cleanup </p><p>Test Data Creation </p><p>Using Debugger</p><p>What causes PAIN?</p><p>PAIN is a consequence of how we interact with the code.</p><p>Semantic Mistakes </p><p>Stale Memory Mistakes </p><p>Association Mistakes </p><p>Bad Input Assumption </p><p>Tedious Change Mistakes </p><p>Copy-Edit Mistakes </p><p>Transposition Mistakes </p><p>Failed Refactor Mistakes </p><p>False Alarm</p></li><li><p>PAIN occurs during the process of understanding and extending the software</p><p>Complex(So*ware(</p><p>PAIN</p><p>Not the Code.</p><p>Optimize Idea Flow</p></li><li><p>Analyzing the Types of Mistakes</p><p>#1 Cause of Mistakes: Misunderstandings of how the system worked </p></li><li><p>ApplicationTests</p><p>The Bugs were in Our Conceptual Models</p><p>Fix: We built a Data Dictionary</p></li><li><p>The Coat of Armor Principle</p><p>Software</p><p>Coat of Armor</p></li><li><p>Alarm Pain</p><p>Six hours of troubleshooting and repairing tests when nothing is broken.</p></li><li><p>The Waxy Coating Principle</p><p>Optimize the signal to noise ratio.</p><p>Software (before)</p><p>Software (after)</p><p>Tests are like a waxy coating poured over the code. </p></li><li><p>We deleted everything</p><p>The automation wasnt solving the problem!</p></li><li><p>What are the specific risks in this release?</p><p>How can we mitigate the risks?</p><p>Handful of manual tests</p><p>Custom Tools</p><p>+</p></li><li><p>Release Size </p><p>(weeks)</p><p>Time</p><p>Defect Rates</p><p>We Could See the Effects</p><p>Effective Risk Management Looks Like This</p></li><li><p>My team spent tons of time working on improvements that didnt make much difference.</p><p>We had tons of automation, but the automation didnt catch our bugs.</p></li><li><p>My team spent tons of time working on improvements that didnt make much difference.</p><p>We had well-modularized code, </p><p>but it was still extremely time-consuming to troubleshoot defects.</p></li><li><p>The hard part isnt solving the problems its identifying the right problems to solve.</p><p>What are the specific problems that are causing the teams pain?</p></li><li><p>Better</p><p>Fifth Discipline: Shared Vision = Shared Better</p><p>The Key to Team Learning: </p><p>Better</p></li><li><p>Better following best practices</p><p>(solution-focused) (problem-focused)</p><p>Better solving the problems</p><p>What does better really mean?</p><p>How do we identify the right problems to solve?</p></li><li><p>Cost &amp; Risk are a Function of Increased Difficulty</p><p>Cost &amp; </p><p>Risk</p><p>Difficulty of Work</p><p>The Difficulty of Doing Our Jobs</p><p>Human Limitations</p></li><li><p>Likelihood)of))Unexpected)Behavior)</p><p>Cost)to)Troubleshoot)and)Repair)</p><p>High)Frequency)Low)Impact)</p><p>Low)Frequency)Low)Impact)</p><p>Low)Frequency)High)Impact)</p><p>PAIN)</p><p>Better = Reduce the Risk</p><p>Reduce Likeliness = </p><p>Mistake Proofing</p><p>Reduce Recovery Cost = Mistake Tolerance</p></li><li><p>What Exactly does Idea Flow Measure?</p></li><li><p>Complex(So*ware(</p><p>PAIN</p><p>Not the Code.</p><p>Optimize Idea Flow</p></li><li><p>FRICTION is a Function of Increased Difficulty</p><p>Cost &amp; </p><p>Risk</p><p>Difficulty of Work</p><p>The Difficulty of Doing Our Jobs</p><p>Human Limitations</p></li><li><p>The Rhythm of Idea Flow </p><p>Write a little code.</p><p>Work out the kinks.</p><p>Write a little code.</p><p>Work out the kinks.</p><p>Write a little code.</p><p>Work out the kinks.</p></li><li><p>Conflict Confirm </p><p>Rework'</p><p>Learn'</p><p>Validate(</p><p>Modify'</p><p>Progress Loop Conflict Loop</p><p>Troubleshoot'</p><p>Write a little code. Work out the kinks.</p><p>The Rhythm of Idea Flow </p></li><li><p>On Intelligence Jeff Hawkins</p></li><li><p>Confirmed Expectations</p><p>Our brain works like a giant prediction engine</p></li><li><p>Conflicting Expectations</p><p>Our brain works like a giant prediction engine</p></li><li><p>Conflict Confirm </p><p>Rework'</p><p>Learn'</p><p>Validate(</p><p>Modify'</p><p>Progress Loop Conflict Loop</p><p>Troubleshoot'</p><p>The Rhythm of Idea Flow </p></li><li><p>Friction in Idea Flow</p><p>Conflict Confirm </p><p>Rework'</p><p>Learn'</p><p>Validate(</p><p>Modify'</p><p>Progress Loop Conflict Loop</p><p>Troubleshoot'</p></li><li><p>Idea Flow Map</p><p>Friction</p><p>A conflict starts with an unexpected observation and ends when understanding is resolved.</p></li><li><p>Idea Flow Mapping Tools </p><p>(Open Source, Supported GA ~June 2016)github.com/ideaflow/tools</p><p>http://ideaflow.org/tools</p></li><li><p>Typical Idea Flow Maps</p><p>Single Problem</p><p>Multi-Problem</p></li><li><p>Reading Visual Indicators in Idea Flow Maps</p><p>Le#$Atrium$</p><p>Le#$Ventricle$</p><p>Right$Ventricle$</p><p>Right$Atrium$</p><p>Whats$causing$this$pa7ern?$</p><p>Similar to how an EKG helps doctors diagnose heart problems...</p></li><li><p>...Idea Flow Maps help developers diagnose software problems.</p><p>Problem-Solving Machine</p><p>Reading Visual Indicators in Idea Flow Maps</p></li><li><p>= Solution Strategy</p><p>The Heart of Software Development(the problem-solving machine)</p></li><li><p>= Solution Strategy</p><p>The Heart of Software Development(the problem-solving machine)</p><p>Choose a general strategy</p></li><li><p>= Solution Strategy</p><p>The Heart of Software Development(the problem-solving machine)</p><p>Understand the system</p></li><li><p>= Solution Strategy</p><p>The Heart of Software Development(the problem-solving machine)</p><p>Code &amp; work out the kinks</p></li><li><p>= Solution Strategy</p><p>The Heart of Software Development(the problem-solving machine)</p><p>Back to the drawing board</p></li><li><p>The symptoms we experience depend on where the disruptions are in the problem-solving process</p></li><li><p>Trial and Error</p><p>The Friction is Here</p><p>Troubleshooting</p><p>Progress</p><p>Learning</p><p>Rework</p></li><li><p>Assumption Risk (Rework)</p><p>Likelihood)of))making)a))</p><p>Bad)Assump4on)</p><p>Cost)to)Correct)Decisions)</p><p>High)Uncertainty)Low)Delay)</p><p>Low)Uncertainty)Low)Delay)</p><p>Low)Uncertainty)High)Delay)</p><p>PAIN)</p></li><li><p>Struggling with Understanding the Code</p><p>The Friction is Here</p><p>Troubleshooting</p><p>Progress</p><p>Learning</p><p>Rework</p></li><li><p>Familiarity Risk (Learning)</p><p>Likelihood)of))working)with)Unfamiliar)</p><p>Code)</p><p>Cost)to)Learn)</p><p>High)Frequency)Easy)to)Learn)</p><p>Low)Frequency)Easy)to)Learn)</p><p>Low)Frequency)Hard)to)Learn)</p><p>PAIN)</p></li><li><p>Struggling with Feedback</p><p>The Friction is Here</p><p>Troubleshooting</p><p>Progress</p><p>Learning</p><p>Rework</p></li><li><p>Quality Risk (Troubleshooting)</p><p>Likelihood)of))Unexpected)Behavior)</p><p>Cost)to)Troubleshoot)and)Repair)</p><p>High)Frequency)Low)Impact)</p><p>Low)Frequency)Low)Impact)</p><p>Low)Frequency)High)Impact)</p><p>PAIN)</p></li><li><p>measures the time spent on:</p><p>Idea Flow</p><p>x</p><p>Troubleshooting</p><p>x</p><p>Learning</p><p>x</p><p>Rework</p><p>Quality Risk Familiarity Risk Assumption Risk</p></li><li><p>What percentage of time do you think your team spends on friction?</p></li><li><p>Troubleshooting</p><p>Progress</p><p>Learning</p><p>7:070:00</p><p>0:00 19:52</p><p>12 year old project after all original developers left.</p><p>Case Study: Huge Mess with Great Team</p><p>70-90% of dev capacity on friction</p></li><li><p>The Teams Improvement Focus: Increasing unit test coverage by 5%</p><p>Case Study: Huge Mess with Great Team</p><p>1. Test Data Generation</p><p>2. Merging Problems</p><p>3. Repairing Tests</p><p>1000 hours/month</p><p>The Biggest Problem: ~700 hours/month generating test data</p></li><li><p>The Retrospective</p><p>Whats the biggest opportunity for improvement?</p><p>Our biggest problem</p></li><li><p>Whats the biggest opportunity for improvement?</p><p>The awful email template engine code!</p><p>Our biggest problem</p><p>The Retrospective</p></li><li><p>Whats the biggest opportunity for improvement?</p><p>Fill in missing unit tests!</p><p>Our biggest problem</p><p>The Retrospective</p></li><li><p>Whats the biggest opportunity for improvement?</p><p>I know how to improve database performance!</p><p>Our biggest problem</p><p>The Retrospective</p></li><li><p>Whats the biggest opportunity for improvement?</p><p>Lets improve maintainability of our test framework!</p><p>Our biggest problem</p><p>The Retrospective</p></li><li><p>Whats the biggest opportunity for improvement?</p><p>Just because a problem comes to mind, doesnt mean its an important problem to solve.</p><p>Our biggest problem</p><p>The Retrospective</p></li><li><p>Whats the biggest opportunity for improvement?</p><p>Our biggest problem</p><p>What do I feel the most intensely about?</p><p>Daniel Kahneman Thinking Fast and Slow</p><p>The Retrospective</p></li><li><p>Whats the biggest opportunity for improvement?</p><p>The awful email template engine code!</p><p>Recency Bias</p><p>Our biggest problem</p><p>The Retrospective</p></li><li><p>Whats the biggest opportunity for improvement?</p><p>Guilt Bias</p><p>Fill in missing unit tests!</p><p>Our biggest problem</p><p>The Retrospective</p></li><li><p>Whats the biggest opportunity for improvement?</p><p>I know how to improve database performance!</p><p>Known Solution Bias</p><p>Our biggest problem</p><p>The Retrospective</p></li><li><p>Whats the biggest opportunity for improvement?</p><p>Sunk Cost Bias</p><p>Lets improve maintainability of our test framework!</p><p>Our biggest problem</p><p>The Retrospective</p></li><li><p>Relying on Gut Feel:</p></li><li><p>18 months after a Micro-Services/Continuous Delivery rewrite.</p><p>Troubleshooting</p><p>Progress</p><p>Learning40-60% of dev capacity on friction</p><p>0:00 28:15</p><p>12:230:00</p><p>Case Study: From Monolith to Microservices</p></li><li><p>The Architecture Looked Good on Paper</p><p>Team A Team B Team C</p><p>Complexity Moved Here</p></li><li><p>We Dont Have TIME To Fix It!</p></li><li><p>PAIN</p><p>The Classic Story of Project Failure</p><p>Problems get deferred </p><p>Builds start breaking </p><p>Releases get chaotic </p><p>Productivity slows to a crawl </p><p>Developers begging for time </p><p>Its never enough </p><p>Project Meltdown</p></li><li><p>The Cost of Escalating Risk</p><p>0%</p><p>100%</p><p>Release 1 Release 2 Release 3</p><p>Troubleshooting</p><p>Progress</p><p>Learning</p><p>Percentage Capacity spent on Troubleshooting (red) and Learning (blue)</p><p>(extrapolated from samples)</p></li><li><p>0%</p><p>100%</p><p>Release 1 Release 2 Release 3</p><p>Percentage Capacity spent on Troubleshooting (red) and Learning (blue)</p><p>Figure out what to do Learning is front-loaded</p><p>Troubleshooting</p><p>Progress</p><p>Learning</p><p>The Cost of Escalating Risk</p></li><li><p>0%</p><p>100%</p><p>Release 1 Release 2 Release 3</p><p>Percentage Capacity spent on Troubleshooting (red) and Learning (blue)</p><p>Rush Before the Deadline Validation is Deferred</p><p>Troubleshooting</p><p>Progress</p><p>Learning</p><p>The Cost of Escalating Risk</p></li><li><p>0%</p><p>100%</p><p>Release 1 Release 2 Release 3</p><p>Percentage Capacity spent on Troubleshooting (red) and Learning (blue)</p><p>Pain Builds Baseline friction keeps rising</p><p>Troubleshooting</p><p>Progress</p><p>Learning</p><p>The Cost of Escalating Risk</p></li><li><p>0%</p><p>100%</p><p>Release 1 Release 2 Release 3</p><p>Percentage Capacity spent on Troubleshooting (red) and Learning (blue)</p><p>Chaos Reigns Unpredictable work stops </p><p>fitting in the timebox</p><p>Troubleshooting</p><p>Progress</p><p>Learning</p><p>The Cost of Escalating Risk</p></li><li><p>This is the Biggest Cause of FAILURE In Our Industry:</p><p>Have you struggled with these problems in your organization?</p></li><li><p>BetterBetter</p><p>Fifth Discipline: Shared Vision = Shared Better</p><p>Developer</p><p>Developer</p><p>Manager Manager</p><p>The Key to Organizational Learning: </p></li><li><p>Lessons Learned</p></li><li><p>How can we reduce the impact of bad architecture assumptions? </p><p>(Assumption Risk)</p><p>The cost of bad architecture decisions in the microservices world </p><p>is EXTREMELY HIGH.</p></li><li><p>If youve got beautiful code</p><p>Because you pushed all the PAIN to your clients</p><p>Your code SUCKS.</p><p>How do we know what the clients experience will be like?</p></li><li><p>Lots of Friction in BRAND NEW code

Recommended

View more >