data-driven software mastery @open mastery austin

Download Data-Driven Software Mastery @Open Mastery Austin

Post on 08-Jan-2017

299 views

Category:

Software

1 download

Embed Size (px)

TRANSCRIPT

  • Janelle Kleinopenmastery.org @janellekz

    Tour Speaker

    Data-Driven Software Mastery

    newiron.com

    http://@openmasteryhttp://newiron.com

  • , Developer, Consultant, CTO @New IronSpecialized in Statistical Process Control (SPC) and Supply Chain Optimization from Lean Manufacturing (I
  • LEARN YOUR WAY TO AWESOME.

    Community of Practice for Data-Driven Software Mastery

    Why Are We Here?

  • Why Should You Care?

    Problem #1: Organizational Dysfunction

  • Why Should You Care?

    Problem #2: Solving the Wrong Problem

  • Why Should You Care?

    This talk is about a REALISTIC STRATEGY

    to solve these problems.

  • Im going to show you:

    POINT A

    (Ive been working on this for 5 years)

  • I Need Your Help to Get to:

    POINT AWESOME

    Point A Point AWESOME?

    Iterate!

  • LEARN YOUR WAY TO AWESOME.

    Why Are We Here?

    Learning Takes Work.

  • Lets Make the PAIN Visible!

    #OpenDX

    Our Campaign:

  • This isnt About Me.

    This is About ALL OF US.

  • Five Years Ago I Read This Book:

    How to Build a Learning Organization

  • Five Disciplines of a Learning Organization

    Personal Mastery

    Mental Models

    Shared Vision

    Team Learning

    Systems Thinking

    These disciplines were emergent practice on our team.

  • Five Disciplines of a Learning Organization

    Personal Mastery

    Mental Models

    Shared Vision

    Team Learning

    Systems Thinking

    These disciplines are emergent practice in software development

  • Whats the Gap?

    What was different about our team?

  • My Day Job is Technical Assessment

    GeorgeMe

    What does better quality mean to you?

  • What does better really mean?

  • What does better really mean?

    Where does my better come from?

  • Better

    Fifth Discipline: Shared Vision = Shared Better

    The Key to Team Learning:

    Better

  • Five Disciplines of a Learning Organization

    Personal Mastery

    Mental Models

    Team Learning

    Systems Thinking

    This book is about the art of group problem-solving.

    Shared LearningShared Better

  • Five Years to Put Better into Words

    How to Measure the PAIN in Software Development

    Janelle Klein

    This is the my story.

  • We were trying to do all the right things.

    About 8 Years Ago

  • We were building a factory automation system...

  • We shipped to production...

  • We shipped to production... (again)

  • We couldnt reproduce the problem.

  • We shipped to production... (AGAIN)

    The rollback failed!

  • Project FAILURE

  • The Retrospective

    What are we going to do?!

    Our biggest problem

    Well, we know weve got a quality problem right?

  • The Retrospective

    What are we going to do?!

    Our biggest problem

    The problem is we dont have enough test automation!

  • So the Test Automation Began

    Our regression testing took 3-4 weeks

    Lets automate the tests!

  • 20%

    Percent Automated:

    So the Test Automation Began

  • 50%

    Maintenance started getting PAINFUL

    Lets refactor the tests!

    Percent Automated:

    So the Test Automation Began

  • 80%

    It was still really PAINFUL

    Well, at least our regression cycle is faster, right?

    Our regression cycle still took 3-4 weeks!

    Percent Automated:

    So the Test Automation Began

  • ApplicationTests

    Our Test Suite

  • ApplicationTests

    Our Test Suite

    Any ideas?

  • The Retrospective

    What are we supposed to do?!

    Our biggest problem

    Well, if we dont understand a problem, we should

    collect data.

  • The Retrospective

    Our biggest problem

    What data would help us understand the problem?

    What are we supposed to do?!

  • Technical Debt Mistakes

    I thought the main obstacle was Technical Debt

  • ?Most of our mistakes were in the

    most well-written parts of the code.

    Mistakes

  • We made significantly more mistakes in code that we didnt write ourselves.

    Lower Familiarity

    More Mistakes=

    There had to be more to the story...

  • Complex(So*ware(

    PAIN

    This is what I knew...

    What made development feel painful?

  • Unexpected Behavior

    Problem Resolved

    Tracking Painful Interaction with the Code (Friction)

    Troubleshooting

    Progress

    5 hours and 18 minutes of troubleshooting...

    PAINFUL

  • The amount of PAIN was caused by

    Likeliness(of((Unexpected(Behavior(

    Cost(to(Troubleshoot(and(Repair(

    High(Frequency(Low(Impact(

    Low(Frequency(Low(Impact(

    Low(Frequency(High(Impact(

    PAIN(

  • What Causes Unexpected Behavior (likeliness)?

    What Makes Troubleshooting Time-Consuming (impact)?

    Semantic Mistakes

    Stale Memory Mistakes

    Association Mistakes

    Bad Input Assumption

    Tedious Change Mistakes

    Copy-Edit Mistakes

    Transposition Mistakes

    Failed Refactor Mistakes

    False Alarm

    Non-Deterministic Behavior

    Ambiguous Clues

    Lots of Code Changes

    Noisy Output

    Cryptic Output

    Long Execution Time

    Environment Cleanup

    Test Data Creation

    Using Debugger

    Most of the pain was caused by human factors.

    What causes PAIN?

  • What Causes Unexpected Behavior (likeliness)?

    What Makes Troubleshooting Time-Consuming (impact)?

    Non-Deterministic Behavior

    Ambiguous Clues

    Lots of Code Changes

    Noisy Output

    Cryptic Output

    Long Execution Time

    Environment Cleanup

    Test Data Creation

    Using Debugger

    What causes PAIN?

    Most of the pain was caused by human factors.

    Semantic Mistakes

    Stale Memory Mistakes

    Association Mistakes

    Bad Input Assumption

    Tedious Change Mistakes

    Copy-Edit Mistakes

    Transposition Mistakes

    Failed Refactor Mistakes

    False Alarm

  • What Causes Unexpected Behavior (likeliness)?

    What Makes Troubleshooting Time-Consuming (impact)?

    Non-Deterministic Behavior

    Ambiguous Clues

    Lots of Code Changes

    Noisy Output

    Cryptic Output

    Long Execution Time

    Environment Cleanup

    Test Data Creation

    Using Debugger

    What causes PAIN?

    Semantic Mistakes

    Stale Memory Mistakes

    Association Mistakes

    Bad Input Assumption

    Tedious Change Mistakes

    Copy-Edit Mistakes

    Transposition Mistakes

    Failed Refactor Mistakes

    False Alarm

    Most of the pain was caused by human factors.

  • What Causes Unexpected Behavior (likeliness)?

    What Makes Troubleshooting Time-Consuming (impact)?

    Non-Deterministic Behavior

    Ambiguous Clues

    Lots of Code Changes

    Noisy Output

    Cryptic Output

    Long Execution Time

    Environment Cleanup

    Test Data Creation

    Using Debugger

    What causes PAIN?

    PAIN is a consequence of how we interact with the code.

    Semantic Mistakes

    Stale Memory Mistakes

    Association Mistakes

    Bad Input Assumption

    Tedious Change Mistakes

    Copy-Edit Mistakes

    Transposition Mistakes

    Failed Refactor Mistakes

    False Alarm

  • PAIN occurs during the process of understanding and extending the software

    Complex(So*ware(

    PAIN

    Not the Code.

    Optimize Idea Flow

  • Analyzing the Types of Mistakes

    #1 Cause of Mistakes: Misunderstandings of how the system worked

  • ApplicationTests

    The Bugs were in Our Conceptual Models

    Fix: We built a Data Dictionary

  • The Coat of Armor Principle

    Software

    Coat of Armor

  • Alarm Pain

    Six hours of troubleshooting and repairing tests when nothing is broken.

  • The Waxy Coating Principle

    Optimize the signal to noise ratio.

    Software (before)

    Software (after)

    Tests are like a waxy coating poured over the code.

  • We deleted everything

    The automation wasnt solving the problem!

  • What are the specific risks in this release?

    How can we mitigate the risks?

    Handful of manual tests

    Custom Tools

    +

  • Release Size

    (weeks)

    Time

    Defect Rates

    We Could See the Effects

    Effective Risk Management Looks Like This

  • My team spent tons of time working on improvements that didnt make much difference.

    We had tons of automation, but the automation didnt catch our bugs.

  • My team spent tons of time working on improvements that didnt make much difference.

    We had well-modularized code,

    but it was still extremely time-consuming to troubleshoot defects.

  • The hard part isnt solving the problems its identifying the right problems to solve.

    What are the specific problems that are causing the teams pain?

  • Better

    Fifth Discipline: Shared Vision = Shared Better

    The Key to Team Learning:

    Better

  • Better following best practices

    (solution-focused) (problem-focused)

    Better solving the problems

    What does better really mean?

    How do we identify the right problems to solve?

  • Cost & Risk are a Funct