Working in Chaos & Complexity for Success

May 11-14, 2009

Richard M. Wallace

Agile Development Methods

in Current Environments

Outline

• Agile does work– Proven in commercial industry, very late to government

procurement

• Software is not manufactured– Government sees software development as a manufacturing

process– Procurement treats software as a "mature technology" item

• Chaos is always present – get used to it– In all development chaos and complexities arise preventing

wanted functionality

• Success is possible with simple steps– Agile development, a framework for traversing organizational

social complexity, and efficient processes equals delivering wanted functionality

Agile Does Work

Proven in commercial industry,very late to government procurement

Agile Accepted in Commercial Industry

The 3rd Annual "State of Agile Development" survey was conducted and sponsored by VersionOne. This global survey ran in June and July of 2008 and received 2,319 completed surveys with 3,061 total respondents from 80 countries.

Percentage of Successful Agile Projects1

1: 3rd Annual "State of Agile Development" survey

3.8%

11.5%

21.2%

55.0%

3.7%4.8%

0

10

25

50

75

90-100

Agile is Fully Adopted in Commercial Industry

Crossing the Chasm, Geoffrey A. Moore, Collins Business; Revised edition (August 20, 2002) ISBN-10: 0060517123

The chasm is between the early adopters of the product (the technology enthusiasts and visionaries) and the early majority (the pragmatists).

Agile has crossed the chasm and is now part of mainstream software development. Mary Poppendieck1 used Geoffrey Moore’s classic product adoption diagram to illustrate where agile practices are in the general software development adoption curve.1: http://www.poppendieck.com/

Accepted Agile Methodologies

• Multiple development methodologies:– Agile Unified Process – AUP (simplified RUP)– eXtreme Programming – XP

– Scrum – Very shorttime frame

• Methodologies not accepted as “Agile”– Waterfall– Incremental– Spiral– Evolutionary (Deming)

[1]

What Northrop Grumman normally uses

Software Is Not Manufactured

Government sees software development as a manufacturing process

Treats software as a "mature technology"

DoD Views Software As A Manufactured Item

Advancing Producibility for Software-Intensive Systems-- Grady H. Campbell, Jr., SEI

(12/2008)

What Engineering Has in Common With Manufacturing and Why It Matters

-- Dr. Alistair Cockburn (4/2007)

Production Planning for a Software Product Line-- Gary J. Chastek, Ph.D., SEI Linda M. Northrop, SEI John D. McGregor, Ph.D., Clemson (1/2009)

DoD Instruction DODI 5000.2 views software as a "part" like hardware

5 Reasons Why the ManufacturingMetaphor Doesn’t Work

1. Software development is much less predictable than manufacturing a product

2. Software is not mass-produced on the same scale as a physical product

3. Not all software faults result in failures

4. Software doesn’t wear out

5. Software is not bound by physics"The creation of software is an intellectual human endeavor. Creating good software relies on the personalities and the intellects of the members of the teams that create it. When applied to a different team of developers a process that delivers great software for one team of developers may fail to deliver anything at all for another team."

-- The Practical Guide to S/W Arch.

What Software is Not

"These manufacturing focused models tell us nothing about the typical back-and-forth activities that lead to creating a final product. In particular, creation usually involves trying a little of this-or-that, developing and evaluating prototypes, assessing the feasibility of requirements, contrasting several design, learning from failure, and eventually settling on a satisfactory solution to the problem at hand."

-- Shari Lawrence Pfleeger, Joanne M. Atlee (7/2005)

"Software development is not a manufacturing process ... it is largely an intellectual and sociological activity"

-- Martyn Ould (10/1999)

Very Significant Paradigm Shift

Chaos is Always Present – Get Used To It

In all development chaos and complexities arise preventing wanted functionality

Map of Complexity Science

http://en.wikipedia.org/wiki/Sociology_and_complexity_science

Development is Chaotic

• Development, by its very nature, has historically been chaotic

• System Engineering treats this chaotic phase as if it were a production phase which leads to overruns as shown by the historical record– This is a root problem with “agile” methods– This has been noted as Fr-”agile” in the literature

• Agile Software Development is done using a defined process framework– Some Agile Methodologies, like Scrum, have been very

prescriptive and lay down rigid practices as “rules of the game”

Complexity Interrelationships1

Scale-Free Networks

Autopoiesis (self-organized

criticality, self-persistence)

Complex Adaptive Systems

form via growth and preferential attachment

Lead toSynchronization• Physical and animate• Phase transitions

Chaos(“Butterfly

effect”)

ComplexityEdge of Chaos = Life

Small Worlds • Small avg # steps• High local clustering• Hubs• Resistance to failure• Security weak points• Tipping points Indicate

Power Law Distributions

FractalsStrange AttractorsSmall changes Big Effects Movement from Order through

Oscillation to chaosEntropy as a measure

Chaos & Complexities

Regardless of method, chaos and complexities arise in many forms:

• Changing Staff

• Changing Budget

• Requirements creep

• Product morphing (outside of budget)

• Intended – Eventual end user directs or wants the change

• Unintended – Difference in vision between development and end user

• Customer satisfaction by rapid, continuous delivery of useful software

• Working software is delivered frequently (weeks rather than months)

• Working software is the principal measure of progress

• Even late changes in requirements are welcomed

• Close, daily cooperation between business people and developers • Face-to-face conversation is the best form of communication

• Projects are built around motivated individuals, who should be trusted

• Continuous attention to technical excellence and good design

• Simplicity

• Self-organizing teams

• Regular adaptation to changing circumstances

Agile Manifesto – Some Chaos Applied

Chaos Factor #1EVMSChaos Factor #2

Slow Contract Changes

Chaos Factor #3Culture Gulf & “Arms Length”

Relationships [1]

[1] Short-term benefits of concurrent engineering

Agile Limitations… or not [1]

• Large scale development efforts (>20 developers), though scaling strategies [2] and evidence to the contrary [3] have been described

• Distributed development efforts (non-co-located teams). Strategies have been described in Bridging the Distance [4] and Using an Agile Software Process with Offshore Development [5]

• Mission- and life-critical efforts

• Chain of Command company cultures force Agile teams to fail

• Development teams need extremely high levels of creativity, innovation, commitment and discipline and cannot bloom in a strictly defined process environment due to the high degree of interaction, informal face to face communication with stakeholders, ability to change directions midway, and learning along the way what is needed to build the best possible solution

Cynefin + Agile

• Cynefin (Cun-ev-in): Welsh

– Has no direct equivalent in English. It is translated as familiar habitat

– Thus, in general, if a community is not physically, temporally and spiritually rooted, it is alienated from its environment and will focus on survival rather than creativity and collaboration

• The use of methods such as eXtreme Programming and Scrum has helped many software projects to success, although it’s not always clear why and how it is accomplished

Cynefin Framework Use

• Using the Cynefin framework to move between domains requires a shift to a different model of understanding and interpretation as well as a different leadership style.

• Understanding differences among the different movements in the framework increases the sophistication of the response of a decision-making group to rapid change.

• This understanding allows Agile development teams to anticipate the pattern of social complexity, creating change or chaos [1]

Four named domains and a fifth in grey that is “disorder”

Chaos Theory and Cynefin

• A system is classified as chaotic if it has the following properties:[1]

– It must be sensitive to initial conditions, – It must be topologically mixing, and – its periodic orbits must be dense

This framework originated in the practice of knowledge management as a means of distinguishing between formal and informal communities, and as a means of talking about the interaction of bothwith structured processes and uncertain conditions.

The Cynefin [2] framework:

Success is Possible with Simple Steps

• Agile development

• A framework for traversingorganizational social complexity

• Efficient processes

Equals delivering wanted functionality

Agile + Chaos

• An understanding of the roots of agility, though, is a requirement for adopting and adapting agile processes in organizations

• New research in social complexity theory provides an explanation for when and how agile methods work

• Using techniques from this relatively new field, developers and managers can more easily adapt agile methods to the idiosyncrasies of specific organizations and projects, are more aware of weak signals that can lead to problems, and can design interventions to correct them [1]

The Human Element in Agile

• The Cynefin framework is based on three ontological states juxtapositioned– Order (Known & Knowable)– Complexity– Chaos

• Success is not getting lost on your way from CONOPS to product instantiation– Relaxing the assumption of order and follow the pattern– Relaxing the assumption of rational choice from rigid formalism

and use context and perspective– Relaxing the assumption of intentional actions allowing for

accident of interaction

Navigation on the Cynefin Framework

• Movement at the known-chaos boundary – (1)Asymmetric collapse – disastrous

movement from known to chaotic– (2) Imposition – forceful movement

from chaotic to known (Draconian)• Movement at the known-knowable

boundary– (3) Incremental improvement –

engine of tech growth, but can be pathological

• Movement at the knowable-complex boundary– (4) Exploration – selective

movement to complexity with trust– (5) Just-in-time transfer – choice of

stable patterns in complex space• Movement at the complex-chaotic

boundary– (6) Swarming – mass movement to

an attractor, some may not survive– (7) Divergence-convergence – Able

to handle disruption

Navigation on the Cynefin Framework Cont.

• Visiting chaos– (8) Entrainment

breaking – Create and validate new sources “breaking out of the groove”

– (9) Liberation – casting idea “seeds” for growth and quick exploitation of “seeds” that grow

– (10) Immunization – Inure people to the devastation of chaos to be better prepared for it in the future

Assuring What Is Wanted By The Customer Is Delivered

• Know that software development is highly subject to– Chaos and Complexity– Changing requirements or "desire-ments"– Human social structure changes

• Assure delivering wanted functionality by use of Agile development as– Agile development is the natural form of human grouping– Agile has demonstrated world-wide acceptance and proof-of-performance

• Scrap the "manufacturing" mentality– Understanding that the "manufacturing" mentality in procuring software is a

failure and that software is an intellectual and sociological activity allows better delivery of software product

– Understanding that what appears to be a process out of control is, in fact, a process in control if complexity and chaos patterns are known

• Embrace the fact that social structures are dynamic and using the Cynefin framework allows informed navigation between styles for successful creation of software products

Schools of Thought for Chaos/Complexity

According to De Wolf and Holvoet (2005), there are actually four central schools of research that influence the way complex behavior of complex systems is studied:

• Complex adaptive systems theory (Santa Fe Institute (SFI) 2006): Complex systems are seen as having similarities that can be studied and exploited in order to ease finding underlying principles of a unified complexity theory (Holland, 1996). The SFI often call complex systems complex adaptive systems (CAS)

• Nonlinear dynamical systems theory and chaos theory: Promulgates the central concept of attractors

• Synergetics: Study of emergence in complex systems with the idea of an order parameter that influences which macro-level coherent phenomena a system exhibits (Haken, 1981)

• Far-from-equilibrium thermodynamics: Based on nonlinear dynamics and far-from-equilibrium thermodynamics. Disciplines include catastrophe theory, chaos theory, and complexity theory. These theories push beyond some of the limitations of classical physics, and explore classes of phenomena outside the traditional linear realm

Hope exists

Truth and Confidence: Some of the Realities of Software Project Estimation

-- Phillip G. Armour (4/2008)