why software development projects fail - june 2010

TRENDS IN APPLICATION

DEVELOPMENTJens Riboe

In collaboration with

June 2010

Trends in Application Development 9 May 20102

Who is Jens Riboe?

För lite olika kunder

Engagements at InformatorErlang BasicsThreads programming in C++Groovy and GrailsCloud Computing


Ericsson experience

AXE-N (first half of -90s) Supervisor of several MSc Thesis C++ and OMT (OOAD) course development and

training Erlang (first half of -90s)

Elemtel, SICS, KTH, Erlang Systems IP-BSS (2000-2001)

Mentor and SW architect Erlang training (2009)

Ericsson Linköping Threads Programming using C++

Ericsson Göteborg

Setting the stage


A quarter of a century of software development

Increased complexity More tasks More users More realisations More ways to failure

A technicians perspective


The cost of fixing bugs

The longer you wait, the more it costs

It will never be as cheap to fix the problem, as right now


Two principles of system design

Make it so simple that there are obviously no deficiencies

orMake it so complicated that there are

no obvious deficienciesSir Charles Antony Richard Hoare


A picture of complexity

Too many cross-wise dependencies Too many modules Too many lines of code per class and

method

MethodologyTechnology

Principles of the solution


Agile principles

Focus on delivery of results Maintain development speed Remove obstacles Simplicity Communication

Change is the only constant

Do morewith less


Ask the three questions every day What have you done? What are you planning to do? What prevents you from doing what

you planned?


Remove obstacles

Isolate the team Allow them to concentrate on their task Enforce meeting free days

Use a common task list Assign one person to do the prioritazion

Work in fixed time boxes Define a delivery at the end

Ditch ancient tools If it gets in the way, throw it out


Deliver components, not source code!

Shift the focus from delivery of source code to delivery of tested and documented components

“Hide” the source code View it as property of the responsible team

A component is a boundary for the progress of complexity unit of understandability unit of testability

Component


Fix Broken WindowsApplied sociology to computer science

Story A luxury car was left for a week in a rough area, nothing

happened. Then one of its windows was broken. After a day, all that

was left of the car, was a smoking car wreck. Conlusions

Fix problems immediately Continuous sanitization Prevent the impression of hopelessness

Hopelessness leads to less ambitious work, which leads to more bugs, which leads to irresponsibility, which creates more hopelessness

Tip Email every production log stack-trace to the developers


Avoid Screaming MonkeysApplied sociology to computer science

Story A ladder leading up to a bunch of bananas was placed

in a cage of monkeys. When the monkeys climbed up, the cage was sprinkled with cold water. Nobody climbed up.

A new monkey was let into the cage, and discovered the bananas. When he tried climbing up, he was beating by the others.

Another monkey was let into the cage and was beaten up by all, including the new monkey

Conclusions Don’t just repeat what other are telling you unless you

understand why and agree. Perhaps the root cause doesn’t exist any longer.

Methodology


DRY – Don’t Repeat Yourself Eliminate all information duplicates

This is the opposite of cut&paste programming, which in many cases is a major source of bugs

A modification of a single element should not affect unrelated elements and logically related elements should change in a predictable and uniformed way From ”The Pragmatic Programmer” 1999


Refactoring

Stepwise improvements of code quality using stepwise transformations of the source code ExtractMethod, FactorOutBaseClass,

MoveClass, ... Software development resembles

gardening Allocate time and plan for refactoring

A common problem Initial simple design for few features More features requires more elborated design Required time for refactoring is denied Work-arounds creates hopelessness . . .


Testability and Unit testing

Ensure every class/module/subsystem is testable

Automate all tests Forbid manual tests Create test reports and coverage

reports Testability improves the code quality

substantially Common problems

Sufficient time is not allocated for non-trivial tests

Old tests that fail, after a code change are, disabled instead of fixed


Configuration by Convention Convention over Configuration

(aka Coding by convention) Avoid XML Hell Use CbC framework and tools

Maven, Grails and many more Common problems

"We have been using make for a very long time"

"We already have a guy that takes care of that bunch of recursive Ant scripts, nobody understands"


Continuous Integration

Ensure the system always is Compilable Buildable Executable Testable

How Work in small increments Check-in frequently Avoid branches

Check-in to the main trunk Unless for very specific and time limited tasks

But, it works on my

machine!


Contemporary project tools Use a new Version Control System (VCS)

No file locks and local state management Attach a Web Front-end to the VCS

Source code must be browsable and linkable Use a powerful IDE

The VCS must not block refactoring Use a Wiki for static info

Editable for everyone on the team Use a Blog for dynamic info

Capture the project history as it evolves Use Configuration by Convention tools Use a Continuous Integration server Provide public access to the task list

Burn down chart

New Dynamic LanguagesNew Dynamic FrameworksNew Deployment Platforms

Technology


New programming languages

Shorter time to result Expressive and compact code Dynamic typing, ”duck typing” Groovy, Ruby, Python, Erlang and

many moredef persons = Person.list().findAll {it.location}def json='[' + persons.collect( {it as JSON} ).join(',') + ']'

[ {name:”jens”, email=“[email protected]”}, {name:”wille”,email=“[email protected]”}, {name:”anna”, email=“[email protected]”}]


Pragmatic functional programming

The current trend of software development Will affect the way working in the same

manner as object-orientation did 15-20 years ago

New concepts Closures Continuations Futures Currying

Affects even old languages such as C++ The new standard library C++0x In the past STL, JGL


Cloud Computing

Will affect how we design, test and deploy new applications

Amazon AWS Elastic Compute Cloud (EC2) Elastic Block Storage (EBS) Simple Storage Service (S3) Simple Queue Service (SQS) Simple DB (SDB) MySQL (RDS) . . .

http://www.slideshare.net/startpad/startpad-countdown-8-amazon-web-services-and-you-470919/32http://www.slideshare.net/startpad/startpad-countdown-8-amazon-web-services-and-you-470919/36

4/11/08 4/13/08 4/15/08 4/17/08 4/19/080

500

1000

1500

2000

2500

3000

3500

4000

From 50 to 3500 servers within a week


Easy and cheap to launch one or more servers

Project server Test server Load generator . . .

27

$0.085 / hour

100 servers1 hour

= $8.50


Large data computations

Apache Hadoop Distributed MapReduce Distributed File Systems (HDFS)

Cheap to run 100-1000 nodes, for a limited time period

The Problem


WHY ???

Despite the advances in Ideas Methodology Technology

The quality and relability of software development is still poor and do not show signs of recovery


BECAUSE !!!

There are opposing forces preventing good software to be developed

Forces


The Consultancy Equation

Mean Utilization

Mean Competence

𝑹= 𝝁𝒊𝜿𝒊𝚮≈ 𝚴𝚮𝛍ഥ∙𝛋ഥ𝑵𝒊=𝟏

Hourly Rate

Number ofconsultantsNumber ofconsultants

Trends in Application Development 9 May 201034 𝑆+ ∆ < 𝑅= 𝑁𝐻𝜇∙𝜅

Cost Contribut

ion

The law of gravity for consultancy life

Salary

Salary plus Cost must be

less than the Revenue.However,

Revenue is a linear function of

the number of delivery units (consultants).


Affecting forcesSeller

Buyer

Purchase principlesStandardization

Perception of competence levelType of engagement


Compensational strategies

Consultancy Seller Junior-shoring

Consultant Job-hopper Career switch

Consultancy Buyer Off-shoring

The economic rules are the same in Stockholm and

Bangalore


The food chain of the consultancy market

•Deliver and charge for results

•Runs specific project with defined deliveries

•Uses its own team

Solutions

•Deliver and charge for hours

•High overhead cost for persons on the bench

Staffing

•Deliver and charge for hours

•Brokerage of independent consultants

•No overhead cost

Brokerage

Trend

Price / Hour

Deliver

Results

Deliver

Hours


Effective Business Model

Deliver results

Deliver hours

Sales

Tech

Sales

Tech

Sell more hours!

Create more bugs so we can stay longer

Sell more projects!

Work smarter and faster so we can

stay shorter


The impact of competence development

Deliver results

Deliver hours

Sales

Tech

Sales

Tech

Minor Major


Opposing ambitions

Methodologies, Languages and Tools More bang for the bucks gained Higher abstraction level required Higher involvement required Higher competence level required

Organizations More bang for the bucks gained Standardization and policies required Lower involvement required Ignorance of competence level required


Less is More

A small high competent team outperforms a large low competent team, many times Code and application quality Development time and time top market Total cost is lower

Organizational obstacles Salary and/or hourly rate level matters Policies and procedures matters The head count matters The total cost is larger


Knowledge and Productivity Collective project knowledge

Formal docs Informal/invisible knowledge

Knowledge transfer Depends on invisible knowledge and the skill

level Steals time from development Might take 6-12 months to reach full productivity

The compensational strategies causes A stream of new (junior) persons A continous demand of knowledge transfer

Draining Contributing

time

knowledge


The knowledge transfer dilemma If the staff turnover rate is too high,

the contributing period becomes too short

If the non-contributing (draining) period takes a significant portion of the project time, the project is gradually loosing its collective ”invisible” knowledge

At the end, ”nobody has a clue”

Time

Accumulated (invisible) knowledge

Conclusions


Software Development Crisis Admit there is a problem Identify the opposing forces Turn the forces

so they are contributing to the same goal


Questions?


Thanks

Jens Riboe+46 (0)730-314040

[email protected]