the sins of it projects and why they fail (sometimes)

University of York March 2010 © 2010 IBM Corporation

The sins of IT projects and why they fail (sometimes)

Maurice Perks ….. IBM Fellow ([email protected])

University of York March 2010

University of York March 2010 © 2010 IBM Corporation2

This lecture is based upon the 2009 BCS Lovelace Lecture given by Maurice Perks

Disclaimer

Acclaimer

This lecture is dedicated to Dr Tony Storey, IBM Fellow, for his contributions to software design and excellence.



The landscape is IT projects for large and medium enterprisesPrivate and Public

Our industry has more project failures than it should have

Function on time and to budget

If we can reduce these failures we can improve the efficiencies of many core businesses and our society which are highly dependent upon IT

systems

If we can identify the risks early and manage them then we can make a real difference

Our IT industry has few if any physical laws to guide us but we do have …..

Patterns that we can recognise

Mistakes that we can avoid



The IT background to large IT projects

Often they are big simply because the business they are to support is big in terms of end-users, locations, functions,

connections

The technologies that we use and deploy change at a fast rate. Few other industries see this rate of change

We have a great diversity of skills that range from business experts to chip experts and therefore a wide range of

languages with which we communicate

Large IT projects are very costly so therefore very visible especially when they don’t deliver on time



What are some of the key technical aspects that we should understand and monitor?

1. The Challenge of Scale

2. Middleware or Fiddleware?

3. Method or Madness?

4. Where are we?

5. Software development is still an art

6. Integration, the final frontier

7. Will it really work?

8. Complexity

9. Change our greatest Challenge



1. The Challenge of Scale

Small systems often don’t scale to large systems

Prototypes are great but you can’t run a bank or a country on a laptop (yet)

Our industry has a wonderful and fundamental way of scaling though

apply ParallelismFor chips (multicore), servers, connectivity, application tasks and threads,

databases we can say: if one instance won’t cope then multiple instances will.

But there is a hidden pitfall

The dreaded Serialisation Point

Sometimes the very nature of the problem that we are dealing with means that a bottleneck is inherent in the solution. We need to understand these bottlenecks and design around them.

Sin #1 We cannot scale because we don’t recognise where the bottlenecks in the system are.



2. Middleware or Fiddleware?

Middleware is that ‘software stuff’ that sits in between the application code (business processes) and the control programme (operating system)

Our industry has a very rich set of standard middleware that is crafted by specialised suppliers some of it open

Enterprise IT projects should not develop middleware. It can become a major inhibitor to real progress

It starts out as a good idea but soon becomes a millstone around the development schedules and subsequent upgrades.

Often a few very skilled people start it and then disappear

Sin #2 If it’s not real application code it’s middleware and this is a foolish way to proceed unless you really know what you are doing



3. Method or Madness?

When we develop and deploy a major IT system we are engineering something that is very complex. You must have a methodical way of managing the complexity of the development.

Our industry does not have a standard engineering method by which we can all work with and to for the technical aspects of projects.

There are numerous ‘proprietary methods’. They are all similar at the highest levels …

Requirements – Design – Build - Deploy

But there are often contradictions in what many of the fundamental stage deliverables are e.g.

Architecture? Macro Design? Micro Design? System Integration Test?

Sin #3a You attempt a large IT project with no proven method

Sin #3b You attempt a large IT project with more than one proven method



4. Where are we?

Large IT projects are often long and have several stages.

It’s hard to know where you are sometimes.

SatNavs don’t help

So what’s the problem?



Le plus grand pont au monde

2460 mètres de long

343 mètres de haut



We do not work in a physical world. None of our senses are useful.

We do not work with physical ‘things’ like a bridge. Much of what we work with is virtual and can’t be seen e.g.

o How components interface

o The accuracy of data

o The logic inside applications

o The execution of an IT process

o The real status of a project

o The complexity of what we are working with and on

The following slide is a picture of an IT project. This is what it looks, smells, sounds and feels like. There is no mistake on this slide!



Sin #4: Assuming that you know where you are in a large IT project without really checking



5. Software development is still an art

Most large projects develop some custom software

The software gets designed and then coded

The software then gets tested in units and then multiple units

Errors are then identified within the units and between the units

The cause of these errors has remained the same through many changes in the software technologies and techniques like assembler, cobol, c, c++, java, structured, object, etc …

Lack of real engineering discipline and the use low level languages

Sin #5 Assuming that anyone can write defect free software and that testing will be a mere formality.



6. Integration the final frontier (but 1)

When IT solutions are big we break them down into parts, components, building blocks, etc..

This is good practice for the development of these parts; often in parallel

But in the end they have to be brought together with themselves and any external parts that are used in the whole system. It’s this integration task that is getting more and more challenging.

Perfect interfaces are the key to success. Perks’ 1st Law of IT Integration

IT components that are not designed to fit together and form a working system only do fit together by chance.

Sin #6 Assuming that breaking an IT solution down into a manageable number of parts for development is success. Success is only achieved when they all work together

If you think planning is hard try execution!



7. Will it work in practice?

There are four main aspects of what has to work

1. The technology of the system

2. The people who run and manage the technology of the system

3. The end users of the business who own the system

4. The end users outside the business of the system

If any of the above do not work correctly then the whole system does not work

Sin #7 Forgetting that the whole system has to work, all the technology and all the people involved have to be able to work with it



8. Complexity – the growing enemy

Many operational IT systems and infrastructures that support businesses and environments are now mature

They are the summation ofHundreds of development projects

Millions of changes

Aging

We try to understand this complexity through flat diagrams that give us static views that show components and connections and static flows.

What we see is STATIC COMPLEXITY

But many businesses and environments and the supporting IT systems are highly dynamic with changes happening to a times that vary from microseconds (say workload) to daily, weekly, monthly, etc modifications

What we experience are the effects of DYNAMIC COMPLEXITYSuddenly we hit bottlenecks and big queues

Suddenly a hole in the security system appears

Suddenly the person who knows how much of the system really works – leaves

We experience unexpected shocks and there can be global consequences

We badly need new ways of describing, modelling and predicting the effects of Dynamic Complexity on business or environments and the IT systems that support them. There is a need for a new union of skills

There are three components to this union of skills:1. Business or environmental knowledge

2. IT knowledge and how to use joined-up computing power

3. Mathematics

Sin #8 Not understanding how complexity can change and how we need to predict this change better



9. Change the final frontier!

Our industry is the most exciting industry many of us can imagine. It gains this excitement because of constant and rapid change.

There are two main sources of the changes that get us so excited:

1. The changes in the businesses that IT supports. These are coming at us faster and faster as the business world changes on a global basis. These changes will continue as we strive to evolve to a smarter planet.

2. The technologies that we use which grow in power and capability, variety and where and how they can be used.

Large IT projects are subjected to many change pressures as they progress. The natural reaction is to resist these changes to maintain stability of the solution design and the plan.

But this stance is often self defeating. IT projects have to learn to live better with change. Why?




A questionable and very debatable hypothesis

The more we can accommodate change properly within a large IT project, rather than ignore it or force it back, the greater the chances of real success for that project will be

Our basic problem is that we often do not understand change and its effects

Who does or did?




It is not the strongest of the species that survives, nor the most intelligent, but the one most responsive to change …. C. Darwin

Patterns



Sir Isaac Newton (1642 – 1727)



Newton’s First LawEvery body continues in its state of rest, or of uniform motion in a straight line, unless it is compelled to change that state by forces impressed upon it.

Newton’s Second LawWhen an external force acts on a body of constant mass then the acceleration produced is directly proportional to the force.

Newton’s Third LawFor every action there is an equal and opposite reaction



Newton’s First LawEvery body continues in its state of rest, or of uniform motion in a straight line, unless it is compelled to change that state by forces impressed upon it.

An IT project proceeds to plan until something changes it

Newton’s Second LawWhen an external force acts on a body of constant mass then the acceleration produced is directly proportional to the force.

When a change is forced upon an IT project the extent of deviation from the plan is directly proportional to the magnitude of the change

Newton’s Third LawFor every action there is an equal and opposite reaction

For every change applied to an IT project there is an equal effect on the project’s chances of success unless we really understand the change

Sin #9 Not listening to Darwin and Newton and their likes and recognising how change comes in predictable patterns



What can we conclude?

Our IT industry is the most exciting industry many of us can imagine. It gains this excitement because of constant and rapid change and the associated challenges.

But our excitement often overrules our common sense and experiences and we induce unknown risk.

We do not have any ‘natural laws’ that are our bedrock but we do have recognisable patterns especially bad patterns.

Nevertheless we are great at enabling real business change through the deployment of Information Technology.

We just need to get better.

Sin #10 Overrunning your presentation time .. Thank you

the sins of it projects and why they fail (sometimes)

Documents