system success and failure

Success and failure in sociotechnical systems, 2013 Slide 1

Success and failure

Ian Sommerville


Non-determinism• A deterministic system is one where a given

sequence of inputs will always produce the same sequence of outputs.

• Software systems are deterministic;

• Systems that include humans are non-deterministic


• A socio-technical system will not always produce the same sequence of outputs from the same input sequence


• Human operational behaviour– People do not always behave in the

same way

– Their actions are determined by a range of internal and external factors


• System changes– System behaviour is unpredictable

because of frequent changes to hardware, software and data in seperately managed systems or components


Subjective behaviour

• Whether or not a system has is effective in ‘doing its job’ depends on the observer of a system


• Complex socio-technical systems serve different stakeholder groups, which may have conflicting objectives

• Behaviour that is effective for one group, may be ineffective for another


Success criteria• Complex systems are developed to address

‘wicked problems’ – problems where there cannot be a complete specification.

• Different stakeholders see the problem in different ways and each has a partial understanding of the issues affecting the system.


• Success is a judgment and cannot be objectively measured.

• Success is judged using the effectiveness of the system when deployed rather than judged against the original reasons for procurement.


Conflicting views of success

• A medical information system may be designed to support multiple, conflicting goals– Improve quality of care.

– Provide better information and care costs and so increase revenue.


• Fundamental conflict– To satisfy reporting goal, doctors and nurses had to

provide additional information over and above that required for clinical purposes.

– They had less time to interact with patients, so quality of care reduced. System was not a success.

• However, managers had better reports– System was a success from a managerial

perspective.


What is failure?• A failure is ‘a deviation from a

specification’.

• An oracle can examine a specification, observe a system’s behaviour and detect failures.

• Failure is an absolute - the system has either failed or it hasn’t


A hospital system• A hospital information system is designed to

maintain information about available beds for incoming patients and to provide information about the number of beds to the admissions unit.

• It is assumed that the hospital has a number of empty beds and this changes over time. The variable B reflects the number of empty beds known to the system.


• Sometimes the system reports that the number of empty beds is the actual number available; sometimes the system reports that fewer than the actual number are available .

• When the system reports that an incorrect number of beds are available, is this a failure?


• The percentage of system users who considered the system’s incorrect reporting of the number of available beds to be a failure was 0%.


Bed management system• Mostly, the number did not matter so long as it

was greater than 1. What mattered was whether or not patients could be admitted to the hospital.

• When the hospital was very busy (available beds = 0), then people understood that it was practically impossible for the system to be accurate.


Failure is a judgement• Specifications are a gross

simplification of reality for complex systems.

• Users don’t read and don’t care about specifications


• Whether or not system behaviour should be considered to be a failure, depends on the observer’s judgment

• This judgment depends on:– The observer’s expectations

– The observer’s knowledge and experience

– The observer’s role

– The observer’s context or situation

– The observer’s authority


Failures are inevitable• Technical reasons

– When systems are composed of opaque and uncontrolled components, the behaviour of these components cannot be completely understood

– Failures often can be considered to be failures in data rather than failures in behaviour


• Socio-technical reasons– Changing contexts of use mean that the

judgement on what constitutes a failure changes as the effectiveness of the system in supporting work changes

– Different stakeholders will interpret the same behaviour in different ways because of different interpretations of ‘the problem’


Conflict inevitability• It is usually impossible to establish a set

of requirements where all stakeholder conflicts are resolved

• Therefore, successful operation of a system for one set of stakeholders will inevitably mean ‘failure’ for another set of stakeholders


Normal failures• ‘Failures’ are not just catastrophic

events but normal, everyday system behaviour that disrupts normal work and that mean that people have to spend more time on a task than necessary


A system failure occurs when a direct or indirect user of a system has to carry out extra work, over and above that normally required to carry out some task, in response to some inappropriate or unexpected system behaviour


Summary• Success and failure are not absolute but

depend on the judgment of the observer

• It is impossible to reconcile all conflicts in a complex STS – therefore some stakeholders may consider system behaviour as ‘failure’

• Failures are normal and inevitable in system operation