system architectureA (hopefully) gentle introduction

Andres KüttFebruary 25, 2016

Riigi Infosüsteemi Amet

introduction

agenda today

∙ Intro. Importance of the topic. Global challenges. Conventionalassumptions

∙ System thinking. Concept of value∙ Form, Function, Concept intro. Examples∙ Identify the entities of the system and their function∙ Exercise: Form, function and concept of a country∙ Relationships among entities. Feedback, emergence∙ Introduction to complexity∙ System thinking and architecture in public sector∙ Role of a software architect

2

andres kütt

∙ Building software for living since 1993∙ In architect-like roles for the past 10 years∙ ≈MSc (UT, Statistika), MBA (EBS), MSc (MIT)∙ Currently architect of EstonianInformation System

∙ Skype, Hansapank, EMTA etc. in the past∙ Some teaching experience

3

structure of today

∙ Two main sections∙ Fundamentals of system architecture∙ Theoretical foundations of public sector architecture

∙ Nine 30-minute blocks of study punctuated by two breaks∙ Each block but one contains∙ 20 minutes of slides,∙ 5 minutes of paired discussion∙ 5 minutes of common discussion

∙ The questions on paired discussion form the basis of grading∙ The 5th block is different consisting of a team exercise followed bydiscussion

4

the contents

∙ The theory is mainly based on Crawley et al. (2015) and variouslectures by prof. Crawley

∙ The goal is to talk about architecture in general not softwarearchitecture in particular

∙ Have patience: it is really hard to that much into 4.5 hours withoutyour heads exploding. Certain things will not make immediatesense

5

fundamentals of system architecture

Architecture is an abstract description of the entities of a systemand the relationships between those entities. It can be seen as aset of decisions.

Crawley et al. (2015)

7

properties of architecture

∙ Architecture can either emerge from a development process or beexplicit∙ Everything has an architecture∙ Explicit architecture provides more (but not ultimate) control

∙ It is about decision quality under the conditions of ambiguity∙ Small improvements here can lead to a lot of benefit down the line

∙ In software, architecture is an outcome of a continuous processnot a one-time activity∙ Design decisions need to be made constantly as the systems evolve

8

benefits of good architecture

What do we generally want from a system?

∙ Meet shareholder needs∙ Deliver value∙ Integrate easily∙ Evolve flexibly∙ Operate simply and reliably

Explicitly developed architecture allows us to have a measure ofcontrol over all this

9

conventional assumptions about organisations

∙ They are culturally, technically etc. homogenous∙ “You and your architecutre can get stuffed, I know better what workslocally” (a random person from Montevideo)

∙ Organisational and legal borders are clearly defined∙ Tightly integrated supply chains∙ Global organisations partnering with each other largely ignoring states

∙ Relative independence from global challenges∙ Many things need to be resolved within next decades. You are eitherpart of the solution or you are part of the problem

∙ Controlled and contained IT∙ Specialised hw/software package operated by highly trained employeesvs. general purpose soft- and hardware all over the place

10

the challenge

As the assumptions no longer hold, we are faced with:

∙ Higher complexity both within the organisation and globally∙ Higher levels of upstream uncertainty∙ No way to separate IT from the rest of the organisation withoutloss of fidelity

We need to figure out a better way to providethe benefits of good architecture

11

This is not about architecture astronautics

What we discuss today is to be applied tomake better stuff every single day

12

discussion

What do you want to get out of today?

13

system thinking basics

A system is a set of entities and their relationships, whosefunctionality is greater than the sum of the individual entities

Crawley et al. (2015)

15

system thinking definition

System thinking is not about thinking systematically

∙ It is about thinking about a question, circumstance, or problemexplicitly as a system∙ Observe, how we make no assumptions about the nature of the entities∙ Emergence of new functions is a key part of the definition. This is whatallows systems to generate value

∙ System thinking is a way of thinking∙ Therefore, it is not a science or a theory∙ It is a mental model∙ All models are wrong but some models are useful (Box, 1976)∙ It should be used alongside with other modes of thinking and logicalreasoning

16

examples of systems

∙ Sand and a funnel combined yield the function of timekeeping∙ Neither the funnel alone nor sand can do that∙ But also, a function of triggering a trap can emerge

∙ A particular group of people combined can yield the function ofwinning football matches∙ In addition to function, performance emerges∙ Performance is an attribute of the emergent function

∙ A set of mechanical and chemical components can be combinedto result in a car∙ A group of “-ilities” - reliability, operability, respectability etc. - emerges∙ Safety is one of these∙ Note that in this model, usability is an emergent property of the system

17

emergence of emergencies

Not all emergency is desirable

∙ In December of 1984, 40 tons of methyl isocyanate was released toatmosphere at a chemical plant in Bhopal, India∙ Worst industrial accident in history∙ 2 000 fatalities, 200 000 injuries, 10 000 permanent disabilities(Leveson, 2011, conservative estimates)

∙ A bona fide systemic failure∙ Unclear job descriptions∙ Inadequately planned safety measures∙ Human error∙ Business error in decreasing safety to save cost

∙ The same principle applies to information securty∙ Security is an emergent behavior of a system∙ Cannot thus be fully predicted, only designed for

18

emergence and value

How exactly do systems generate value?

∙ This is how the work of an architect is justified∙ A well-architected system might generate more value

∙ Benefit stems from∙ The system performing the intended function with intendedperformance

∙ All the ilities being there∙ Lack of emergencies∙ Basically, the emergence happening in a desired way

∙ The concept of benefit implies existence of a user, as benefit isalways relative

∙ Value is defined as benefit at cost∙ All artificially constructed systems incur a cost

19

key questions of systems thinking

These questions should be asked when thinking about a system

1. What is the system made of?∙ Again transcending discipline boundaries

2. What is the particular mental model of the system?∙ Remember the one about all models being wrong?∙ Many are applicable, which one is the most useful at the moment?

3. Where are the system boundaries?∙ Everything is interconnected, where do we draw the line?

4. What is the system context?∙ What interfaces do we need to keep in mind?

20

discussion

What is architecture?

21

answering question 1: what is the system made of?

∙ This is where you define subject of your architecture∙ Do we get hardware involved?∙ How about people? (NB! people are hard)

∙ There is no such thing as a green field, you are always workingwith an existing system∙ At the very least the users are there∙ It is also beneficial to re-visit this question as the system evolves

22

the steps

Steps to answering “what the system is made of?”:

1. Identify form and function2. Identify entities of the system and their form and function3. Identify the relationships4. Predict emergence

23

step 1. identify form and function

∙ Form is what the system is∙ form is stable over a period of time∙ form is the part of the system that is made or assembled

∙ Function is what the system does∙ function is what causes performance and ilities∙ this is why the system exists or is employed∙ emergence occurs in the function domain∙ function consists of a process and an operand: something (operand)changes state as the system does something (process)

24

examples of form and function

∙ An amplifier∙ form is the wires, transistors and resistors∙ functions is amplification (process) of a signal (operand)

∙ Circulatory system∙ form is heart, lungs, capillaries∙ function is transporting (process) oxygen (operand)

∙ Project team∙ form is the team members∙ function is delivering (process) result (operand)

25

form and function. observations

∙ Both can only be expressed via an abstract model∙ Software, as opposed to physical things, cannot be experienced directly∙ It is thus vital the abstract model is as useful and unambiguous aspossible

∙ The same function can be performed by several sets of formelements and vice versa∙ Therefore we need a third notion, the concept, creating a one-to-onemapping

∙ These mappings are not equivalent∙ I am yet to see a useful representation of this relating functional andform models

26

form and function. more observations

∙ It is much more difficult to do for some systems than for others∙ Physical things are much easier than abstract concepts like software∙ Form of an ice sculpture. Is oxygen in the water part of the form?∙ What is the Internet actually made of?

∙ There is a tendency to get carried away with form∙ Where does the system end?∙ Is IETF part of the Internet? What about sysadmins of the core DNSboxes?

∙ In the universe, everything is interconnected

27

discussion

Where does concept of a system come from?

28

the steps

Steps to answering “what the system is made of?”:

1. Identify form and function2. Identify entities of the system and their form and function3. Identify the relationships4. Predict emergence

29

step 2. identify the entities of the system and their function

What is the structure of form and function of our system?

1. Include everything you feel is important2. Throw out everything that turns out not to be3. See what can be generalised and clumped together4. Define boundaries and interfaces5. Repeat unless the result is both useful and manageable

The basic algorithm:

Keep adding and removing things until you like the result

30

on adding and removing things

Think holistically

∙ Add people, technology, hardware, software, everythingimportant-looking

∙ Be aware of unknown-unknowns: things you do not know you donot know.∙ For software folks this is often people∙ For legal folks, this is often anything related to real life

Focus

∙ The 7± 2 rule. There is a limit to human cognition (Miller, 1956)∙ ”Is the entity important in determining the outcome and theemergence that I am interested in?”

31

two key types of clumping

Abstraction Replace instances with class. “Employee” instead ofAlice, Bob and Charlie

Modularisation What things relate to each other more than toothers?∙ Footnote: there are great tools for doing thatalgorithmically. See Browning (2001)

32

guidelines for creating abstractions and modules

∙ Conceal the unimportant and expose the important∙ On system level, exact class structure might not be relevant but theinterface probably is

∙ Make sure appropriate relationships are retained∙ Team members might cooperate a lot even if teams themselves do not

∙ Create abstractions at the right level of decomposition oraggregation∙ “Screw” is a pretty useless abstraction

∙ Simplify, until you notice loss of fidelity, and no further∙ “Software running on hardware being used by a user” is a valid but notuseful model

33

define system boundaries

God, grant me the serenity to accept the things I cannot change,the courage to change the things I can, and the wisdom to knowthe difference

∙ Choose system boundary in a way that is useful∙ By defining system boundaries, you define interfaces: therelationships crossing the system boundary

∙ Cleary differentiate between∙ Things you can change∙ Things that make up a system

∙ These two very seldom overlap∙ Ignore the big picture and get a useless system∙ Try to change the big picture and get a bloody nose

34

stop condition

All steps alter the results of previous steps, so there’s a cycle. Whendo you stop?

∙ Do all entities contribute to the function and emergence you areinterested in?

∙ Is there an element of form contributing to each function?∙ Can you explain the system on a single sheet of paper?∙ Is the result useful?

35

discussion

What if the resulting system is too complex?

36

exercise: form function and concept of a country

∙ Go through first two steps of system analysis for a country of yourchoosing∙ Identify form and function∙ Identify entities of the system and their form and function

∙ Present the results

37

the steps

Steps to answering “what the system is made of?”:

1. Identify form and function2. Identify entities of the system and their form and function3. Identify the relationships4. Predict emergence

38

step 3. relationships among entities

Functional relationships do something, they are dynamic in nature

∙ operands are exchanged or acted upon∙ sometimes called interactions∙ a heart exchanges blood with a lung

Formal relationships are relationships that exist, they are static

∙ these relationships exist stably for some period of time∙ usually a connection or a geometric relationship∙ a class is within a package∙ sometimes called structure

Typically, (but much less in software), a formal relationship isrequired for a functional relationship

39

representing relationships

The same system represented in two ways:

A standard UMLcomponent diagram

A DSM (see Browning(2001) for details)

40

step 4. emergence

Emergence is why we study systems in the first place

∙ Firmly in the functional domain, form leads to function that leadsto emergence∙ It is about how the elements are put together∙ The elements themselves are important but architecture leads toemergence

∙ Cannot be fully predicted or controlled by definition∙ Information/Cyber security and safety are emergent properties ofa system∙ Understanding how emergence works is thus crucial∙ Since emergence stems from architecture, so do security and safety

41

predicting emergence

This is the crux of system thinking

Precedent We have experience and thus know, what shall happenExperiment We can build an experiment to see, what shall happen

Model We can build a model to simulate, what shall happen

∙ Many modern systems are new, too large to do experiments forand too complex to model

∙ These we can reason about using system thinking or otherframeworks

42

discussion

Can we talk about emergence in legal systems?

43

defining complexity

There are many definitions, Mitchell (2009) brings examples.Complexity as …

∙ …size. Function of the number of elements and their relationships∙ …(Shannon) entropy. Information content of the system∙ …algorithmic information content. Compressability∙ …logical or thermodynamic depth. The cost of construction∙ …relationship between input and output. Predictability∙ …fractal dimension

44

more about complexity

∙ Complexity is an emergent property of a system and therefore ageneral concept

∙ The arithmetical definition yields most practical tools∙ Counting things is easy∙ Limited in usefulness because of its static nature

∙ Complexity vs. complicatedness∙ Complicatedness is the extent to which a system appears complex

∙ Static vs. dynamic complexity∙ Dynamic complexity is the ability of a system to exhibit complexbehaviour over time

∙ Static complexity stems from formal relationships, dynamic complexityfrom functional ones

45

exponential nature of complexity

Com

plex

ity

# of elements / time

Limit of abilities

Complexity seems to have a tendency to increase exponentially

46

implications of this

This is why complexity needs to be actively managed. There arethree basic strategies

Flatten the curve Working on the system should add the leastamount of complexity possible. Engineering.

Stop moving Stop adding features, growing the org. etc. This is whateventually happens. Management.

Raise the capabilities Increase the ability of an organisation tocope. Dynamic capabilities.

47

discussion

How to assess complexity of a set of laws?

48

foundations of public sector architecture

know and respect your context

∙ Understand dependencies between layers∙ Know your limits (i.e. define boundariescarefully)

∙ Learn to accept what you cannot change∙ Maintain alignment between layers∙ Build horizontal and vertical relationships∙ Ponder the source and impact of changes

Concept

Functions

Peopleware

Software

Infrastructure

50

on legal issues

∙ In public setting, the top layers a heldtogether by a legal framework, usuallysubject to democratic process

∙ The bottom layers are held together byopen source and the internet

∙ The former is local, the latter is global∙ This creates a barrier to change

OS

S &

In

tern

et

Le

ga

l fr

am

ew

ork Concept

Functions

Peopleware

Software

Infrastructure

51

peculiarities of public sector

∙ There is no profit∙ Taxes are collected to provide public services∙ If these are not spent, the customer is not getting their moneys worth∙ Therefore, there is no intrinsic efficiency pressure

∙ You can’t risk to break the law∙ In private sector, you can accept any risk∙ In public sector, one cannot accept the risk of breaking the law∙ At least this is my understanding as an engineer∙ E.g. you can’t put encrypted personal data of citizens to a public cloud asbreaking the crypto in the future is a real risk that would lead to exposingpersonal data

52

discussion

How to express legal architecture?

53

role of a software architect in government setting

∙ Manage, control and own complexity∙ Prevent lock-in∙ Build bridges∙ Predict emergence

54

manage, control and own complexity

∙ Do not forget complicatedness!∙ Everybody else is interested in more of it∙ Bureaucrats do not mind complex processes seeking to control reality∙ Engineers love to engineer∙ Project managers have deadlines and budgets∙ Entropy tends to increase

∙ Complexity has the property of being complex∙ Once the threshold is breached, backing down might be impossible∙ But it would be considered the job of an architect∙ Thus, constant effort is required to keep system complexity manageable

∙ Complexity needs to be represented at decisions∙ Governments love waterfall but less and less problems are simpleenough to be solved using it

∙ Legal, business and organisational design decisions are good examples

55

prevent lock-in

∙ In private sector, IP governance can prevent lock-in∙ In public sector, there is no concept of IP and thus lock-in canhappen more easily

∙ How lock-in occurs∙ Vendor develop architecture embedding their concept in the process∙ Every new vendor has to learn and adapt to the the mental models ofthe original vendors, which is costly

∙ Architect can actively drive concept development∙ Now all vendors have the same barrier of entry∙ And therefore an incentive to dethrone the architect

56

build bridges

From sufficiently high up, most fights look ridiculous

∙ Systems thinking leads to the need to span disciplines∙ PMs, developers, the customer, end users, accounting etc. all focus ontheir job

∙ Architect is often the only one seeing the big picture

∙ Establish reasonable communication lines between disconnectedteams (Hickey, 2015)

∙ Architect should have very few vested interests thus being theideal middleman

∙ Very few other roles are in a position to argue for global optimums

57

predict emergence

Architect should have a holistic view and thus the singular ability topredict emergence

∙ Opportunities∙ What other functions could a system perform?∙ What new functions can appear with additional effort?∙ Is the emergent function worth the added complexity?∙ Close cooperation with the policy folks a must

∙ Threats∙ What could go wrong?∙ Much more frequent∙ It takes a huge amount of skill not to appear as a buzzkill∙ Close cooperation with cybersec people a must

58

discussion

Given the general nature of architect’s value,do we need architects in other fields?

59

bibliography

George E. P. Box. Science and statistics. Journal of the AmericanStatistical Association, 71(356):791–799, 1976.

Tyson R Browning. Applying the design structure matrix to systemdecomposition and integration problems: a review and newdirections. Engineering Management, IEEE Transactions on, 48(3):292–306, 2001.

Edward Crawley, Bruce Cameron, and Daniel Selva. SystemsArchitecture: Strategy and Product Development for ComplexSystems. Prentice Hall, 2015.

Kevin Hickey. The Role of an Enterprise Architect in a Lean Enterprisehttp://martinfowler.com/articles/ea-in-lean-enterprise.html,November 2015.

Nancy Leveson. Engineering a safer world: Systems thinking appliedto safety. MIT Press, 2011.

61

George A Miller. The magical number seven, plus or minus two: somelimits on our capacity for processing information. Psychologicalreview, 63(2):81, 1956.

Melanie Mitchell. Complexity: A guided tour. Oxford University Press,2009.

62

license

theme

Get the source of this theme and the demo presentation from

http://github.com/matze/mtheme

The theme itself is licensed under a Creative CommonsAttribution-ShareAlike 4.0 International License.

cba

64

contents

The contents of the slides is lidecensed under a Creative CommonsAttribution-NonCommercial-ShareAlike 4.0 International

cbna

65

Questions?

66