we know less than you think (but we do know something)

We Know Less Than You Think

(But We Do Know Something)

Copyright © 2009 Gregory V. Wilson.This presentation may be freely used and distributed with attribution.

Prof. Greg WilsonDept. of Computer Science

University of Torontogvwilson@third-bit.com

Computer “Science” (?)

http://www.third-bit.com 00001

Some of computer science is applied mathematics

But what kind of “science” is software engineering?– Most of it isn’t (yet?) as rigorous as

bridge design– Practitioners would object to being

called alchemists, organizational psychologists, or cheerleaders

Show Me

Shaw 1990: “Prospects for a Discipline of Software Engineering”– Called for evidence-based software

engineering– Built on empirical work since the 1970s

by Boehm, Basili, Glass, and othersEmpirical studies now expected by

top journals and conferences

http://www.third-bit.com 00010

There’s a Lot of Folklore…“The best programmers are up to 28

times more productive than the worst.” (or 40, or 100, or…)– Trace it back: [SAC68] had 12 (self-

taught) subjects[BOE75]: best to worst is a factor of 5

– Borne out by other studies– Consistent with other fields

http://www.third-bit.com 00011

Non-Linearity

[WOO79]: 25% increase in problem complexity 100% increase in solution complexity– Interactions grow as N2

This can work in your favor– Cut 1/4 of your features, problem

because 1/2 as hard to solve

http://www.third-bit.com 00100

Glass’s Law

Any new tool or technique initially makes you less productive– Which is why so many senior students

program with Notepad[GLA99]: 5-35% improvement is

the best you can expect from changing tools– Ah, but techniques…

http://www.third-bit.com 00101

Project Failure

#1 cause: poor estimation– Short iterations cure this in the small,

but don’t help in the large#2 cause: unstable requirements

– But research code often doesn’t have “requirements”

See [VGE91], [COL95], [HOF01]

http://www.third-bit.com 00110

Boehm’s Curve

http://www.third-bit.com 00111

[BOE01] and many others

Re-Use

“It takes three times longer to build a reusable component than it does to build something used only once.”

“If more than 25% of a component has to be revised, it’s better to rewrite it from scratch.”

See e.g. [THO97]http://www.third-bit.com 01000

Lines of Code

“Productivity and reliability depend on the length of a program’s text, independent of language level.”

See [PRE00] for a recent experiment

However, machine-independent code has machine-independent performance

http://www.third-bit.com 01001

“Maintenance makes up 40-80% of the total cost of a software project.”– [BOE75] and many more since

But roughly 60% of “maintenance” is enhancement

And 30% of maintenance time is spent figuring out things work

http://www.third-bit.com 01010

Maintenance

Perceived Importance versus Understanding of Standard Software Engineering Concepts

52.049.7

79.3

45.4 46.6

59.4

46.1

30.0

46.444.1

81.6

52.4

60.2

45.0

31.1

22.9

5.6 5.6

-2.3-7.0

-13.6

14.4 15.0

7.1

-20

0

20

40

60

80

100

sw

requirements

sw design

sw constructionsw verification

sw testing

sw maintenance

sw product

mgmt.

sw project

mgmt.

Software Engineering Concept

Percent

good/expert understanding important/very important gap

Lack of documentation was the #1 complaint for over 40% of respondents, and in the top three for almost 80%

…but ranked second to last in list of where respondents spend their time

http://www.third-bit.com 01010

What Hurts?

Emphasis on grids and GPUs is inappropriate and unproductive– Improving skills will yield greater

dividends on shorter timescalesswc.scipy.org for current course

– software-carpentry.org for the next one

– We’ll be webcasting July 13-31

http://www.third-bit.com 01011

Thoughts

http://www.third-bit.com 01100

References

[BOE75] Boehm: “The High Cost of Software”[COL95] Cole: “Runaway Projects—Causes and Effects”[GLA99] Glass: “The Realities of Software Technology Payoffs”[HOF01] Hofmann and Lehner: “Requirements Engineering as a Success

Factor in Software Projects”[PRE00] Prechelt: “An Empirical Comparison of Seven Programming

Languages”[SAC68] Sackman et al: “Exploratory Experimental Studies Comparing

Online and Offline Programming Performances”[THO97] Thomas et al: “An Analysis of Errors in a Reuse-Oriented

Development Environment”[VGE91] Van Genuchten: “Why Is Software Late?”[WOO79] Woodfield: “An Experiment on Unit Increase in Problem

Complexity”

Recommended Reading

Doar Practical Development EnvironmentsFeathers Working Effectively with Legacy CodeFogel Producing Open Source SoftwareFord The Productive ProgrammerGlass Facts and Fallacies of Software EngineeringJohnson GUI BloopersKernighan almost anythingKniberg Scrum and XP From the TrenchesMak The Martian PrinciplesNygard Release It!Spinellis Code Reading and Code QualityWeiner Time, Love, Memory

01101http://www.third-bit.com