kernel development: how things go wrong - linux kongress · kernel development: how things go wrong...

62
Kernel development: How things go wrong (And why you should participate anyway) Jonathan Corbet LWN.net [email protected]

Upload: vodung

Post on 04-Jun-2018

229 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Kernel development:How things go wrong

(And why you should participate anyway)

Jonathan CorbetLWN.net

[email protected]

Page 2: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Kernel development is a success

~5 releases/year> 10,000 changes/release> 1000 developers/release

Linux is showing up in everything

...it works!

Page 3: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Kernel development is a success

~5 releases/year> 10,000 changes/release> 1000 developers/release

Linux is showing up in everything

...it works!

Page 4: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

So why talk about failure?

Page 5: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

High profile failures give the kernel a bad name.

Page 6: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

“A key Linux contributor has admitted the the developer community can be intimidating and hard to break into.”

Page 7: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

“A key Linux contributor has admitted the the developer community can be intimidating and hard to break into.”

(Seen on slashdot - must be true)

Page 8: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Failure can teach us things

Page 9: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

“It's fine to celebrate success, but it is more important to heed the lessons of failure.”

Page 10: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

“It's fine to celebrate success, but it is more important to heed the lessons of failure.”

--Bill Gates

Page 11: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

“A bridge, under its usual conditions of service, behaves simply as a relatively smooth level surface on which vehicles can move. Only when it has been overloaded do we learn the physical properties of the materials from which it is built.” -- Herbert Simon

Page 12: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

One note

The kernel community does not lack for clowns.

Page 13: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

One note

The kernel community does not lack for clowns.

I am not talking about them.

Page 14: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

This talk will be naming names

Every developer I name has my respect!

Page 15: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

“Hey, all my other theories made sense too. They just didn't work. But as Edison said: I didn't fail, I just found three other ways not to fix your bug.” -- Linus Torvalds

Page 16: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Photo: fireflythegreat

Page 17: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Tux3

A next-generation filesystem by Daniel Phillips

2008-07-23 Initial announcement2008-11-25 Booting as root filesystem2009-08-16 Last commit

Page 18: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

“Do NOT fall into the trap of adding more and more stuff to an out-of-tree project. It just makes it harder and harder to get it merged. There are many examples of this.” -- Andrew Morton

Page 19: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Daniel kept adding features...then lost interest

Page 20: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

“Anyway, Andrew Morton was right, we should have merged into mainline as soon as Tux3 was booting as root.” -- Daniel Phillips

Page 21: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Lessons

Out-of-tree code is nearly invisibleFew usersFew contributorsLittle momentum

Page 22: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Photo: Team Traveller

Page 23: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Lessons

Get it into the mainline early!

Page 24: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

em28xx

...a video4linux driver

2005-11-08 Initial driver merge...2008-01-05 Markus Rechberger's final

em28xx patch2008-11-02 Replacement patch rejected2009-08-09 Markus's final kernel patch

Page 25: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

“Companies should be aware that if they try to submit any code to you they will loose the authority over _their_ work.” -- Markus Rechberger

Page 26: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Another example

May, 2004Hans Reiser tries to block the addition of new functionality to reiserfs.

Page 27: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Lessons

Contributing means losing control

Others will improve your code

Page 28: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Photo: Yuliya Libkina

Page 29: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

“The fact is, maintainership does _not_ mean ownership. It means that you should be _responsible_ for the code, and you get credit for it, but if problems happen you do NOT “own” it. Not at all.” -- Linus Torvalds

Page 30: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

2.5.x IDE

2002-02-15 Martin Dalecki's first “IDE cleanup” patch

2002-03-08 IDE18, subsystem takeover2002-08-09 IDE115 merged2002-08-16 Martin quits, all IDE work

reverted

Page 31: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

“Breakage is the price you have to pay for advancements” -- Martin Dalecki

Page 32: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Lessons

Don't break things!

Listen when people complain

Page 33: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Deadline scheduler

Con Kolivas's scheduler rewrite

2007-03-04 First post2007-03-05 Linus amenable to merging2007-03-19 Linus gets irritated2007-04-13 Molnar posts CFS2007-07-10 CFS merged for 2.6.232007-07-25 Con leaves the kernel

community

Page 34: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

“So, I've had enough. I'm out of here forever. I want to leave before I get so disgruntled that I end up using windows.” -- Con Kolivas

Page 35: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Lessons

Improve the kernel for everybody...or at least don't make it worse

Page 36: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Lessons

Some parts of the kernel are hard to change.

Page 37: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Lessons

Participate in the wider discussion-ck list did not help

Page 38: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Lessons

Aim for a solution to the problem...rather than inclusion of specific code

Page 39: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

reiser4

2002-10-29 First code post2003-07-24 2.6.0-test merge request2004-08-19 Added to 2.6.8.1-mm22005-09-11 Push for 2.6.142006-07-20 Push for 2.6.192006-10-11 Hans Reiser arrested

Page 40: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

What were the problems?

Non-POSIX filesystem behaviorNumerous technical difficultiesHard-to-reproduce benchmarksAntagonistic approach to othersMemories of reiser3

Page 41: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Lessons

Linux is not a research system

Page 42: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Lessons

Visionary brilliance will not excuse a poor implementation

Page 43: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Lessons

It's better not to accuse others of conspiring against you

Photo: Rob!

Page 44: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Lessons

The community remembers past actionsDevelopers also think far into the future

Photo: krupp

Page 45: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

SystemTap

2003-11 DTrace debuts2005-10 RHEL4 introduces SystemTap2008-07 FTrace merged2009-06 Perf Events merged2009-09-22 SystemTap 1.0 released???? SystemTap merged

Page 46: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

2008 Kernel Summit

50% had tried to use SystemTap20% succeeded

Page 47: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

“I thought everyone learned the lesson behind SystemTap's failure: when it comes to tooling/instrumentation we don't want to concentrate on the fancy complex setups and abstract requirements drawn up by CIOs as development isn't being done there. Concentrate on our developers today, and provide no-compromises usability to those who contribute stuff.” -- Ingo Molnar

Page 48: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

In other words...

If kernel developers don't see the value...it won't go in.

Page 49: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

TALPA

Posted in August 2008Never merged as such

The goal:Provide hooks for virus scanners

Page 50: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Problems with TALPA

Kernel developers disliked itWhy bother with broken security models?

Badly-expressed requirementsNo threat modelSolutions not needs

Page 51: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

fanotify

Merged in August, 2010 (2.6.36)

Provides hooks for virus scanners

Page 52: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

What changed?

Featured a cleanup of file event notification

Replaced inotify and dnotify

Rephrased requirement:“Enable virus scanners to hook into file operations without using rootkit techniques.”

Page 53: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Lesson

Patches must be sold to developersNot managers or customers

Page 54: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Other examples

Android wakelocksDistributed storageTuxOnIceWireless extensionsCML2msleep()Xenutrace...

Page 55: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Why bother?

Page 56: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

It's not as hard as it seems

Page 57: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Fun!

Fun!

Page 58: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

A slightly elite club

“Well, you don't get to be a kernel hacker simply by looking good in speedos” -- Rusty Russell

Page 59: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Jobs

If you show that you can get code into the kernel, you will get job offers.

Page 60: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Influence

It's how you get the kernel to meet your needs.

Page 61: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

“If we don't succeed we run the risk of failure” -- Dan Quayle

Page 62: Kernel development: How things go wrong - Linux Kongress · Kernel development: How things go wrong ... said: I didn't fail, ... 2007-03-05 Linus amenable to merging

Questions?