10/13/2007 2007 SOSP Women's Workshop 1

Beyond the OS textbook - getting started in OS research

Kim KeetonHP Labskkeeton@hpl.hp.com

Dilma Da SilvaIBM Researchdilmasilva@us.ibm.com

http://www.hpl.hp.com/personal/Kimberly_Keeton http://www.research.ibm.com/people/d/dilma/

10/13/2007 2007 SOSP Women's Workshop 2

Getting started - a steep uphill climb?

TextbookLessons

CompletedResearch

Project

10/13/2007 2007 SOSP Women's Workshop 3

A little bit about Kim Education

PhD, Computer Science, UC Berkeley (1999) MS, Computer Science, UC Berkeley (1994) BS, Computer Engineering and Engineering and

Public Policy, Carnegie Mellon (1991) Career

HP Labs, 1999 - present Research interests

Storage systems, storage management, dependability, information management

10/13/2007 2007 SOSP Women's Workshop 4

A little more about Kim

10/13/2007 2007 SOSP Women's Workshop 5

A little bit about Dilma Education

PhD, Computer Science, Georgia Tech (1997) MS, Computer Science, Univ Sao Paulo, Brazil (1990) BS, Computer Science, Univ of Sao Paulo, Brazil (1986)

Career IBM Research, 2000-present Assistant Professor, University of Sao Paulo, 1996-2000

Research interests Operating systems, distributed systems

Parallel computing, software engineering, mobile computing

10/13/2007 2007 SOSP Women's Workshop 6

A little more about Dilma

Musical activities: embracing lack of “natural talent” Passions: books , knitting

Navigating politics: large extended family

10/13/2007 2007 SOSP Women's Workshop 7

Outline

Getting prepared Picking the “right” problem Finding a creative solution Quantitatively evaluating your solution Communicating your results The lessons along the way

10/13/2007 2007 SOSP Women's Workshop 8

Getting prepared: system building Sharpening your system-building skills

Find teams and partners that are right for you Learning by hand-on work, learning by example

(Knuth’s literary programming) Develop aesthetic for elegant, simple solutions;

fight temptation (yours, others) to build Taj Mahals! Your arsenal:

Tool chains, scripting, debugging, performance evaluation, tracing

Model checking Simulation

10/13/2007 2007 SOSP Women's Workshop 9

Getting prepared: Map the territory Read papers to learn … and to reflect on

the approach taken Identify the good, the bad, and the ugly Reading groups can be useful

Be curious … with focus Challenge your pride & prejudice

tendencies Common ineffective behavior:

Blindness to whole bodies of work Drowning on papers

10/13/2007 2007 SOSP Women's Workshop 10

Picking the “right” problem There are lots of right problems, not just one! Many ways to find a great problem to work

on: Understand technology trends and changing

usage patterns Run experiments to understand real problems Attend talks, including interesting ones in other

areas Look for open questions in future work section Carve out niche in larger group project Do an internship to understand “real world”

problems Solve a problem that impacts you personally Stumble across it while solving another problem

10/13/2007 2007 SOSP Women's Workshop 11

Picking the “right” problem

Do “real stuff”: solve problems that someone cares about Exposure to industrial-strength real-world problems

Pick a small problem and let it grow Course projects are a great starting point

Easier to make contributions if you arrive early to a field Part of the contribution is framing the research question “Good research is done with a shovel, not with tweezers.” --

Roger Needham Things to remember:

It doesn’t need to be ACM-Dissertation-Award perfect You’re not going to work on it for your entire career Make sure you are excited enough to work on it for several years

10/13/2007 2007 SOSP Women's Workshop 12

Finding a creative solution Keep things simple unless there’s a

good reason not to Pick innovation points carefully; be

compatible everywhere else Best results are obvious in retrospect

Borrow techniques from other fields Machine learning Data mining Optimization Control theory User interfaces

10/13/2007 2007 SOSP Women's Workshop 13

Quantitatively evaluating your solution Computer systems ideas must

be quantitatively evaluated Apply the scientific method Use experiments to answer

questions Use standard benchmarks Use measurements of real

systems Understand your results

10/13/2007 2007 SOSP Women's Workshop 14

Apply the scientific method Goal: (dis)prove hypothesis

Not to measure what’s easy to instrument Think of it as minimizing the effort to prove your point

Useful exercise #1: back-of-the-envelope calculations to see potential of an idea

Useful exercise #2: draw graph you expect to see If experimental results differ:

Flaw in intuition? Flaw in system? Flaw in measurement? Uncovered interesting behavior?

Use intuition to ask questions, not to answer them

10/13/2007 2007 SOSP Women's Workshop 15

Use experiments to answer questions Microbenchmarks

Focus on a particular aspect of system Easy to reason about

Macrobenchmarks Do not focus on particular aspect of system More realistic evaluations of idea

10/13/2007 2007 SOSP Women's Workshop 16

Use standard benchmarks “For better or worse, benchmarks shape a field.” – Dave

Patterson Benefits of standard workloads:

Workload is well-defined, agreed upon by community Provides comparability with other systems May provide load generation tools

Transaction Processing Performance Council (TPC) (http://www.tpc.org)

Standard Performance Evaluation Corporation (SPEC) (http://www.spec.org)

10/13/2007 2007 SOSP Women's Workshop 17

Use measurements of real systems Traces of actual system behavior will ground your work in reality

Storage Networking Industry Association (SNIA) IOTTA Repository (http://iotta.snia.org/repository/resources/) System call traces: Coda, Drew Roselli's traces and others Network file system traces: Berkeley Auspex, Harvard NFS Block-level storage traces: HP Labs cello

Computer Failure Data Repository (CFDR) (http://cfdr.usenix.org/) Node outages in more than 20 large HPC clusters

Other options: Work with industrial partners Measure your local systems

10/13/2007 2007 SOSP Women's Workshop 18

Understand your results Use knowledge of statistics to evaluate results

Example: perform multiple iterations of experiment and calculate confidence intervals

Give yourself enough time to collect and understand the data Things always take longer than you think You won’t always get the measurements right the first time You may decide you need measure something new to make

the point

10/13/2007 2007 SOSP Women's Workshop 19

Communicating your results

Report in sufficient detail for others to reproduce results

Deadlines: friend or foe?Ask colleagues for feedback

Early drafts of papers Practice talks

Benefits You’ll get useful feedback People will be more aware of what you’re doing

10/13/2007 2007 SOSP Women's Workshop 20

Learning from mistakes Innovation involves risk

“Would you like the formula for success?" TJ Watson asked. "Double your rate of failure."

Being smart: early failure detection Inventory of “failures” and “successes”

We learn from both

10/13/2007 2007 SOSP Women's Workshop 21

It’s a learning process We can make the journey smoother:

Improve system building skills Learn from past and current efforts in the research community Learn about trends and problems in the field Adhere to scientific methods Practice careful evaluation Communicate

Summary

TextbookLessons

CompletedResearch

Project

10/13/2007 2007 SOSP Women's Workshop 22

More importantly …

HAVE FUN!

10/13/2007 2007 SOSP Women's Workshop 23

Acknowledgments

Eric Anderson Carla Ellis Manish Gupta Jim Gray Arif Merchant Jeff Mogul Brad Morrey Dave Patterson

Sharon Perl Anna Povzner Craig Soules Alistair Veitch Janet Wiener John Wilkes Jay Wylie