research industry panel review
DESCRIPTION
Summary of ICSE 2011 Panel on "What Industry wants from Research". This is a summary of all the presentations from that panel that I presented in an invited talk at the CSER meeting in Toronto, November, 2011.TRANSCRIPT
Highlights from an ICSE 2011 Panel: What Industry Wants from Research
Panel Organizers: Jorge Aranda, Daniela Damian, Marian Petre, Margaret-‐Anne Storey, Greg Wilson
1
2
A conversaEon is started…
3
QuesEons we asked…
What is your percep0on of soMware engineering research? Conferences, journals, collaboraEons…
What counts as evidence? How do you find out about results?
What topics should soMware engineering research focus on?
4
Percep0ons of so7ware research
Focused on organiza0ons that don’t compare to us
Results may fail to scale
Piecemeal improvements
Irrelevant, dated, ignoring cuWng edge problems
hXp://Enyurl.com/icse2011panel 5
“[I’m afraid] that industrial so9ware engineers will think that I’m now doing academic so9ware engineering and then not listen to me. (…) if I start talking to them and claim that I’m doing so9ware engineering research, a9er they stop laughing, they’re gonna stop listening to me. Because it’s been so long since anything actually relevant to what pracFFoners do has come out of that environment, or at least the percentage of things that are useful that come out of that environment is so small.”
6
What counts as evidence?
Preference for quanEtaEve data and staEsEcal significance
7
What counts as evidence? Preference for quanEtaEve data and staEsEcal
significance
“managers are coin operated in some sense. If you can’t quanFfy it in terms of Fme or in terms of money, it doesn’t make much difference to them. (…) I think there does need to be some noFon of a numeric or at least an objecFve measure.”
8
What counts as evidence? Preference for quanEtaEve data and staEsEcal
significance
AssociaEon of qualitaEve data with 'mere anecdote‘
And yet…. InfluenEal opinions persuasive “I trust the people I hire”
9
Difficulty in applying our findings
Resistance to change for small potenEal gains
“(…) it would depend in part of how cumbersome your techniques are; how much retraining I’m going to have to do on my staff. (…) I might decide that even if you’re legit and you actually do come up with 15%, that that’s not enough to jusFfy it.”
10
Dissemina0on of results
Results are not geWng out there! SuggesFons:
Researchers should be aXending pracEEoner conferences
Need to disEll research results for pracEEoners
11
Research topics of interest
Developer issues Tool issues
Code issues
User Issues
EvaluaEon Issues
Management issues
12
Reac0ons to our blog?
hXp://Enyurl.com/icse2011panel
13
Broadening this conversaEon
14
QuesEons we posed to the panelists…
What percep0on does industry have of academic research?
What kind of evidence is compelling?
Which research ques0ons should be addressed?
Most useful empirical finding? Success story?
How to improve dissemina0on of research results?
How to engage industry in research?
15
Panel parEcipants
David Weiss, Iowa State University John Penix, Google
Lionel Briand, Simula Research Laboratory
Peri Tarr, IBM Thomas J. Watson Research Center
Tatsuhiro Nishioka, Corporate SoMware Engineering Center, Toshiba CorporaEon
Wolfram Schulte, MicrosoM Research
16
Panel Highlights: David Weiss, perspec0ve from Avaya
17
Panel Highlights: John Penix, Google
• Striving for conEnuous improvement of useful tools: – “We can’t improve what we can’t measure” – “Our goal: make the tools disappear from the workflow”
• In terms of scale…. 5000 devs in 40 offices, 2000 acEve projects, single monolithic code tree with mixed languages, 20+ code changes per minute, 50,000 builds per day, 50 million test cases run per day….
18
Panel Highlights: John Penix (2)
• Problems he cares about… – How do developers work and collaborate? – Team producEvity vs. individual producEvity
• All MS students should know how to do a user study
• Faculty advice: Maintain contact with your students (good links to industry)
19
Panel Highlights: Lionel Briand, Simula Research Labs • PercepEons from industry: – Disconnect, engage researchers in industry – Scalability (heurisEcs versus exact methods) – Applicability (context factors, constraints), realisEc condiEons (human factors)
– Fundamental research quesEons haven’t changed… – Relevant empirical findings? • (Model based) tesEng and empirical research at MicrosoM, InspecEons
20
Panel Highlights: Lionel Briand (2)
21
• So you want to marry an industrial? • Success comes at a high price! – Huge Eme commitment, opportunity cost – You become a development resource, for beSer and for worse
– Effort in developing trust, what you do maXers to them, problems of today
• Industry balances strategies and considers ROI across porqolios – Do you add risk? – Evidence that customers can use it, want it, will pay for it?
Panel Highlights: Peri Tarr, IBM Research
22
Panel Highlights: Peri Tarr (2)
23
• Find someone with a forward looking perspecEve, Build strong, trusEng long term partner relaEonships
Panel Highlights: Peri Tarr (3)
24
Panel Highlights: Tatsuhiro Nishioka, Toshiba
25
Panel Highlights: Wolfram Schulte, Microso7
26
Panel Highlights: Wolfram Schulte (2)
27
Discussion Points
• SoMware development is a wicked problem… how to compete with snake oil salesmen with our modest soluEons?
• How to deal with kinds of evidence accepted… many kinds of important soMware dev problems are not amenable to controlled experimentaEon?
• How to become a beXer storyteller? How to deal with the apparent disconnect?
• How to do industrially relevant research without hurEng one’s academic career?
28
ICSE Panel Slides
hXp://catenary.wordpress.com/2011/06/08/icse-‐2011-‐panel-‐slides-‐and-‐recap/
29