case studies in industry - what we have learnt

What We Have

Technische Universität München

Case Studies in Industry

Daniel Méndez

Technical University of MunichGermany

CESI 2016Austin, Texas

@mendezfe

Joint work withStefan Wagner (University of Stuttgart)

Based on material from a joint work with:Andreas Jedlitschka (Fraunhofer Institute for Experimental Software Engineering)Stefan Wagner (University of Stuttgart)

Feedback fromAntonio Vetrò (Technical University of Munich)

Jonas Eckhardt@binsanoj

Compliments

Critique

(…)

Background

But before we start…

Reviewer Request

“ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”

Reviewer Request

“ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”

Selective perception

“Everything went great!”

Reflection

Reviewer Request

“ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”

Selective perception

“Everything went great!”

Reflection

Reviewer Request

“ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”

ExcuseSelective perception

“Everything went great!”

“Well, maybe not everything, but it wasn’t our fault […]”

Reflection

Reviewer Request

“ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”

ExcuseSelective perception Denial

“Everything went great!”

“Well, maybe not everything, but it wasn’t our fault […]”

“Seriously, how should I have reacted?”

Reflection

Reviewer Request

“ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”

ExcuseSelective perception Denial

“Everything went great!”

Acceptance

“Well, maybe not everything, but it wasn’t our fault […]”

“Seriously, how should I have reacted?”

“Ok, this was really naive | stupid | biased | not really scientific | …”

Reflection

Reviewer Request

“ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”

ExcuseSelective perception Denial Active learning

“Everything went great!”

Acceptance

“Well, maybe not everything, but it wasn’t our fault […]”

“Seriously, how should I have reacted?”

Learning by doing…it wrong.

“Ok, this was really naive | stupid | biased | not really scientific | …”

Finding contacts

Image source: https://c1.staticflickr.com/1/74/196191428_7dc71b7bf7_z.jpg?zz=1

» Integrate studies into ongoing / planned research projects

» Actively approach

– alumni of your university

– companies in your area

– practitioners at (local) events

Finding contacts

Image source: https://c1.staticflickr.com/1/74/196191428_7dc71b7bf7_z.jpg?zz=1

Convincing contacts to participate

Convincing contacts to participate

» Find appropriate terminology (technology transfer)

» Find appropriate incentives for project partners

» Respond to their local needs

» Provide early feedback / results

» In funded projects, probability is higher that practitioners collaborate

Dealing with uncertainty

• What will the stakeholder characteristics be?• Availability• Skills• Motivation to participate• Commitment to goals• Representativeness• …

• What will the data look like?• Quality of the data• Quantity of the data

Dealing with uncertainty

• What will the stakeholder characteristics be?• Availability• Skills• Motivation to participate• Commitment to goals• Representativeness• …

• What will the data look like?• Quality of the data• Quantity of the data

» Take early samples and test them!

» Test instruments and data quality via pilots

» Get to know your subjects early

» Be flexible, always ask yourself (and others): “What is the potential of the data I’m getting?”

Context variables (and phenomena)• hard to determine (or unavailable)• hard to measure

» Context often hard to describe (hence, hard to reproduce by others)

Variables actually reported

Properly characterising the context

Variables you should measure

Variables youcan measure

Context variables (and phenomena)• hard to determine (or unavailable)• hard to measure

» Context often hard to describe (hence, hard to reproduce by others)

Variables actually reported

Properly characterising the context

» Orient yourself on

» other (similar) studies or

» on classification schemes from the area of software process models (“Tailoring”)

» Focus on whole context, not only on cases for which you want to draw conclusions

» Ask yourself: “What information would I myself need to understand and replicate the study under the assumption it would yield same or similar results?”

Variables you should measure

Variables youcan measure

Working with sensitive data

Working with sensitive data

» Clarify the rules:

» what you need for the case study

» what can be stored and reported (internally and externally)

» Plan review and acceptance cycles for reports and publications

» The “easier” way (sometimes): Joint publication

Possible distortions in data collection:• biased researchers

(trying to sell own PhD topic)• weak moderation skills• hidden agendas of respondents

(missing trust!)• (…)

Skills matter

Possible distortions in data collection:• biased researchers

(trying to sell own PhD topic)• weak moderation skills• hidden agendas of respondents

(missing trust!)• (…)

Skills matter

» Don’t be biased!

» Involve other researchers in:

» Design of instruments

» Data collection (e.g. interviews)

» Continuously practice your

» Moderation skills

» Rhetoric and listening skills

» Build trust by showing loyalty and respect

Possible problem: Support by management…only by management.

» No backup in projects (organisational culture)

Beware the ivory tower

Possible problem: Support by management…only by management.

» No backup in projects (organisational culture)

» Make clear study plan including needed subjects & cases (and formally agree on it)

» Talk early to subjects

» Find active supporter (not sponsor) with own interests in project (= their own incentives)

Beware the ivory tower

Case study data should be disclosed• reliability and trustworthiness?• reproducibility and replicability?

» The success of empirical SE relies on a collaborating community

(Data) Openness and transparency

Case study data should be disclosed• reliability and trustworthiness?• reproducibility and replicability?

» The success of empirical SE relies on a collaborating community

(Data) Openness and transparency

» If possible, make your (anonymised) data accessible to others

http://openscience.us/repo/

Properly reporting the results

Case study results need to be disseminated • to practice• to academia

» How to properly report?

Properly reporting the results

Case study results need to be disseminated • to practice• to academia

» How to properly report?

» Report results according to target group:

» Small presentations to management w/ focus on results

» Peer-reviewed publication w/ (selected) research scope

» (Technical) Reports with full data (and analysis)

» …

» Rely on established guidelines for reporting on case studies

KEY TAKE AWAYS

Case Studies are a little bit like dating

• Finding someone sharing your interest is difficult… getting his attention to get further with you even more.

• You need deal with uncertainties

• You need to compromise (be pragmatic and, maybe, also a little bit of opportunistic)

• Social skills — especially listening skills — matter

• Sometimes, it takes very long to get lucky

Case Studies are a little bit like dating

• Finding someone sharing your interest is difficult… getting his attention to get further with you even more.

• You need deal with uncertainties

• You need to compromise (be pragmatic and, maybe, also a little bit of opportunistic)

• Social skills — especially listening skills — matter

• Sometimes, it takes very long to get lucky

But there is one difference

• Null results matter

• In contrast to your dating experiences, we want to hear about your case study experiences (especially the failed ones)

Thank you!

Daniel Méndez

Daniel.Mendez@tum.de

@mendezfe

• Slides will be made available (and probably tweeted)

• Approach him if you need material (studies, templates, …)