innovation experiment systems practices (icsob 2015)
TRANSCRIPT
Teemu Karvonen, Lucy Ellen Lwakatare, Tanja Sauvola, Pasi Kuvaja. Markku Oivo
MALMÖ UNIVERSITYHelena Holmström Olsson
CHALMERS UNIVERSITY OF TECHNOLOGYJan Bosch
Hitting the Target: Practices for Moving towards Innovation
Experiment Systems
AGENDABACKGROUND AND MOTIVATION
RESEARCH GOALS (RESEARCH QUESTIONS)
RESEARCH DESIGN (METHODS AND STUFF)
RESULTS• CASE STUDY FINDINGS• ”EXTENDED STAIRWAY TO HEAVEN”
Software business- & development challenges
SPEED &EFFICIENCY
BUSINESS ACCURACY”Hitting the Target”
SW company’s capability for continuous delivery
SW company’s capability for customer experimentations
Agile software development (Highsmith J., 2009)• Agile Manifesto• Scrum, XP, TDD, Continuous Integration…
Lean Software Development (Poppendieck M, 2007)• Lean thinking, Toyota Way• Continuous Improvement, Kaizen, Value stream mapping, Kanban…
Beyond agile methods• Continuous delivery I.e build, test and deployment automation (Humble & Farley,
2010) • Continuous experimentation (Fagerholm et al. 2014)• Continuous innovation & Lean Startup-method (Ries E., 2011)• Continuous software engineering Continuous * (Star) (Fitzerald et.al, 2014)
• Stairway to Heaven model, Innovation experiment systems & HYPEX model (Holmström Olsson H., Bosch J. 2013)
BACKGROUND & RELATED STUDIES
Stairway to Heaven model”EVOLUTION OF SOFTWARE DEVELOPMENT COMPANIES”
(Holmström Olsson H., Bosch J, Alahyari H. 2012)
Traditional Development
Agile R&DContinuous Integration
Continuous Deployment
R&D as Innovation Experiment
Systems
Step A:Step C: Step E
Step D
R&DValidation
R&D
Product ManagementR&D
Validation(Customer)
Product ManagementR&D
ValidationCustomer
Validation
R & D
Customer
Product Management
Step B:
RESEARCH
GOALS
To understand software development companies
current ways of working and their status in relation to Stairway to Heaven model steps
To undertand software development companies
opinions about benefits and barriers in transition towards Continuous Deployment and Innnovation Experiment Systems
Research questions:RQ1: What are the benefits and barriers that software development companies experience when moving towards Innovation Experiment Systems
RQ2: What are the key practices that software companies need to adopt in order to evolve their software development practices according to the Stairway to Heaven model?
Research design
• Replication/rerun of multiple-case study conducted 2012 in Sweden (Holmstöm Olsson et al. 2012). Minor modifications to interview plan.
• Method: Multiple-case study– Semi-structured interviews and qualitative data analysis (Data coding in Nvivo)– Interpretive data analysis approach
• Interviews conducted during Oct-Dec 2014 in 5 software development companies from Finland (Company A,B,C…)
• Five case companies from Finland– 4 companies mainly operating in embedded systems domain (R&D services, telco, wireless
systems etc.)
– 1 company mainly operating in information technology (IT) services domain – Products: Base stations, wireless device platforms, PC-client sofware, web-applications…
• 24 interviews, 4-5 interviews per company• 1,5-2 hour individual semistructured interviews• Interviewees from multiple roles (product managers, project managers, architecture
designers, UX designers, developers, testers, quality managers…)
RESULTS
CASE STUDY FINDINGS• All 5 companies had adopted Agile development practices (Typically Scrum or Kanban or
”some mix of them”). 1 company (interviewed team) had just started using Scrum method.
• 4/5 had adopted continuous integration practices and was able to run automated builds and test cases -> Developers were able to get fast feedback of how their code changes integrates and works.
• Typical challenges in Continuous Integration
– Expenses and resources for CI, embedded systems hardware configurations (Special HW accelators etc.…) Improvement of test automation coverage
• ”The challenge”: Moving towards continous deployment
– Improvement of software development capabilities for Continuous Deployment
– Increasing customer’s awereness of the benefits of Continuous Deployment? Findinglead customers.
Traditional Development
Continuous Integration
Continuous Deployment
R&D as Innovation Experiment
Systems
Step DStep B
Agile R&D
− “You actively seek the feedback and send interim versions of SW and prototype HW, to collect that feedback. In the end, the result would be a product that the customer actually want and not a product that they specified in the beginning, if those two things are not the same anymore.”
− “Mainly the customer feedback, I think that's the most valuable information that you get from deploying the early releases to customer.”
− “More visibility to the, how the development is proceeding”
− “It is also a way to show the customer that we are on schedule to deliver what we have promised, to do”
− “We (and customer) are both saving money. We can keep our schedules better. You can have those features that you really want and need, more.. faster than earlier. “
− “I think that, we benefit, and I benefit from not doing work that’s going to be trashed, that we develop stuff that’s really required and we develop in an order that customer needs it. “
Moving towards Continuous Deployment and IES
Key Benefits – Quotes
− “If we do handsets, deployment is fairly easy. We can do over the air updates, it's very simple. But then, in some cases we have very complex systems where multiple pieces of the system needs to be updated to specific versions. In those cases, it is more challenging. It might take even, days to get the system updated and back up and running again - It is almost impossible to do it continuously.“
− “In the early stages of the development it is risky. If the quality of your deliverables is R&D (quality), which it is in the early stages.”
− “If you start deploying too early, to your customers, it might cause some negative effects. Because sometimes customers are not as educated as us how, the development cycle goes. They might freak out if they see the status of the deliverables too early. So I would be cautious when starting the continuous deployment.”
Moving towards Continuous Deployment and IES
Key Challenges – Quotes
− “I think the obstacle is the money. It is expensive but, doing it in a right manner could make the next projects cost less money.”
− “Business is one limitation here. This level E (experimentation) is of course interesting, and it would be very nice to work in project with this kind of a way of working. But, I can't see, it’s not so near future for us. Maybe someday. “
− “Of course our customer they do have their own processes and way of working. It’s quite hard for us to, tell them that please change your, processes. “
− “My first idea of (R&D as Innovation Experiment System) would be a nightmare. I would not want to go there. We do this type of work when we develop, demonstration, product for some customer. “
− “Customers need to get involved and they have to be continuously willing to check things, which might be hard “
Moving towards Continuous Deployment and IES
Key Challenges – Quotes
WHAT ARE THE BENEFITS OF
• The innovation validation is fast• Immediate feedback is obtained• New business opportunities are identified• Development resources are focused
Continuous Integration
Continuous Deployment
• Customers get fast and incremental delivery of relevant functionality
• Customers can perform their own testing and business activities on top of deliveries
a) Moving from Continuous Integration to Continuous Deployment
b) Moving from Continuous Deployment to R&D as Innovation Experiment Systems
Continuous Deployment
R&D as Innovation Experiment
Systems
WHAT ARE THE BARRIERS OF
Continuous Integration
Continuous Deployment
a) Moving from Continuous Integration to Continuous Deployment
b) Moving from Continuous Deployment to Innovation Experiment Systems
Continuous Deployment
R&D as Innovation Experiment
Systems
• The shortening of the validation and verification cycle is complex and expensive
• Lack of trust in software quality• Missing functions may cause a negative
impression – Traditional delivery expectations
• Customer feedback integration into the short development and business planning cycle.
• Difficult to conduct experiments in safety critical systems
PRACTICES FOR
Continuous Integration
Continuous Deployment
Moving from Continuous Integration to Continuous Deployment
Moving from Continuous Deployment to R&D as Innovation Experiment Systems
Continuous Deployment
R&D as Innovation Experiment
Systems
• Identify and incorporate lead customers in development process
• Renew business model, contracts, marketing and sales strategies
• Provide architecture where software functionalities can be deployed independently
• Improve automated system testing and adopt a continuous release process
• Ensure that System/UX design and business development work in short cycles and in alignment with R&D
• Adopt data-driven strategic decision-making model.
• Implement A/B testing with the customer• Adopt product platform (e.g. virtualization,
cloud technologies) that enables flexible experimentation.
• Establish a short customer feedback loop and process for data-driven decision making
• Synchronize supplier and customer organization in short development cycles
Extended Stairway to Heaven model Practices for Moving towards Innovation Experiment Systems
Extended Stairway to Heaven model (Practices)
20 practices in 4 categories:
– Business
– Architecture
– Process
– Organisation
4 adoption levels
Institutionalised
Product
Team
Not adopted
Business practices Architecture practices
Innovation experiment systems A/B testing Product Architecture supporting run-
time variation of
Product
Continuous Deployment Business model
transformation
Not adopted Componentisation for partial
release & rollback
Product
Continuous Integration Reduce cost of bad quality Institutionalised Modularisation to improve
ability to do unit &
Institutionalised
Agile R&D Product owner to represent
the customer
Institutionalised Lead architect to protect
architecture from erosion
Institutionalised
Traditional Development Customer validation at the
end of the project
Not adopted Early-on requirement freeze Not adopted
Process practices Organisation practices
Innovation experiment systems Customers real usage data
collection practices
Not adopted Product management &
business development
Not adopted
Continuous Deployment Continuous release process Team UX/system design integrated
in team
Product
Continuous Integration Test-driven development &
automated test scripts &
Institutionalised V&V function integrated in
agile team (continuous
Institutionalised
Agile R&D Sprints & daily standups Institutionalised Feature teams (cross-
functional)
Institutionalised
Traditional Development Milestone-driven
development
Not adopted Large development teams
divided into discipline
Not adopted
Extended Stairway to Heaven model (Example)
CASE COMPANY A
Business practices Architecture practices
Innovation experiment systems A/B testing Product Architecture supporting run-
time variation of
Product
Continuous Deployment Business model
transformation
Not adopted Componentisation for partial
release & rollback
Product
Continuous Integration Reduce cost of bad quality Institutionalised Modularisation to improve
ability to do unit &
Institutionalised
Agile R&D Product owner to represent
the customer
Institutionalised Lead architect to protect
architecture from erosion
Institutionalised
Traditional Development Customer validation at the
end of the project
Not adopted Early-on requirement freeze Not adopted
Process practices Organisation practices
Innovation experiment systems Customers real usage data
collection practices
Not adopted Product management &
business development
Not adopted
Continuous Deployment Continuous release process Team UX/system design integrated
in team
Product
Continuous Integration Test-driven development &
automated test scripts &
Institutionalised V&V function integrated in
agile team (continuous
Institutionalised
Agile R&D Sprints & daily standups Institutionalised Feature teams (cross-
functional)
Institutionalised
Traditional Development Milestone-driven
development
Not adopted Large development teams
divided into discipline
Not adopted
FUTURE STUDIES?
Ideas about directions of the future studies
Investigating different stakeholder’s point of view.How customers perceive benefits and challenges associated to continuous deployment / experimentation?How subcontractors and technology platform providers perceive benefits and challenges associated to continuous deployment / experimentation?
Expanding view from company level to whole supplychain or ecosystem level.
What are the challenges and benefits of continuous deployment/experimentation in ”system level” e.g. software ecosystem?
SUMMARYResearch
New ways of working -Steps towards Innovation Experiment Systems
Agile & CI – ”First two steps”• Integrated and iterative Code & Test
-cycles• Automated I&V, rapid feedback for
developers• Sprint based planning, reviews and
repetitive retrospectives (continuous improvement)
Continuous deployment ”3rd Step”• Integrated and iterative Plan-Design-
Code-Test-Deploy -cycles• Ability to deploy at will. Frequency
not deciding factor. Could be weekly or daily deployments to production.
Innovation experiment systems• Continuous customer feedback• Data driven development decision
making• Plan-Design-Implement-Test-Deploy-
Get Feedback• Capability to learn from customer
real usage data
Hitting the Target: Practices for Moving towards Innovation Experiment Systems. Karvonen T., Lwakatare L., Sauvola T., Bosch J., Holmström Olsson H.,
Kuvaja P., Oivo M.
RESEARH METHOD:• A multiple-case study conducted in Finland (October-December 2014). • 24 interviews, 5 software development companies, Qualitative data analysis
RESEARCH QUESTIONS:• Moving towards Continuous Deployment & Innovation Experiment Systems.
RQ1: What are the improvement steps and key practices?RQ2: What are the benefits and barriers?
MAIN CONTRIBUTION:• Extension of the Stairway To Heaven model
– Introduction of steps and practices for moving towards Innovation Experiments Systems
STYDY IN A NUTSHELL:
COLLABORATION PARTNERS:UNIVERSITY OF OULU, MALMÖ UNIVERSITY,
CHALMERS UNIVERSITY OF TECHNOLOGY
