creating a software factory · creating a software factory accelerate development, augment code...
TRANSCRIPT
V1.0 | 2019-04-24
Steve Barriault – Director Sales and Field Engineering APAC – VectorCAST – Vector Informatik
Creating A Software FactoryAccelerate development, augment code quality
2
u Motivation
Software and Testing
Accelerating Development and Testing
How To Build the Ultimate Test Environment
Conclusion
Agenda
3
Who we are?
Motivation
Aerospace
Automotive
Medical
Industrial
Business-Critical
Railway
Founded 1990, Rhode Island, USA
Founded 1988 , Stuttgart, Germany
Since 2017 there is only one
4
‘Conforming to a rule’
Did we build it right?
u Not necessarily did we build the right thing: requirements!
In this presentation, we will just concentrate on developing software
Compliance
Motivation
ISO 26262-6 2011; Page 5
5
u “80% of embedded systems are delivered late” *
u “New code generally has 50 to 100 bugs per thousand lines” *
Software has bugs
Motivation
* Jack Ganssle, embedded systems commentator
Quality
Speed Price
Idealbut
possible?
Good and fast
Good and cheap
Cheap and fast
Expensive Slow
Poor
6
In many projects: test == buffer
Motivation
time
Design and Requirements
Coding TestConcept and
Planning
Releasedate
Startdevelopment
Plan
Reality 1
Reality 2
Design and Requirements
Coding Test
Design and Requirements
Coding Test
Releasedate
Commercial and reputational risk: delay or bugs
Release and Maintenance
7
Motivation
u Software and Testing
Accelerating Development and Testing
How To Build the Ultimate Test Environment
Today’s Demo
Conclusion
Agenda
8
What should a Software Developer be?
Software and Testing
Software is critical…
What we need is a
Software Factory
Would you put a house together the way we put software together?
What are carpenters always have in their toolbox – and constantlyusing?
Answer: a level and a meter tape!
9
Motivation
Software and Testing
u Accelerating Development and Testing
The Ultimate Testing Environment
Conclusion
Agenda
10
Key defining principles to build the ultimate software factory
Accelerating Development and Testing
Nbr Key principle
1 Leverage tried and tested manufacturing standards
2 Understand and track the requirements
3 Create a continuous testing environment
4 Work smarter not harder
5 Build a scalable constant integration engine
11
u Understand the System
u Just-In-Time delivery
u Find bottlenecks
u Boost communication and collaboration
u Stop the production if there is an issue
u Everyone helps if there is a major issue
#1 Leverage tried and tested manufacturing standards
Accelerating Development and Testing
An example of a Kanban board applied to Software development process
Managing the ‘work in progress’
12
#2 Understand and track the requirements
Accelerating Development and Testing
Coding
System Testing
Requirements Analysis
High LevelDesign
DetailedDesign
Integration Testing
UnitTesting
System Requirements
High-Level Requirements
Low-Level Requirements
Run Test CasesLink
Run Test CasesLink
Run Test CasesLink
Customer
Request
▪ Basis for SW Development
▪ Basis for SW Tests (functional)
▪ Allows Traceability
• Requirement <-> Test Case
• Requirement <-> Source Code
▪ Allows Change Impact- Analysis
• When source code changes:
• -> what requirement might be affected
• When a requirement changes:
• -> what source code might be affected
• -> what test case might be affected
Write Test
Cases
Write Test
Cases
Write Test
Cases
13
#3 Create a continuous testing environment
Accelerating Development and Testing
u Test as early as possible
u Use simulation
u Define test cases with the requirements
u Regression Testing on each code change
u Change-based testing –only run tests affected by source code changes OR requirement changes
u "Baseline test cases“ option for legacy code
u How? Code coverage tells us which tests go through which functions.
void func1(void)
{
/* I have changed! */
}
void func2(void)
{
/* I have NOT changed! */
}
Tests A, B, D, …
Tests A, C, E, J, Z
14
#4 Work smarter not harder
Accelerating Development and Testing
An example?
Early Detection
1.Engineer modifies code & injects a BUG!
2.Engineers commits code
3.Continuous Integration server runs impacted test casesERROR DETECTED IMMEDIATELY
4.Notification is given to the team, correction is brought to the code
Error Prevention
1.Engineer modifies code & injects a BUG!
2.Engineer runs tests affected by the change immediately on his own PC
3.ERROR DETECTED – no one is impacted except the engineer
4.The code is committed when no tests are failing(Quality gate check)
Both approaches reduce debugging time significantly but error prevention completely eliminates the (negative) impact
of bug introduction onto the other team members
Now?
1.Engineer modifies code & injects a BUG!
2.Engineers commits code
3.Tests run once in a while (2 weeks?) and maybe find the trouble
4.DEBUG PHASEWork until found and fixed!
15
u Continuous integration in an assembly line process
u A fully tested source code base should always be available to produce a product
u Each stakeholder can run all tests
u Developer
u Tester
u CI Server
u And what about different code configurationsand calibrations?
u Tests should be recyclable as much as possibleacross all possible configs and calibrations
#5 Build a scalable constant integration engine
Accelerating Development and Testing
Tests from sports-car configuration being used (and specialized, if need be) by SUV, at a click of a mouse
16
Motivation
Software and Testing
Accelerating Development and Testing
u How To Build the Ultimate Test Environment
Conclusion
Agenda
17
Test Cases grow during project development
The Ultimate Testing Environment
System Requirements
System
Test Cases
High-Level Requirements
Integration Test Cases
Low-Level Requirements
Unit
Test Cases
Test Cases
Source Code
Coding
18
What can change?
u Requirement
u Invalidate Test Cases linked to that Requirement
u Source Code Traceability can give a clue what code needs to be changed
u Test Case
u Re-run the changed Test Case
u Source Code
u A real challenge
u Do we need to execute all test cases?
u Tools can help reducing the test execution times with an intelligent Test Case Selection
Change Impact Analysis
The Ultimate Testing Environment
19
Continuous Testing of Code Changes
The Ultimate Testing Environment
System Requirements
System
Test Cases
High-Level Requirements
Integration Test Cases
Low-Level Requirements
Unit
Test Cases
Test Cases
Source Code
Coding
Code Change
20
Change-Based Testing Integrated Into Continuous Integration
Simulator / Target
VectorCAST/AnalyticsWeb dashboard analysis
Simulator / Target
Managers
VectorCAST Integration/functional automation (CANoe, scripts, etc)
VectorCAST Unit/module testing
High-level requirement black-box testing
Low-level requirement White-box testing
The Ultimate Testing Environment
VectorCASTand other tools, reqs
21
Parallel Testing
The Ultimate Testing Environment
Independent Test Environments
Test Environment
Full Run
Test Environment Full Run
Test Environment Full Run
Test Environment Full Run
Test Environment
Full Run
Test Environment
Full Run
timeTest Environment
Full RunTest
Environment Full Run
Test Environment Full RunTest Environment
Full Run Test Environment Full
Run
Test Environment Full Run
Sequentialexecution
Parallelexecution
22
u It is important to visualize project metrics in charts, graphs and heat maps
Visualization
The Ultimate Testing Environment
23
u Define and use quality gates
u Build documented workflows for each Activity
u SCM commit requires > Clean status analysis report?
> Completed peer review?
> All tests OK on home machine? Some tests OK?
> +More?
u Branch cannot go to QA until all existing tests pass
u Quality gates Must be Automated (factory!)
Potential work flow: Software change
The Ultimate Testing Environment
24
Motivation
Software and Testing
Accelerating Development and Testing
How To Build the Ultimate Test Environment
u Conclusion
Agenda
25
SW Factory
How will you test in your next project?
Conclusion
Weeks Minutes
Time to Test
Manual Automatic
Level of Automation
Per Release Per Change
Frequency of Test
QA Everyone
Who Runs Tests